Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Encrypting Largish Files

by filmo (Scribe)
on Sep 23, 2001 at 07:03 UTC ( #114128=perlquestion: print w/replies, xml ) Need Help??

filmo has asked for the wisdom of the Perl Monks concerning the following question:

I'm dumping mySQL tables to a file. Before they are transfered off the server, I want to encrypt them to keep prying eyes out. (this is all being executed via a Perl script to be run in cron) I've looked at the Crypt::xxx modules, but I'm not clear on the concept of how to implement them. Do I need to break the file down into chunks the size of "blocksize" or can I feed the whole file to the encryption algorithm and let it do it's thing.

Also, I assume that when they say "key", they're refering to the password of x length. I.E. "myPassord123" would be a key 11 bytes in length???
Filmo the Klown

Replies are listed 'Best First'.
Re: Encrypting Largish Files
by wog (Curate) on Sep 23, 2001 at 07:28 UTC
    You probably do not want to try encrypting largish files all at once, to save yourself RAM. You can look at Crypt::CBC which allows you to encrypt a file as you read it without needing to worry about the encryption algorithm's blocksize and without needing to keep it all in memory. (With Crypt::CBC you can use Twofish, Blowfish, DES, and IDEA, at least.)

    The key for any of these encryption algorithms is usually just random data. You are probably better off using a key composed of random data, since it will probably be longer and more random (especially since non-printable characters would be included) and thus harder to brute force. But, a password should work.

Re: Encrypting Largish Files
by blakem (Monsignor) on Sep 23, 2001 at 07:09 UTC
    Are you worried about prying eyes on the network during transit, or keeping the file secure after it arrives at its new destination? If you just want to secure the file during transit, why not use Net::SSH which will keep the file (along with your login and password) encrypted from end to end. It's a lot simpler to use a secure channel, than to pour encrypted data down an insecure one.


Re: Encrypting Largish Files
by Nitsuj (Hermit) on Sep 23, 2001 at 08:24 UTC
    PGP it. It's not that there aren't perl solutions to this problem, it's that there are already VERY good standalone executable solutions to it. If it's too many files to do this by hand, either use a clever command line or a script that calls PGP. One of the keys to good software is wise use and reuse... you gotta know what your assets are, and so forth. This is one of those cases where a full blown script just isn't necessary or even the best idea.

    Just Another Perl Backpacker
      Sounds good, but isn't there a signficant performance penalty (I've heard as high as 1000 x) using public key encryption versus a private key encryption system? I can't have the server brought to its knees while trying to encrypt a large mySQL dump.
      Filmo the Klown
        There will be a performance hit, but I would expect the same hit from most encryption algorithms as well as any script running them. Another suggestion would be a secured FTP client of some sort or another similar device if you're talking about doing this "on the fly" if it's in backups, encrypt them on a separate system.

        Just Another Perl Backpacker
Re: Encrypting Largish Files
by John M. Dlugosz (Monsignor) on Sep 23, 2001 at 07:50 UTC
    I'm doing just that for my network "backup buddy" program!

    I settled on a in-script implementation of CyberSaber to avoid tons of modules and dependancies. In my script, the password is right there in the config, because if you could read that script, you could read the files too.

    No, you should not need to break up the file yourself. Any module worth its salt should have an initialization phase followed by multiple calls to a feeder function where you shovel in a bufferfull and get the encoded text out. Keep calling that, followed by a final flush call.

    Re key and password length: Except for RC4/CyberSaber, the key is a binary value generated from a password by some algorithm such as a cryptographic hash. The length of the password has nothing to do with the final key size.

    If you want the big guns, rather than a simple in-script implementation, go right to Crypt::OpenPGP. It's new, it's awsome, and it's standard RFC2440 compliant.


      Encryption module worth its salt... hehe, crypto puns.
Re: Encrypting Largish Files
by chromatic (Archbishop) on Sep 23, 2001 at 22:58 UTC
    Crypt::CipherSaber (similar to what JMD mentioned) allows you to encrypt or to decrypt to and from filehandles. It'd be as simple as:
    use Crypt::CipherSaber; my $cs = Crypt::CipherSaber->new('Your secret key here'); open(IN, 'mysqldump.sql') or die "Can't open infile: $!"; open(OUT, '>mysqldump.cs1') or die "Can't open outfile: $!"; # depending on your platform binmode(INFILE); binmode(OUTFILE); $cs->fh_crypt(\*INFILE, \*OUTFILE, 1);
Re: Encrypting Largish Files
by suaveant (Parson) on Sep 23, 2001 at 07:30 UTC
    Well, you could kill two birds with one stone, zip allows you to encrypt a zipfile with a password. That way you can compress and protect in one step... dunno how secure it is, but it's worth looking into

                    - Ant
                    - Some of my best work - Fish Dinner

      Zip file security is extremely weak. It won't even stop a 5 year old if he can download a few utilities off the net.

        I was curious as to how well it encrypted... any other zippers have that function?

                        - Ant
                        - Some of my best work - Fish Dinner

      In fact zipfile encryption is weak, the cipher is vulnerable to a known plaintext attack, if you already know a part of what is encrypted you can easily break the encryption.

Re: Encrypting Largish Files
by IndyZ (Friar) on Sep 23, 2001 at 23:53 UTC
    Also, I assume that when they say "key", they're refering to the password of x length. I.E. "myPassord123" would be a key 11 bytes in length???

    "myPassord123" (sic) would actually be a 12 byte key (it has twelve characters). The real problem is that by using only the characters available on your keyboard, and by using english words, you are making a brute force attack on your encrypted data a relatively easy task. You should choose a truely random key generated from your systems entropy pool and then processed further (via a hashing function, possibly) to make it more random and secure. The Crypt::Random module can generate cryptographically secure random numbers (keys) for you.


      The passphrase length in RC4/CyberSaber doesn't have anything to do with the final key length. In fact, RC4 doesn't really have a "key" in the obvious sence the way block ciphers do.

      Rather, the internal state is an array of 256 bytes, one holding each value, that is shuffled into some permutation. The input letters of the passphrase control the shuffling details.

      So, the actual "strength" of the cipher is that it has 256! possible states. So, the "key" is approximatly 1683 bits, since there are 2**1683 possible states for the cipher to be set-up into.

      If you know something about how the passphrase is chosen, then an attacker can search far less than that many possibilities. E.g. with 96 ASCII chars and 8 char length, only 96**8 or 2**52 of the possible keys will ever be used. In order to say the system has an effective key size of 52 bits, the attacker would have to know that, and know which 2**52 keys are available. That's the case with "40 bit" SSL.


        Just for clarification, could you explain your math. How do you get from 256 states to to 1683 bits as 256 * 8 = 2048?

        Also, when you say "only 96^8 or 2^52" how are these two equivalent as 96^8 = 7.21e15 and 2^52 = 4.5e15 or did you mean something different?

        I'm only asking because I'm just starting to learn about cryptography..

        As a side note, how long does it currently take to brute force your way through 4.5e15 keys using non-military strength computers. (assumes that hackers have only consumer to academic calibur machines.)?
        Filmo the Klown

      "Passord" isn't an english word ;)

      My co-worker told me about a program he wrote in his hacking days which tried a brute force assault on a server over http, giving a name and password from a list of first names.

      Apparently a lot of people use their first name for both their username and password - so all you need is a list of popular first names....

      How about hard-coded passwords - can anyone comment on the security of having an admin password written into a cgi script?

      I put an admin section into a site, which is based around a single cgi script. The password for using the admin controls is held as a scalar in the cgi code, and I'm hoping thats pretty safe (because the server will always execute that script and not list it to the browser). Am I fooling myself?

      "Home is a castle you built in my mind; I'm home anywhere, anytime."
      Donny Hathaway
        You're probably fooling youself a bit... The first stage in almost any cgi exploit is to find a way to read the source code. There are lots of ways to do this, but a classic one is to use one insecure CGI to read the source of another. I frequently get entries in my access_log that look like this:
        If the author of some.cgi wasn't careful, its possible that some.cgi will spit back the source to someother.cgi.


      I don't know if I'm clear on the concept. If you let the program choose a randomly large key (eg '23klsd9fjlsd09u3asdljf0' plus a few non-keyboard entries...) and then encrypt the file and send it off site to another location, how do you obtain that automatically generated key for decryption when needed in the future.

      Remember, the original purpose of this was to encyrpt mySQL dumps on a daily basis and send them offsite for backup purposes.

      If it involves sending the key to the backup site, then I'd need a secondary encryption system such as PGP to send the keys securly, in addition to a way to tie each unqiue key with each unique data file. Murphy's Law says that when you try to decrypt one of 365 daily files in a year, its going to be that days key that you've misplaced.

      It seems to me for this to be practical, you would want to avoid having to track randomly generated keys to try and decrypt your files at some future date. My limited understanding is that Public key encryption is much more processor intensive than private key encryption so this is also a concern given the large size of mySQL dumps and the possiblility of simply PGPing the mySQL dump.
      Filmo the Klown

        Partially right. PGP uses private (symetric) key encryption for the data and public (asymetric) key encryption for the secret (symetric) key. Clever guy that Phil, eh?

        alex pleiner <>
        zeitform Internet Dienste

        If you want to use the same key every day, you can randomly generate the key, and then move it via a physical medium between locations, or just move the key with PGP (once). With some algorithms, this will make you more vulnerable to an attack. Using a different key everyday is similar to a onetime pad system. It will probably be the most secure option because if the attacker breaks one days key, he will have to start over from scratch for every other day. If they break the PGP key that you are using the transfer the daily key, however, they will be able to read all of your one time keys. Asymetric (public/private) keypairs probably aren't necessary, since you shouldn't have to give out either key. It could be a good idea though, since if your server (with only copy of the public key) is compromised, the archives will still be safely encrypted.

        Cryptography is a complicated science that I don't personally feel qualified to make a specific recommendation about. For a good overview of different cryptographic techniques and algorithms, go get a copy of Applied Cryptography by Bruce Schneier. It's quite good, especially if you like books with source code included.

        Now, let's think simpler: The real question here is how secure does your data have to be? If absolute security is a must, the don't send your data over the internet at all. Put it onto DATs and hire a courier to transport it for you. You might also want to consider a VPN, which will automatically encrypt all traffic over a network link. Your server must be secure also. If the attacker can get into the server and steal the unencrypted data, there is no point in encrypting it first.

        Wrap up: Sorry that was so long a rambling. Remember, your server must be secure first. Think simple, but secure. Physically moving the medium is more secure than emailing or ftp'ing it. If your data is that important to you (credit card numbers, social security numbers, bank transactions), hire a professional who has experience in this field.


Re: Encrypting Largish Files
by Count Zero (Initiate) on Sep 25, 2001 at 16:19 UTC
    I use Perl to call external tools to create a backup tree and encrypt it.
    These are the actual commands to create the tree and encrypt.
    cd /var/lib /usr/bin/zip -9 -y -q -r /tmp/ mysql /usr/bin/gpg --output /var/backup/ --recipient m --encrypt /tmp/ rm /tmp/
    Use perl to calc the required commands and timestamped filename, for the current host, and then system() them. Note full paths for all root commands!

    Regards Jeff

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://114128]
Approved by root
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (2)
As of 2022-09-29 17:43 GMT
Find Nodes?
    Voting Booth?
    I prefer my indexes to start at:

    Results (125 votes). Check out past polls.