What *are* the best ways to encrypt data?

[bradcathey]

All this talk about encryption over the past few days has me second guessing myself. And because I'm still new at the security game, but not to flog a dead horse, I'm curious as to my methods.

I write Perl for web CMS apps and forms processing. I'm using Crypt::CBC to 1) store passwords for database access and 2) admin page log-on. Currently:

1) Everytime Perl connects to MySQL db, the encrypted connect password, stored in the root (outside my public_docs) is decrypted using a key, also stored in a file in the root. This has been addressed, even by myself here and here, and I'm still okay with this scenario, though open to advice. More curious about...

2) A user enters a password in an non-secure (no SSL—which now has me nervous) HTML form. The Perl encrypts the password with a static key (again, stored in the root outside of my public_docs) and inserts it into a MySQL table of users. When they log in, the encrypted password is pulled from the db, decrypted, and compared with the log-in.

3) Future application: I want to encrypt larger amounts of text (several paragraphs of plain text) before storing in the database.

Questions:

• Is #1 still sound?

• What are the hazards of #2, and what would be better?

• What are some approaches to #3? Found this but would like to know more about Digest::MD5 or others.

Sorry for all the questions, could break it down into smaller ones, but that just drags this all out even further. Thanks!

[jbware]

As far as #2 goes:
1. Use SSL
2. A better method would be to hash the user's password into the DB. Then, instead of decrypting & comparing to the one they entered, you would hash the one they entered and compare to the hash in the DB. You can verify they actually typed the correct password since the hash will be unique. Although this doesn't stop brute-forcing if the password hash is compromised, it does protect from "knowing an algorithm" (since you can't reverse the results of a hash).

Footnote: Recent events bring some question to the uniqueness of hashes, but the results they found are for very special cases (so far). Its something we need to keep an eye on, but I don't think it invalidates hash-usage approaches just yet.


- jbWare

[bradcathey]

A better method would be to hash the user's password into the DB

This is the asymetric solution I've been reading about, right? Would this be a good place for Digest::MP5? Or can you suggest another? Thanks.

[bgreenlee]

For passwords, Digest::MD5 is fine, although for hashing longer texts you might use Digest::SHA1, as it uses a 160-bit key (vs. MD5's 128-bit).

[neilwatson]

I have a module to do this for a couple of websites:

###########################
# use mapps;
#
# CREATE TABLE users (
#     auid int(10) unsigned NOT NULL auto_increment,
#     auname varchar(30) default NULL,
#     PRIMARY KEY  (auid)
# ) TYPE=MyISAM;
# 
# CREATE TABLE secrets (
#     auid int(10) unsigned NOT NULL auto_increment,
#     passwd char(40) NOT NULL default '',
#     salt int(11) NOT NULL default '0',
#     PRIMARY KEY  (auid)
# ) TYPE=MyISAM DEFAULT;
##########################

package Mapps::Auth;
use Exporter;
use Digest::SHA1;
use DBI;
use warnings;
use strict;
use vars qw($VERSION @ISA @EXPORT);

our $VERSION     = 1.00; 
our @ISA         = qw(Exporter);
our @EXPORT      = qw(&new &auth);

sub new {
    my $class = shift;
    my $self  = {};
    return bless $self, $class; 
}

sub auth {

    my ($self, $dbh1); 
    my $uname = shift;  
    my $passwd = shift;  
    my ($dbsecret, $salt, $uid);

    $dbh1 = DBI->connect('dbi:mysql:itiv', 'lwriter', '**I can't tell 
+you!') or die "Couldn't connect: $dbh1->errstr";
    # get secret from db
    my $statement="SELECT
    admin_users.auid,
    auname,
    passwd,
    salt
    FROM
    admin_users,
    secrets
    WHERE
    admin_users.auid=secrets.auid
    AND
    auname='$uname';";

    my $sth = $dbh1->prepare($statement) or die "Couldn't prepare stat
+ement: ".$dbh1->errstr;
    $sth->execute or die "Couldn't execute statement: ".$dbh1->errstr;

    while (my $ref = $sth->fetchrow_hashref){
        $dbsecret = $ref->{'passwd'};
        $salt     = $ref->{'salt'};
        $uid      = $ref->{'auid'};
    }
    
    # encrypts password using
    # SHA-1 algorithm
    my $sha1 = Digest::SHA1->new;

    # reset algorithm
    $sha1->hexdigest;

    # encrypt
    my $secret = Digest::SHA1::sha1_hex($passwd . $salt);    

    #die "$uid, $dbsecret, $secret, $salt ";
    # does generated secret match database secret?
    if ($secret eq $dbsecret){
        return (1, $uid);
    }
    return (0, $uid);
}
1;

[jbware]

To clarify, there are asymetric & symetric encryption algorithms, which is a seperate topic from hashes. Hashes are best defined as "one way" (you can hash it, but not unhash it), whereas encryption you can encrypt & decrypt it.

On the other point, correct, Digest::MD5 (or other hashing solutions) are best used here. The timing is amazing, check out a current post MD5 - what's the alternative that discusses that hashing question.

- jbWare

[hardburn]

Getting strong cryptographic algorithms was hard (centuries upon centuries of human study into mathmatics). But it's a mostly-solved problem now. It also turns out that this was the easy part. The hard part is useing these algorithms correctly.

I'm not a cryptographer. My math background isn't strong enough. But that's OK, because a lot of other people are cryptographers, and I can benifit from their studies. Further, people who are cryptographers often have little experiance with implementing their algorithms in real applications. So that's where I find my niche: practical application of cryptography.

I'd say that #1 was never really sound, but if you've read about and understand the problems of this approach, then keep it in. (Other posters seem to have mentioned this one, and I don't really have anything more to add).

For #2, I'd use a cryptographic hash with salt value (pseudo-code below):

hash( salt + hash( salt + plaintext_passwd ) )

You store the resulting value and the salt in the database. When you need to authorize a user, you put the password they gave you through the same hashing function and then compare it with the value in the database. The beneifits of this approach is that it is computationally infesible, even for the superuser, to reverse the function to get a plaintext password. The problems are that if a user forgets their password, you'll have to reset it to something instead of giving them the orginal password. Which I don't consider to be a big deal.

However, in your case, I'd be more concerned that the user's channel (HTTP) is not encrypted.

For #3, you'll run into the same problems with #1--how do you keep the master key safe?

The recent collision found in MD5, all on its own, is not a big deal. MD5 wasn't expected to last much longer (indeed, it was expected to fall a lot sooner). Cryptographers and security people have been discouraging the use of MD5 for years.

The big thing about this attack is that it might be extended to other algorithms. SHA-1 might be vulnerable, and we expected that to be good for a while.

[bradcathey]

Thanks hardburn, for that great node. I need to find out more about the crytographic hash and find some examples. Looks like the way to go.

[Joost]

Assuming good use of UNIX file permissions, the encryption of the passwords in 1) doesn't give you any extra security, it just buys you (a little) time. Consider that people who can read the script can see the algorithm to decrypt the password file.

2) and 3) suffer from the same problem.

Note that your scheme does not protect the user or the communications channel in any way.

I can't really figure out why you would want to encrypt these files on the server. If you don't trust the people with access to the files, you have much bigger problems, and I would suggest moving to another machine.

[bradcathey]

I can't really figure out why you would want to encrypt these files on the server.

It's not a matter of "trust" but of trying to comply with still-somewhat-vague HIPAA guidelines for some patient records. Thanks for your reply.

[Joost]

Well, if your web-system doesn't need to READ the data (assuming you're reading it with another, not publically accessible system), you could use a public/private key encryption like PGP, so you can encrypt the data on the server, but only decrypt it using another key only known to the other system.

Joost.

[mortis]

Many of the previous posts are filled with good advice.

The only think I'd like to recommnend is the book Applied Cryptography. That book has very good coverage of the families of cryptographic algorithms you're likely to need/encounter. It can be read to get an overview, or, if you have the desire, it also describes each of the algorithms in detail. It even includes discussion of known attacks against each algorithm.

What I learned from that book has served me very well.

Kyle

[Yendor]

As mortis recommended, Applied Cryptography is a great book for learning how to write and understand cryptographic algorithms. If, however, you are looking for a simpler approach to just getting your feet wet with cryptography, I strongly recommend Simon Singh's "The Code Book". It's a great introduction to cryptographic ideas, all the way from a simple substitution cipher, through Enigma, and onto today's public key cryptosystems.

[freddo411]

I'd like to add a bit to the discussion of data security.

Too often, organizations add encryption/authentication and other schemes to their various systems without understanding what it is they are trying to accomplish. Meeting gov't or corporate rules is often the motivator but you can do this without accomplishing the what the rule was put in place to accomplish. I'm afraid that may be the case here.

So why DO we use encryption and authentication? A: to keep the data out of view of everyone except for those allowed to see it.

We can break this down specifically like this:

* End user machine: (screen, memory, perhaps disk) probably unencrypted and viewable by everyone but out of your scope.

* Network transmission: unencrypted and viewable in your case. Bad monk, no cookie for you! (USE SSL !!!). This is where your effort is most important. Protect the user's credentials otherwise all other security is meaningless because the user can be spoofed. Read and follow jbware's advice above.

* Web Server and DB: (memory, disk, DB) Note that the root user can read everyfile, and the contents of memory if she/he desires, so this person must be trusted and the machine must be physically secured (otherwise I can become root by booting with an external media).

The database on the otherhand can be setup to hold only encrypted data. Thus encrypting the data in the DB can help prevent other staff, db users, the gov't, etc. from accessing this data. To do this effectively, you'll want to use the user's passwd hash as the key to encrypted that row of data. This is not often done however, because the data is NOT recoverable if the passwd is lost. I do NOT recommend building a web site on this principle.

You can use a recoverable key to encrypted the DB data. Some commercial DBs like oracle have encryption routines built in. You can use these to provide some level of security against DB hacking, but keep in mind this is only as good as the security of the key used to access the data, which would have to be stored on the webserver. This an easy way to placate the rule makers.

Cheers

[sintadil]

Note that the root user can read every file, and the contents of memory if she/he desires, so this person must be trusted

This is really getting off of the subject of Perl, but as knowledge is paramount in securing any system, I'm posting it anyway.

One should note also that there have been recent developments of security systems which -- even in their default configurations -- deny even root access to system objects. Such things are sometimes known as Mandatory Access Controls. If security is an issue (and I cannot imagine a situation where it is not), then the administrator should investigate these avenues to be implemented in tandem with existing methodologies.

[derby]

There are no best ways ... just some are more sound than others.

<toot-horn>Check out my writeup</toot-horn> - especially the second and last paragraphs.

[tbone1]

There are no best ways ... just some are more sound than others.

Oh, I dunno. Have you ever read an IRS instruction booklet? That's so encrypted that even they can't decipher it.

[dragonchild]

That's not because they're oneway hashed ... that's because logic professors have paid off the IRS to write sentences that are impossible to parse into predicate logic, thus providing a tax-payer funded source of test questions.

[Anonymous Monk]

1. So you are sending the password in plain text over the network each time you connect to the db? Apparently, you don't trust your system (otherwise, you wouldn't use the rather pointless "encrypt the password, but have the decryption key on file" technique), so why would you trust your network?
2. Two things here. First you let your users send their passwords in plain text over the internet. Which already means that there isn't much security. Second, storing the passwords such that they can easily be decrypted is insecure, and not needed. Standard technique is to encrypt the password using a one-way technique (like crypt) and store the encrypted password. When a password comes in, encrypt it, and compare the encrypted passwords.
3. Digest::MD5 isn't a suitable encryption technique, as this merely creates a hash. It's a one-way function, and you can't go back and retrieve the unencrypted text. To determine what the most suitable technique is, you have to state why you want to encrypt. What do you want to hide against whom? How much time can you spend encrypting/decrypting? How much resources is an attacker willing to employ? Note that security is as weak as its weakest link. There isn't much point in running a "secure OS" which is as hardened as it can be, and using 4096 bit keys if the data you want to protect comes in plain text over the internet, from a virus infected PC.

[hardburn]

Digest::MD5 isn't a suitable encryption technique

There are ways to use a hash function as a cipher. IIRC, it involves using the digest as an output-feedback stream cipher. The security of the resulting cipher is based on the security of the hash function (in other words, don't use MD5 for this). Also, keep in mind that the creaters a cryptographic hash function generally don't have in mind its use as a cipher.

The only practical use I've been able to find for this is a situation where a government has banned strong encryption software, but not digest functions. In which case a coder can grab an existing digest function and make themselves a strong encryption program. Other than that, you're better off using a real cipher.

[Anonymous Monk]

The only practical use I've been able to find for this is a situation where a government has banned strong encryption software, but not digest functions. In which case a coder can grab an existing digest function and make themselves a strong encryption program.

If a government falls for that, why not just take integer arithmetic and build a strong encryption program from that?

[zentara]

This is not perl, but it relates to this discussion.

The FBI would rather be able to spy on us, than let us have secure internet connections.

security article

So you can be pretty sure that the "government" is sniffing all network traffic, and probably harvesting all passwords it can. So the question is "can you trust the government?. I shudder to think about the possibility that people, like those from Enron, would get control over the internet. Ooops, isn't that Microsoft? :-)

[sintadil]

I feel obligated to echo the imperatives of other monks to use SSL. If you don't have the money to pay an organisation like VeriSign (not that I would give them a penny after their .com fiasco), you can create a self-signed certificate which -- although not signed by an established CA -- grants you the intended benefits of SSL, for free¹. If you're using Apache, it ships with documentation on creating a self-signed certificate. And (at least in 2.x, haven't checked into SSL on a 1.x server yet) the SSL configuration file is extremely well-commented, so you should have no problems setting it up.

----

1: Okay, so your users have no reason to believe that it's you who signed it. But by accessing your site right now without SSL, they're saying how much they trust you, so I think that that's a moot point.