If you are using https, then there is no problem. All you would need to do is setup a program to decrypt your files on the server, then send it out to the browser with https. You still would have to limit access with a login, where you could use GPG keys auth, or just a plain password authentication, but you say you have that covered.
in reply to Re^4: Encrypt files on server and then decrypt when user downloads
in thread Encrypt files on server and then decrypt when user downloads
On the server, the first method to decrypt which comes to mind is a GPG pipe. See GnuPG tie to gpg binary gives broken pipe error in CGI output
and he gives a working code example.