Beware of the newline at the end of the encode_base64() encoded strings

[kirillm]

Just some friday thoughts I'd like to share.

There's a line oriented network protocol where each command ends with a newline character. One of the commands there uses base64 encoded string as it's last argument.

On another side there's a client. It's written in Perl and it wasn't written by me. And it started misbehaving when an additional parameter was added after the one that was base64 encoded. The troubles were caused by a newline...

The client produced base64 data with encode_base64() from MIME::Base64 like this:

my $base64 = encode_base64("<some data>");

This seems perfectly fine. Later these data were sent to the server:

$sock->send("<data>$base64\n") or die "Can't write to $socket";

This also seems perfectly fine. Now a new parameter was added:

$sock->send("<data>$base64<param>\n") or die "Can't write to $socket";

The new parameter was ignored by the server when it processed the command and seeing what was wrong was not easy. Finally debugging at the server side showed that the client sent the new argument as a command by itself. This was very strange. Then I thought that maybe a newline somehow came between base64 data and the new argument... Reading the docs revealed that encode_base64() by default returns the data with a newline at the end!

  encode_base64($str, $eol);

    Encode data by calling the encode_base64() function.
    The first argument is the string to encode. The second
    argument is the line-ending sequence to use. It is
    optional and defaults to "\n".

The fix was easy.

Beware of that newline and have a happy week end!

[graff]

It's a perfectly reasonable sort of trap to fall into. I would just point out that you could have debugged it at the client side, simply by printing that same output to a local file or to stdout instead of (or in addition to) sending it to the socket.

If you had done that sort of test before adding the extra parameter to the string, you might have noticed the extra line-feed in the output, and wondered about that -- maybe it would have no effect on what happens at the server end, but I can easily imagine some situations where an extra line-feed (i.e. a blank line) going through the socket could have an effect, which might be bad.

[ikegami]

That bit me once too. It's particularly easy to forget encode_base64's $eol when using functions like encode, uri_escape and encode_entities regularly.

The solution is simple (and documented): Pass an empty string as second argument if you do not want the encoded string to be broken into lines.

my $base64 = encode_base64("<some data>", '');