Really Odd Bug With LWP and Device::SerialPort

[kschwab]

I've found an odd bug / interaction between LWP and Device::SerialPort and/or Win32::SerialPort. I've also found a workaround, so the reason for the question is pure curiousity...and perhaps notifying the module owner of the issue if I can figure out what it is.

The issue, in short, is this:

If I take a scalar variable that was created using LWP::UserAgent's $req->decoded_content method, and pass it to Device::SerialPort as something to be written, I get:

• On Windows (Win32::SerialPort), a buffer overflow warning: "Win32::API::Call: parameter 2 had a buffer overflow at C:/Perl/site/lib/Win32API/CommPort.pm line 220"
• On Linux, no warnings, but the data going out the serial port is corrupted and/or truncated.

Here's the really odd part.

If I "disassociate" the variable that came from LWP before passing it to Win32::SerialPort or Device::SerialPort, everything works great. No warnings, no corrupted data:

my $data = $response->decoded_content;
# doing what should be a "no-op", below, makes the problem
# dissappear.
$data=substr($data,0);

Even more odd, other ways of "disassociating" the variable don't fix the problem.

# doesn't work
$data=~/^(.*)$/;
$data=$1;

# pass $newvar on instead of $data...doesn't work
my $newvar=$data

Aside from $data=substr($data,0), the only other method that fixed the problem was writing the data to a file, and reading it back into a fresh new variable to pass to Device::SerialPort (and/or Win32::SerialPort).

Other oddities:

• The bug shows itself on Windows, Debian, and Raspbian, on three different machines.
• It only occurs when $response->decoded_content is pulling from an https (ssl) url.

I did double-check myself by doing a hex dump of the contents in $data...nothing odd. Just a short text string like "THIS IS A TEST".

Any ideas on what this might be, and why $data=substr($data,0) would fix it when other methods don't?

[runrig]

What version of perl? On 5.8.8, I get:

use strict;
use warnings;

my $url='https://github.com';
use LWP::UserAgent;
use Encode qw(is_utf8);
use Data::Dumper;

my $useragent = LWP::UserAgent->new();
my $request = new HTTP::Request;
$request->method("GET");
$request->url($url);
my $response = $useragent->request($request);

my $data = $response->decoded_content();
print "1: ",is_utf8($data),"\n";
$data = substr($data,0);
print "2: ",is_utf8($data),"\n";

# Prints:
1: 1
2:

On 5.14, I get:

1: 1
2: 1

[kschwab]

5.14.12 on the debian box, 5.16.2 on windows. My customer has the raspbian machine, so I'm not sure what version for them. On both machines above (which both show the bug), your code prints:

1: 1
2: 1

So, if the idea was that making the data non-utf8 was the cure, that doesn't seem to be it.

[runrig]

Well, there is some differences in the 'MAGIC' attributes before and after the substr (no MAGIC before, but MAGIC after) when you do Devel::Peek::Dump($data), but I don't know enough about that stuff to know what it means (non-magic unicode strings don't work with serial ports?).

This appears after the substr:

  MAGIC = 0x6336a0
    MG_VIRTUAL = &PL_vtbl_utf8
    MG_TYPE = PERL_MAGIC_utf8(w)
    MG_LEN = -1

I would think that you might want to specify the encoding when writing to a serial port though.

Update: Sounds somewhat similar to this issue?

[kschwab]

Here's the code to reproduce it. Easier to see on windows, since it produces a buffer overflow. On linux, you would have to hook up a serial analyzer to see that the bytes don't get written (or are garbled). You have to put a valid serial in for $port (COM1, COM2, etc, on windows.../dev/ttyXXX on linux) to see the bug. There's a line to uncomment that will "cure" the bug with what should be essentially a no-op. Also, if you replace the $url with a non-https url, the bug doesn't show itself.

#!/usr/bin/perl
use strict;
use warnings;
#
# You must change what's below to a valid comm port, or the bug won't 
# show itself
#
my $port="COM9";
my $url='https://github.com';
use LWP::UserAgent;
my $serial;
BEGIN {
    if ($^O eq "MSWin32" or $^O eq "cygwin") {
        eval "use Win32::SerialPort";
        die "$@\n" if ($@); 
    } else {
        eval "use Device::SerialPort";
        die "$@\n" if ($@); 
    }
}
if ($^O eq "MSWin32" or $^O eq "cygwin") {
    $serial = Win32::SerialPort->new($port,1);
} else { 
    $serial = Device::SerialPort->new($port,1);
}
my $useragent = LWP::UserAgent->new();
my $request = new HTTP::Request;
$request->method("GET");
$request->url($url);
my $response = $useragent->request($request);
my $data = $response->decoded_content;
#
# the line below grabs the first 6 bytes, which would be <!DOCT
# but, the bug is triggered
#
$data=~s/(......).*/$1/s;
#
# now, uncomment the line below, and the bug doesn't show itself
# $data=substr($data,0);
#
print "data is [$data]\n";
$serial->baudrate(9600);
$serial->parity('none');
$serial->databits(8);
$serial->stopbits(1);
$serial->handshake('none');
$serial->write_settings();
$serial->write($data);

[bulk88]

In Windows (Win32::SerialPort), a buffer overflow warning: "Win32::API::Call: parameter 2 had a buffer overflow at C:/Perl/site/lib/Win32API/CommPort.pm line 220"

It means something in C land went writing off the end of a scalar string. That is a bug in Win32::SerialPort or Win32API::CommPort. Now why would WriteFile the perl sub, which is a read-only function, get something written into the buffer, within, or beyond C malloc block bounds? IDK.

On further research, this sub https://metacpan.org/source/BBIRTH/Win32-SerialPort-0.22/lib/Win32API/CommPort.pm#L1667 looks totally wrong. The buffer passed to WriteFile is lexical, and will be freed at the end of the write_bg sub (but before write_done returns control, which is when the async buffer is released from the Windows Kernel because the async IO operation ended), which is incompatible with asynchronous IO. The scalar should have been inside that object as a reference, and write_done should undef the ref to free the buffer. I did not run any code.

[kschwab]

Makes sense, though it's odd that if I use Device::Serial port on a linux box instead, I get basically the same bug. No Win32 errors, of course, but nothing goes out the serial port. Device::SerialPort and Win32::SerialPort only share a common outside interface...the code itself is quite different.

[Anonymous Monk]

I see this issues on strawberry perl 5.16.2.1 and Win32::API::CommPort 0.20 and Perl Tk both with a separate thread for comms and without it. Error manifests only on certain writes (not always) and solution described in the first post does fix the problem. This definitely requires a fix, as it costs days to debug... And probably is relevant to many people not being vocal...

[Anonymous Monk]

This is an old issue, but apparently still active. But I don't think it is an issue with the SerialPort modules at all.

I started getting "Win32::API:Call: parameter X had a buffer overflow" errors in a large project that has been alive for years. It was quite easy to see what had changed recently: I had started using Encode::decode() to read UTF-8-encoded strings.

It looks to me that the problem is with Win32::API. It cannot handle UTF-8 strings properly. I have filed a bug here.

This could easily be the same problem. On linux side, it would then be a completely different bug, but still likely caused by the same utf-8 confusion.