Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

problems accepting HTTP packets from Amazon

by coontie (Sexton)
on Jun 22, 2011 at 18:31 UTC ( #910958=perlquestion: print w/ replies, xml ) Need Help??
coontie has asked for the wisdom of the Perl Monks concerning the following question:

Hi. I'm using Amazon SNS to send/receive HTTP notifications. Sending works ok but for receiving, I need a simple socket server that would accept inbound HTTP messages. I created this (quick & dirty)
#!/usr/bin/perl -w use IO::Socket; use strict; # Turn on autoflushing $| = 1; my $port = 54321; my $server = IO::Socket->new( Domain => PF_INET, Proto => 'tcp', LocalPort => $port, Listen => SOMAXCONN, Reuse => 1, ); die "Bind failed: $!\n" unless $server; my $new_sock = $server->accept(); while(<$new_sock>) { print $_ . "\n"; } close($server);
However, what happens is that I get ALL of the data incoming EXCEPT that very last byte. The last byte is a squiggly } and I don't get it until much later. It's really strange. And then Amazon attempts to resend the message again. So, it ends up getting all screwed up. I have a feeling accepting raw TCP packets is not the right way, that I'm missing something inherent to HTTP. What am I doing wrong, can you please help? Thank you.

Comment on problems accepting HTTP packets from Amazon
Download Code
Re: problems accepting HTTP packets from Amazon
by 7stud (Deacon) on Jun 22, 2011 at 19:34 UTC
    I have a feeling accepting raw TCP packets is not the right way, that I'm missing something inherent to HTTP. What am I doing wrong, can you please help? Thank you.

    When you send a message over a socket, you have no control over how many pieces the message gets chopped up into. For instance, if you send the message 'hello world', that could get sent in 1 piece, 2 pieces, 3 pieces, etc., and the pieces could all be different lengths. Therefore, the sender and the receiver of the message have to agree on a signal for the end of the message--that way the receiver knows when it can stop trying to read from the socket.

    Some protocols use end-of-file as the signal that the receiver should stop trying to read from the socket. The eof signal is sent when the other side closes the socket. In your case, amazon is doing a retry, so amazon doesn't close the socket, which means your while loop doesn't end after it reads the entirety of the message.

    I don't think you can use the closing brace as a signal to tell you when you should stop trying to read from the socket, because closing braces can exist within a JSON formatted message as well. Maybe look for a line that consists of just a closing brace? Alternatively, you have to figure out what characters amazon sends between the end of the message and the retry, and look for those characters in your while loop so you can break out of it.

    I looked around a little bit, and the retry feature is something people have been requesting, but nothing indicates that amazon has implemented retry yet or how to configure it. Your results indicate that amazon has indeed implemented that feature.

    Also, note that when you use the line input operator, <>, you are not telling perl to read whatever data is available in the socket--instead you are telling perl to block and keep trying to read until perl reads in a newline. After reading a newline, the parts of the message that have been read up to that point are returned and the body of the while loop executes.

      "instead you are telling perl to block and keep trying to read until perl reads in a newline. " OK, good point - thanks. How do I switch it so that I grab whatever's there all at once?
Re: problems accepting HTTP packets from Amazon
by 7stud (Deacon) on Jun 22, 2011 at 22:50 UTC

    I didn't necessarily mean for you to change your line oriented input to something else, I just wanted you to know that the read will block and accumulate pieces of the message until it encounters a newline.

    You can treat a socket as if it were a file. Ideally, amazon would close the socket after sending the message, which would send an eof signal into the socket, and just like when reading a file, your while loop would end. But based on what you say about the input you are receiving, amazon doesn't do that. You could read the whole file in one go, but then you would get the duplicated message in the retry and any subsequent retrys until amazon closed the socket.

    So, you need to examine the input to see what characters lie between the end of the message and the retry. You should inspect() the string to see all the hidden characters. It may be as simple as testing for a line that contains only a closing brace and no whitespace, i.e. the final closing brace of the JSON message, and then breaking out of your while loop.

    I just checked the JSON spec here, json, and whitespace before or after a brace is ignored, so valid JSON could look like this:

    { "glossary": { "title": "example glossary" } "book": { "name": "Hello World" } }

    ...which means a closing brace on a line by itself would not signal the end of the message. Whether amazon actually does that is another thing entirely.

Re: problems accepting HTTP packets from Amazon
by Anonymous Monk on Jun 23, 2011 at 00:13 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://910958]
Approved by philipbailey
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (7)
As of 2014-09-19 06:02 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (131 votes), past polls