Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

converting tcpdump output

by RnC (Sexton)
on May 16, 2005 at 01:28 UTC ( #457321=perlquestion: print w/ replies, xml ) Need Help??
RnC has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

I need an utility to fetch GET, POST, and AUTH payloads. I started searching around, and I found this code, and that's exactly what I need. However, I didn't manage to make the output to be one-lined, as Lincoln showed in the forementioned page.
The code:
#!/usr/bin/perl $LIMIT = shift || 5000; $|=1; open (STDIN,"/usr/sbin/tcpdump -lnx -s 1024 dst port 80 |"); while (<>) { if (/^\S/) { last unless $LIMIT--; while ($packet=~/(GET|POST|WWW-Authenticate|Authorization).+/g) { print "$client -> $host\t$&\n"; } undef $client; undef $host; undef $packet; ($client,$host) = /(\d+\.\d+\.\d+\.\d+).+ > (\d+\.\d+\.\d+\.\d+)/ if /P \d+:\d+\((\d+)\)/ && $1 > 0; } next unless $client && $host; s/\s+//; s/([0-9a-f]{2})\s?/chr(hex($1))/eg; tr/\x1F-\x7E\r\n//cd; $packet .= $_; }

I do get some output, but sliced, as if the "line builder" above didn't work.

I found this thread on Perlmonks on the same subject, which happens to have a very similar example. When trying that one, I don't get any output at all.

My OS: Slackware GNU Linux 10.0

Any help is appreciated. Cheers.

EDIT: Here's the output I'm getting when using the above code.
root@iceblue:~# perl sn tcpdump: verbose output suppressed, use -v or -vv for full protocol de +code listening on eth0, link-type EN10MB (Ethernet), capture size 1024 byte +s 192.168.0.249 -> 209.197.123.153 GET /ind P.......GET./ind 192.168.0.249 -> 209.197.123.153 GET /ind P...`...GET./ind 192.168.0.249 -> 209.197.123.153 GET /ind P...X...GET./ind 192.168.0.249 -> 209.197.123.153 GET /ima P.......GET./ima 192.168.0.249 -> 209.197.79.97 GET /i/p P.......GET./i/p 192.168.0.249 -> 66.39.54.27 GET /ima P...;...GET./ima

Comment on converting tcpdump output
Select or Download Code
Re: converting tcpdump output
by mrborisguy (Hermit) on May 16, 2005 at 01:56 UTC
    can you possibly post some output? or maybe it would help to print out some of the values to debug it, for example just print $client, just print $host, just print $&, etc.
      
      root@iceblue:~# perl sn
      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on eth0, link-type EN10MB (Ethernet), capture size 1024 bytes
      192.168.0.249 -> 209.197.123.153        GET /ind P.......GET./ind
      192.168.0.249 -> 209.197.123.153        GET /ind P...`...GET./ind
      192.168.0.249 -> 209.197.123.153        GET /ind P...X...GET./ind
      192.168.0.249 -> 209.197.123.153        GET /ima P.......GET./ima
      192.168.0.249 -> 209.197.79.97          GET /i/p P.......GET./i/p
      192.168.0.249 -> 66.39.54.27            GET /ima P...;...GET./ima
      
      
      

      Thanks again.
Re: converting tcpdump output
by brian_d_foy (Abbot) on May 16, 2005 at 04:11 UTC

    I wrote a little script to do this for me. You might be able to adapt it more easily.

    --
    brian d foy <brian@stonehenge.com>
      Hi brian,

      I modified your script to match my local config ("eth0" instead of "en1", removed "or src port 80" since I don't need that, and ran it as root directly, so no need for sudoing before). However, I got no output when using it. Here's what I have:
      #!/usr/bin/perl use strict; my( $header, $packet ) = ('', ''); open STDIN, "/usr/sbin/tcpdump -lx -s 1024 dst port 80 |"; #die "Could not open tcpdump" unless $?; while( <> ) { chomp; if( /^\S+/ ) { print_packet( $header, $packet ); $header = $_; $packet = undef; next; } $packet .= $_; } sub print_packet { my( $header, $packet ) = @_; $packet =~ s/\s//g; $packet =~ s/.{104}//; # remove protocol info return unless length $packet > 0; $packet =~ s/0d0a0d0a.*/0d0/i; # Just keep the HTTP header my $pretty = ''; BYTE: while( $packet =~ m/\G([0-9a-f]{2})\s*/ig ) { my $hex = hex $1; if( $hex == 0x0D ) { $pretty .= "\n"; next BYTE } if( $hex == 0x0A ) { next BYTE } $pretty .= do { if( $hex < 0x20 or $hex > 0x7f ) { '.' } else { chr $hex } }; } if( $pretty =~ m/^(?:HTTP|GET|POST)/ ) { print_header( $header ); print $pretty; } return length $packet; } sub print_header { my $header = shift; my ( $time, $src, undef, $dst, ) = split /\s+/, $header; $time =~ s/\..*//; $dst =~ s/:$//; print "-" x 73, "\n"; print "Time: $time Src: $src Dst: $dst\n"; }

      Cheers.
Re: converting tcpdump output
by polettix (Vicar) on May 16, 2005 at 09:45 UTC
    Your code works well for me:
    192.168.0.69 -> 143.48.220.86 GET /~lstein/talks/WWW6/sniffer/ HTTP/ +1.1 192.168.0.69 -> 209.197.123.153 GET /?node_id=457321 HTTP/1.1 192.168.0.69 -> 209.197.123.153 GET /?node_id=234493 HTTP/1.1 192.168.0.69 -> 209.197.123.153 GET /?node_id=6182 HTTP/1.1 192.168.0.69 -> 209.197.123.153 GET /images/clearpixel.gif HTTP/1.1 192.168.0.69 -> 209.197.79.97 GET /i/pair-banner-current.gif HTTP/1. +1 192.168.0.69 -> 209.197.123.153 GET /images/dibona_sm.gif HTTP/1.1
    I suspect there's something weird with your tcpdump. Could you post a sample output of /usr/sbin/tcpdump -lnx -s 1024 dst port 80?

    Flavio (perl -e 'print(scalar(reverse("\nti.xittelop\@oivalf")))')

    Don't fool yourself.
      My pleasure:

      19:57:52.537475 IP 192.168.0.249.1505 > 143.48.220.86.80: P 0:625(625) + ack 1 win 65535 0x0000: 4500 0299 4188 4000 8006 89ae c0a8 00f9 E...A.@..... +.... 0x0010: 8f30 dc56 05e1 0050 5f7c 01ef 931d b19f .0.V...P_|.. +.... 0x0020: 5018 ffff 5545 0000 4745 5420 2f7e 6c73 P...UE..GET. +/~ls 0x0030: 7465 696e 2f74 616c 6b73 2f57 5757 362f tein/talks/W +WW6/ 0x0040: 736e 6966 6665 722f 2048 5454 502f 312e sniffer/.HTT +P/1. 0x0050: 310d 0a48 6f73 743a 2073 7465 696e 2e63 1..Host:.ste +in.c 0x0060: 7368 6c2e 6f72 670d 0a55 7365 722d 4167 shl.org..Use +r-Ag 0x0070: 656e 743a 204d 6f7a 696c 6c61 2f35 2e30 ent:.Mozilla +/5.0 0x0080: 2028 5769 6e64 6f77 733b 2055 3b20 5769 .(Windows;.U +;.Wi 0x0090: 6e64 6f77 7320 4e54 2035 2e31 3b20 656e ndows.NT.5.1 +;.en 0x00a0: 2d55 533b 2072 763a 312e 372e 3629 2047 -US;.rv:1.7. +6).G 0x00b0: 6563 6b6f 2f32 3030 3530 3331 3720 4669 ecko/2005031 +7.Fi 0x00c0: 7265 666f 782f 312e 302e 320d 0a41 6363 refox/1.0.2. +.Acc 0x00d0: 6570 743a 2074 6578 742f 786d 6c2c 6170 ept:.text/xm +l,ap 0x00e0: 706c 6963 6174 696f 6e2f 786d 6c2c 6170 plication/xm +l,ap 0x00f0: 706c 6963 6174 696f 6e2f 7868 746d 6c2b plication/xh +tml+ 0x0100: 786d 6c2c 7465 7874 2f68 746d 6c3b 713d xml,text/htm +l;q= 0x0110: 302e 392c 7465 7874 2f70 6c61 696e 3b71 0.9,text/pla +in;q 0x0120: 3d30 2e38 2c69 6d61 6765 2f70 6e67 2c2a =0.8,image/p +ng,* 0x0130: 2f2a 3b71 3d30 2e35 0d0a 4163 6365 7074 /*;q=0.5..Ac +cept 0x0140: 2d4c 616e 6775 6167 653a 2065 6e2d 7573 -Language:.e +n-us 0x0150: 2c65 6e3b 713d 302e 350d 0a41 6363 6570 ,en;q=0.5..A +ccep 0x0160: 742d 456e 636f 6469 6e67 3a20 677a 6970 t-Encoding:. +gzip 0x0170: 2c64 6566 6c61 7465 0d0a 4163 6365 7074 ,deflate..Ac +cept 0x0180: 2d43 6861 7273 6574 3a20 4953 4f2d 3838 -Charset:.IS +O-88 0x0190: 3539 2d31 2c75 7466 2d38 3b71 3d30 2e37 59-1,utf-8;q +=0.7 0x01a0: 2c2a 3b71 3d30 2e37 0d0a 4b65 6570 2d41 ,*;q=0.7..Ke +ep-A 0x01b0: 6c69 7665 3a20 3330 300d 0a43 6f6e 6e65 live:.300..C +onne 0x01c0: 6374 696f 6e3a 206b 6565 702d 616c 6976 ction:.keep- +aliv 0x01d0: 650d 0a52 6566 6572 6572 3a20 6874 7470 e..Referer:. +http 0x01e0: 3a2f 2f77 7777 2e67 6f6f 676c 652e 636f ://www.googl +e.co 0x01f0: 6d2e 6175 2f73 6561 7263 683f 686c 3d65 m.au/search? +hl=e 0x0200: 6e26 713d 7463 7064 756d 702b 7065 726c n&q=tcpdump+ +perl 0x0210: 2662 746e 473d 5365 6172 6368 266d 6574 &btnG=Search +&met 0x0220: 613d 0d0a 4966 2d4d 6f64 6966 6965 642d a=..If-Modif +ied- 0x0230: 5369 6e63 653a 204d 6f6e 2c20 3134 2053 Since:.Mon,. +14.S 0x0240: 6570 2031 3939 3820 3230 3a32 313a 3030 ep.1998.20:2 +1:00 0x0250: 2047 4d54 0d0a 4966 2d4e 6f6e 652d 4d61 .GMT..If-Non +e-Ma 0x0260: 7463 683a 2022 3261 3662 3030 2d31 3234 tch:."2a6b00 +-124 0x0270: 322d 3335 6664 3761 6163 220d 0a43 6163 2-35fd7aac". +.Cac 0x0280: 6865 2d43 6f6e 7472 6f6c 3a20 6d61 782d he-Control:. +max- 0x0290: 6167 653d 300d 0a0d 0a age=0....


      Now if you can let me know your output, I would appreciate a lot =) If that's the problem, I'll try recompiling it from the source. I'm using the slackware 10.0 package for tcpdump right now.
        Now I see and yes, it's a tcpdump-related issue. I run an older tcpdump (package for Slack 9.0), and it basically misses all beautifying around the raw data: the address-indicator at the beginning and the plain dump at the end of each line.

        A quick hack could be the following. Please note that I'm currently not in the condition of testing it (momentarily in a Win32 environment):

        #!/usr/bin/perl $LIMIT = shift || 5000; $|=1; open (STDIN,"/usr/sbin/tcpdump -lnx -s 1024 dst port 80 |"); while (<>) { if (/^\S/) { last unless $LIMIT--; while ($packet=~/(GET|POST|WWW-Authenticate|Authorization).+/g) { print "$client -> $host\t$&\n"; } undef $client; undef $host; undef $packet; ($client,$host) = /(\d+\.\d+\.\d+\.\d+).+ > (\d+\.\d+\.\d+\.\d+)/ if /P \d+:\d+\((\d+)\)/ && $1 > 0; } next unless $client && $host; s/^\s+\S+\s+//; # remove initial address ind. s/\s{2}.*//; # remove trailing dump s/\s+//; s/([0-9a-f]{2})\s?/chr(hex($1))/eg; tr/\x1F-\x7E\r\n//cd; $packet .= $_; }

        Flavio (perl -e 'print(scalar(reverse("\nti.xittelop\@oivalf")))')

        Don't fool yourself.
Re: converting tcpdump output
by eXile (Priest) on May 16, 2005 at 14:14 UTC
    you also might want to take a look at using ethereal (GUI) and/or tethereal (CLI) as replacements for tcpdump. It's packet selection mechanism is able to analyze HTTP, so you can do your selection of traffic there, for instance this selects all GET and HEAD requests from a live capture on one of my ethernet interface:
    tethereal -i eth1 -R'http.request.method == "GET" || http.request.meth +od == "HEAD"'
    outputs:
    7.518730 66.27.72.141 -> 192.172.226.123 HTTP GET / HTTP/1.1 7.616699 66.27.72.141 -> 192.172.226.123 HTTP GET /HOME/images/nav_1 +_1_HOME.gif HTTP/1.1 7.638719 66.27.72.141 -> 192.172.226.123 HTTP GET /HOME/images/nav_2 +_1_HOME_overview.gif HTTP/1.1 7.641536 66.27.72.141 -> 192.172.226.123 HTTP GET /images/nav_1_2_GL +OBAL_globe.gif HTTP/1.1 7.670158 66.27.72.141 -> 192.172.226.123 HTTP GET /images/nav_2_2_GL +OBAL_globe.gif HTTP/1.1 7.694446 66.27.72.141 -> 192.172.226.123 HTTP GET /HOME/images/nav_3 +_1_HOME_overview.gif HTTP/1.1 7.714087 66.27.72.141 -> 192.172.226.123 HTTP GET /HOME/images/nav_3 +_2_HOME_globe.gif HTTP/1.1
    replace -i <iface> with -r <file> to read from file.
      Thanks again!

      Before I posted I was already thinking about installing (t)ethereal to do that. The reason why I prefer tcpdump is the fact that it's default on 90% of GNU/Linux and Unix systems.

      Now consistency worries me. I didn't know that there could be so much variation in the data output format between versions of tcpdump, and specially, that I wouldn't be able to format this output by supplying different parameters. tethereal will end up as my definite solution.
      Another thing that I missed when running the code you mentioned is that POST payloads are getting stripped as well.

      I managed to see the contents by commenting this line:
      s/^\s+\S+\s+//; # remove initial address ind. #s/\s{2}.*//; # remove trailing dump <-- commented s/\s+//;
      And redirecting the output to a file, then grepping it later. Is there a way of printing it along with the target?

      EDIT: So far, I managed to get the values by modifying the regex line:
      if (/^\S/) { while ($packet=~/(GET|POST|WWW-Authenticate|Authorization|[a-z]+=[a- +z]+).+/g) { ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localti +me(time);
      Basically, I extended the regex to capture a string followed by "=" and then string again.

      But as you can see, it's not a concise solution, since it gets a lot of rubbish as well.
        Ok, I figured it out myself. Here's the complete code.
        #!/usr/bin/perl $|=1; open (STDIN,"sudo /usr/sbin/tcpdump -lnx -s 1024 dst port 80 |"); while (<>) { if (/^\S/) { while ($packet=~/(GET|POST|WWW-Authenticate|Authorizat +ion|Content-Length: \w+\s*).+/g) { ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)= +localtime(time); printf "%4d/%02d/%02d %02d:%02d:%02d",$year+1900,$mon+ +1,$mday,$hour,$min,$sec; print " - $client -> $host\t$&\n"; } undef $client; undef $host; undef $packet; ($client,$host) = /(\d+\.\d+\.\d+\.\d+\.\d+).+ > (\d+\.\d+\.\d ++\.\d+\.\d+)/ if /P \d+:\d+\((\d+)\)/ && $1 > 0; } next unless $client && $host; s/^\s+\S+\s+//; # remove initial address ind. s/\s{2}.*//; # remove trailing dump s/\s+//; s/([0-9a-f]{2})\s?/chr(hex($1))/eg; tr/\x1F-\x7E\r\n//cd; $packet .= $_; }
        Still not suitable, since it creates a line to just contain POST data, when it would be better if it appended the content to the end of a POST line, or something like that.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://457321]
Approved by kvale
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (17)
As of 2014-07-31 22:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (253 votes), past polls