Beefy Boxes and Bandwidth Generously Provided by pair Networks Cowboy Neal with Hat
more useful options
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

A sad tale of disappointment, confusion, betrayal and enlightenment. Also an annoying bug that you should never run into, but could cost you quite a lot of hair pulling if you do. Perl is only tangentially related to this problem, so if you are looking for perl code, skip this node.

The Gathering Storm

The task seemed simple enough. A new Unix box had come in and I needed to install some standard (but non-core) modules. No problem.

# perl -MCPAN -e install Foo . . . Could not connect to host ftp.perl.org with Net::FTP

Say what? The local system is up and on the net. A quick browser pop confirms that we can see google (and therefore, by definition, the whole Internet ;-). Maybe the CPAN module is fubared?

# ftp ftp.perl.org ftp: ftp.perl.org: Unknown host ftp>

Okaaayyy. That's odd. I was able to get to external sites by name just a second ago. Maybe ftp.perl.org's DNS is hosed at the moment?

# nslookup ftp.perl.org Server: nameserver Address: 10.0.0.53 Non-authoritative answer: ftp.perl.org canonical name = ftp.cpan.ddns.develooper.com. Name: ftp.cpan.ddns.develooper.com Address: 64.74.207.33 Name: ftp.cpan.ddns.develooper.com Address: 131.93.97.19 # ftp 64.74.207.33 220 ProFTPD 1.2.9 Server (mirror1.internap.com ) [mirror1.internap.com +] Name (64.74.207.33:root):

So. I can look up the IP number by name. I can ftp to the number. But I cannot ftp to the name. Another quick check shows that I can ftp to plenty of other FTP archives by name, just not ftp.perl.org. Odd.

It's a clean OS install, so I go look on another machine that's been in use for a while. Same symptoms. A different (Unix) OS. Same symptoms. I try directly from my desktop (Win*). Works just fine.

Huh?

Golf It Down

Well, when you've got one working and one not working, you've got a solution if you can just find it. The simplest test case is pretty obvious. I can either ftp to ftp.perl.org or I cannot. I pop two windows, one local to my Win box, one on the Unix box:

UnixWindows
$ ftp ftp.perl.org
ftp: Unknown host ftp.perl.org            
C:\> ftp ftp.perl.org
ftp: Unknown host ftp.perl.org

WHAT?!? But that just worked! I check to make sure nothing else has crashed in the meantime.

UnixWindows
C:\> ftp ftp.perl.org
220 ProFTPD 1.2.9 Server (mirror1.internap.com )
Name (64.74.207.33:foo):
$ ftp ftp.perl.org
ftp: Unknown host ftp.perl.org            

Right. Fine. No problem.

After the Nervous Breakdown

It looks like there is some sort of problem caused when the Unix box looks up ftp.perl.org which not only screws up the requesting machine, but also temporarily screws up every other machine that uses that DNS server. Sounds like a DNS server bug, but what is triggering it?

New test case: packet sniff the DNS traffic during each scenario to see WTF is going on. What immediately pops out is that the Unix box (the one causing the problem) has twice as many exchanges with the DNS server than the Windows box. And the first request it makes is not for the same information as the Windows box asks for.

The Unix box first asks for an AAAA record. And the nameserver barfs up a lung.

It turns out that if I force the Windows box to request the AAAA record first, its behaviour matches the Unix box. So the problem is being triggered by the Unix boxes defaulting to trying IPv6 first ... so much for progessive thinking.

I check to see if some other IPv6 sites have the same behaviour. They don't. So what's so funky about ftp.perl.org's AAAA record?

$ nslookup > set type=AAAA > ftp.perl.org Server: 10.0.0.53 Address: 10.0.0.53#53 Non-authoritative answer: ftp.perl.org canonical name = ftp.cpan.ddns.develooper.com.

Notice anything missing? That's right, there's no actual IP address there. And ftp.cpan.ddns.develooper.com. has no AAAA record at all. Apparently this causes our particular DNS to freak out and refuse to even try to look up the domain until that entry falls out of the cache.

Always the Last to Know

Thus armed, I cast my google net and found CERT Vulnerability #714121 from 2003:

Some DNS servers respond with an inappropriate error message if queried for nonexistent AAAA records, which can lead to possible denial of service.

And indeed, pointing the systems at an external DNS (not running our out-of-date software) showed that the problem disappeared. Check and mate.

The travails of getting the corporate DNS updated are a horror story not fit for the polite company of the Monastery.


The intelligent reader will judge for himself. Without examining the facts fully and fairly, there is no way of knowing whether vox populi is really vox dei, or merely vox asinorum. — Cyrus H. Gordon

In reply to perl.org and DNS: A Hate Story by idsfa

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others chilling in the Monastery: (5)
    As of 2014-04-19 00:33 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      April first is:







      Results (473 votes), past polls