Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

perl.org and DNS: A Hate Story

by idsfa (Vicar)
on Feb 23, 2007 at 20:20 UTC ( #601816=perlmeditation: print w/ replies, xml ) Need Help??

A sad tale of disappointment, confusion, betrayal and enlightenment. Also an annoying bug that you should never run into, but could cost you quite a lot of hair pulling if you do. Perl is only tangentially related to this problem, so if you are looking for perl code, skip this node.

The Gathering Storm

The task seemed simple enough. A new Unix box had come in and I needed to install some standard (but non-core) modules. No problem.

# perl -MCPAN -e install Foo . . . Could not connect to host ftp.perl.org with Net::FTP

Say what? The local system is up and on the net. A quick browser pop confirms that we can see google (and therefore, by definition, the whole Internet ;-). Maybe the CPAN module is fubared?

# ftp ftp.perl.org ftp: ftp.perl.org: Unknown host ftp>

Okaaayyy. That's odd. I was able to get to external sites by name just a second ago. Maybe ftp.perl.org's DNS is hosed at the moment?

# nslookup ftp.perl.org Server: nameserver Address: 10.0.0.53 Non-authoritative answer: ftp.perl.org canonical name = ftp.cpan.ddns.develooper.com. Name: ftp.cpan.ddns.develooper.com Address: 64.74.207.33 Name: ftp.cpan.ddns.develooper.com Address: 131.93.97.19 # ftp 64.74.207.33 220 ProFTPD 1.2.9 Server (mirror1.internap.com ) [mirror1.internap.com +] Name (64.74.207.33:root):

So. I can look up the IP number by name. I can ftp to the number. But I cannot ftp to the name. Another quick check shows that I can ftp to plenty of other FTP archives by name, just not ftp.perl.org. Odd.

It's a clean OS install, so I go look on another machine that's been in use for a while. Same symptoms. A different (Unix) OS. Same symptoms. I try directly from my desktop (Win*). Works just fine.

Huh?

Golf It Down

Well, when you've got one working and one not working, you've got a solution if you can just find it. The simplest test case is pretty obvious. I can either ftp to ftp.perl.org or I cannot. I pop two windows, one local to my Win box, one on the Unix box:

UnixWindows
$ ftp ftp.perl.org
ftp: Unknown host ftp.perl.org            
C:\> ftp ftp.perl.org
ftp: Unknown host ftp.perl.org

WHAT?!? But that just worked! I check to make sure nothing else has crashed in the meantime.

UnixWindows
C:\> ftp ftp.perl.org
220 ProFTPD 1.2.9 Server (mirror1.internap.com )
Name (64.74.207.33:foo):
$ ftp ftp.perl.org
ftp: Unknown host ftp.perl.org            

Right. Fine. No problem.

After the Nervous Breakdown

It looks like there is some sort of problem caused when the Unix box looks up ftp.perl.org which not only screws up the requesting machine, but also temporarily screws up every other machine that uses that DNS server. Sounds like a DNS server bug, but what is triggering it?

New test case: packet sniff the DNS traffic during each scenario to see WTF is going on. What immediately pops out is that the Unix box (the one causing the problem) has twice as many exchanges with the DNS server than the Windows box. And the first request it makes is not for the same information as the Windows box asks for.

The Unix box first asks for an AAAA record. And the nameserver barfs up a lung.

It turns out that if I force the Windows box to request the AAAA record first, its behaviour matches the Unix box. So the problem is being triggered by the Unix boxes defaulting to trying IPv6 first ... so much for progessive thinking.

I check to see if some other IPv6 sites have the same behaviour. They don't. So what's so funky about ftp.perl.org's AAAA record?

$ nslookup > set type=AAAA > ftp.perl.org Server: 10.0.0.53 Address: 10.0.0.53#53 Non-authoritative answer: ftp.perl.org canonical name = ftp.cpan.ddns.develooper.com.

Notice anything missing? That's right, there's no actual IP address there. And ftp.cpan.ddns.develooper.com. has no AAAA record at all. Apparently this causes our particular DNS to freak out and refuse to even try to look up the domain until that entry falls out of the cache.

Always the Last to Know

Thus armed, I cast my google net and found CERT Vulnerability #714121 from 2003:

Some DNS servers respond with an inappropriate error message if queried for nonexistent AAAA records, which can lead to possible denial of service.

And indeed, pointing the systems at an external DNS (not running our out-of-date software) showed that the problem disappeared. Check and mate.

The travails of getting the corporate DNS updated are a horror story not fit for the polite company of the Monastery.


The intelligent reader will judge for himself. Without examining the facts fully and fairly, there is no way of knowing whether vox populi is really vox dei, or merely vox asinorum. — Cyrus H. Gordon

Comment on perl.org and DNS: A Hate Story
Select or Download Code
Re: perl.org and DNS: A Hate Story
by Zaxo (Archbishop) on Feb 23, 2007 at 23:40 UTC

    Having a three year old CERT Vulnerability should ease the politics of getting the DNS server upgraded. In my experience it makes a nice friendly implied threat, but you'll know your organization better than I.

    After Compline,
    Zaxo

      Wow idsfa++! Great debugging, perseverance, documentation and willingness to teach others. You're hired!
Re: perl.org and DNS: A Hate Story
by smahesh (Pilgrim) on Feb 24, 2007 at 13:34 UTC

    What a wonderful piece of detective work. ++idsfa for the simple and easy to follow description of the entire process. I like these kind of "straight from the trenches" stories.

    I wonder how many people would have given up after the first DNS error and just reported the issue to the IT department to fix.

    Mahesh

      Yeah. he should send it to Sharky and get a free T-Shirt :=)


      holli, /regexed monk/
Re: perl.org and DNS: A Hate Story
by xmath (Hermit) on Feb 25, 2007 at 12:27 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://601816]
Approved by Joost
Front-paged by liverpole
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (12)
As of 2014-07-30 19:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (240 votes), past polls