Re^4: DBI and FEDERATED table slowdown

[afoken]

some firewall or router in between your machine and the other machine that is very overloaded and/or very misconfigured.

Time for a "war story" about missing communication between departments.

In my first job, there were two IT departments for historic reasons. The bigger one took care of what you would expect from an IT department. Wiring, power, cooling, fileservers, mailservers, print servers, printers, computers, data links between the offices. I worked for the smaller IT department, formally a part of the knowledge management department. We had our own servers, a database, a few webservers, and a load balancer for a document management system a.k.a. intranet, plus some smaller servers for data research.

One day, our intranet servers were moved from a dark corner in the old office building to a small datacenter in the new office building, together with all other servers. The "wiring IT" people did a good job, after the expected downtime, everything just worked smoothly. A few weeks later, people complained that they could not download Powerpoint presentations from the intranet. All other file formats were ok. Uploads were ok. But you could not download any powerpoint file.

What do you think when you hear that story from a user? Right, "layer 8 problem". Too stupid to download a file from a webserver. But it was really the problem. It seemed like the document management software had a severe problem with downloading Powerpoint files. We could upload Powerpoint files, they appeared in the shared filesystem of the webservers as expected, no bits harmed. We could remotely login to one webserver and download powerpoints from localhost just fine. We had several people search for a problem for almost the entire day, digging even deep into the progam code of the document management, searching for a place were Powerpoint files were handled in a different way than other files. We did not find a single location, of course.

"Oh yes, we have installed content filtering firewalls between the servers and the corporate network to improve server security" said one of the "wiring IT" department, who happended to come into our office room for a different reason. "So our servers become secure if you prevent that users DOWNLOAD potentially malicious files from the server that your F---ING firewall did NOT prevent from being UPLOADED to the servers in the first place? What did you smoke? And why the hell did you hide that information from us? Our servers are hardened, they form an isolated network and expose only HTTP and an API port to the corporate network. How do you expect an unidirectional content filter on only the HTTP port to improve security?"

I don't remember the answers any more, but they were on the level of "Um, yes, but our boss ..." The boss that could hardly tell the difference between a mouse and a rack server had decided to spend a lot of money on useless firewalls. I also don't remember if the firewalls were removed or just had their content filter disabled and thus only prevented access to TCP and UDP ports that were never open on the intranet servers.

A lot of money was burned, but money was no problem during the time of the dot-com-bubble. And we actually had a lot of fun digging though the software and network layers almost all the way down to the bare copper wire while searching for the problem.

Alexander

[LanX]

From my experience it's generally good to have multiple lines of defense.

But those should be clearly communicated, documented and automatically tested.

The fact that it took several weeks before the problem became obvious is a flaw in the concept I encountered before:

Personal "war story":

hell broke out when my boss started openly complaining in team meetings, that I broke one of my own applications. I told him that it must be another factor, since neither me nor anyone else touched the code for 6 years(!). And I immediately suspected the database server.

He didn't believe me and started many desperate experimental patches in my product, degrading the codes quality little by little.

After two weeks - I had a concussion and nobody was listening anyway - I informed them that their MariaDB upgrade 4 month ago was to be blamed and SQL backwards compatibility was subtly broken in undefined edge cases.

And I provided a patch fixing that.

2 or 3 month later I got a phonecall from the main user saying

"Hey, your app stopped working!"

Turned out my boss never applied my patches believing in his hacks, and my annoyed colleagues instead continued to work around the problems. Till the data was so corrupted that my app kept raising alarms. (Thankfully I had another line of defense throwing annoying error messages.)

Took me two weeks to apply patches to identify all the data corrupted in the meantime.

Bottom line:

this showed a big management problem, which relied on "human testing". If the issues had been communicated right after the users noticed it we could have correlated it straight away with the DB update. But like in your case there was a delay.

All this communication worked in the past when our team had one third the size at just one location. We literally sat in the same office.

Now with new employees who hardly ever saw or spoke to me, these problems were kept under the radar.

The intelligent solution, better automatic test suits for our web app was declined as "too expensive and unnecessary" ¹

It's a classic combination of Peter Principle and Mythical Man Month which is biting my ex job and I doubt they'll ever fix it.

Meta analysis:

They recently brought in new external management babbling about switching to Python because of the lack of Perl experts ,without really understanding the structural problems of failing communications in a blown up team. ²

Goes without saying that the management doesn't have any qualifications in software development and prefers short term feel good Design by Committee °

And I was required to stay still and not to interfere with their decisions, because I was "only" hired as a freelancing programmer and was threatening their authority.

°) imagine my horror when couldn't attend meetings and decisions were made, like tasking a newbie to switch to another MariaDB engine core in a big bang.

²) Adding a bunch of new Python hackers to the mix will certainly help (sarcasm)

¹) "we need you to do other stuff we don't understand either"