(somewhat OT) How to talk to "though" boss?

[techcode]

There is a ton of articles and discussion on this subject - I realize that ("30 new ways to make your boss listen to you" articles LOL). But since most of the smart people are fellow mongers - I'm asking for your help here. I also figure that with Perl being less picked solution with manager types - there is probably more people with hands on experience on how to make your boss think it was his idea :) And I'm asking for the technical guidance as well ... if how I think that system should be optimized is correct ...

The boss is long time in IT, made a small fortune, but his stubbornness is (beside making us the employees jobs tougher than they should be) costing him money (well he is earning less than he and we could). We end up doing all the things twice - first we do it his way (even though we say there is some better way), and then do it again using our initially proposed way ... But by then, there is a lot of other things breaking down, and you don't have enough time to work on it for real - ans just end up making it work (tm)

It culminated recently when whole system broke down under load - the night before (well that morning) I moonshined (freelanced beside full time job) for an well known client of mine until ~5 AM when I went to bed - updating some old projects. At 8 AM I was woken up by one of the 24/7 support staff "Nothing is working, I can't open the console" (console = web admin interface written in PHP that's overloading the system more than the system is overloading itself :)

Sure enough, load on DB server was in twenties. CPU is like a total of 400% (quad core Xeon and all that GB's of RAM didn't help). And then I realized that for that first 8 hours of the day - we had more new data than total on a better of days. It took me and main system admin (my "supervisor") a few hours of manually stopping everything, vacuuming the DB (PostgreSQL - 8.2 - at least the whole thing lead to supervisor finally updating it to 8.4 :) ) to fix the mess. We hit the new record - load of 42 in the process.

Anyway - company is in SMS business, small bits of data (140 Bytes = 160 chars due to 7bit encoding in GSM), store and forward. Core system is in Perl and is using Net::SMPP for both servers and clients (towards providers/operators). Database is PostgreSQL, and we are using it as queue (a lot of them) as well. Each message has about 5 states, and there is a queue (table constantly being written to, read, and deleted) for each of those. But maybe even worse is that for each of those states, we are updating one extra large table (8 indexes on it, almost 30 fields, many are defined as TEXT type (so no fixed length reserved). So for each message we have 5x row versions and hence need to vacuum and reindex quite often. And we are talking about only having 500K new records (rows) per day, which should laughable amount with say 1 KB of data worth for each.

Boss realized that we should start using cache of some sort (though he still thinks that you can easily say: select * from cache; ) so I'm starting to use Memcache to help out the poor database.

I need a way to convince him that we need to redesign the DB (and my supervisor who is great Linux admin, who is in love with Erlang - yet continually doesn't get it that event based still boils down to the "receiver" process asking at some point - "is there something new for me" - even if thats "is there a signal/semaphore for me that marks there is new data for me").

Move the queues into RAM (perhaps implement queues using Memcache), and move the 3 (INT type) of fields that are the only fields being updated in the before mentioned large table - into separate table. And hence not make a copy of the whole row (with 3 or 4 TEXT fields) for each update, just the three int fiels (2 being updated + id to reference the "all/master details" table).

And start using Memcache for all the reads since basically everything for the operation of the system is searched by INT id (you read ID from the queue, and then read data associated with that ID, do some work, update the big table and place ID in next queue). And we have too many stupid reads that should and could be avoided even if that means having duplicated fields here and there.

The other thing that I need to convince the boss (supervisor got that - guess he liked that he could simply use svn export instead of manually copying files to production servers) is that we need to start using Subversion or similar. There are 5 of us working on code spread among 5 servers, and we are stepping on each other toes by overwriting someones changes, or trying to revert to previous version (do you have a backup of that code - um at best we have previous version~, not the 4th version in the past).

O and console written in PHP - well it loads all (most) data in memory, munges it there (instead of in DB/SQL) and every time someone hits refresh while working with it - we see a couple of seconds to minutes (or an hour) of 100% CPU usage on DB and Web servers. It does that all the time (instead of say calculating it once every 5 minutes and serving it from that cache).

It would be great if while I'm at all of it - I get a raise :D I totally deserve it since I haven't got one since I started working there 9 - 10 months ago - and I proved myself by implementing a new service from scratch that totally rocks and it even cleans up itself (I've set up a cron to vacuum and reindex it) - though would benefit by using queue in RAM instead of DB ;) And I even freelanced for the company before that - so they knew what they are getting.

I'm thinking about dedicating a weekend on writing, testing and benchmarking code and DB optimizations, and fitting it all in a nice presentation to show (projector and all) - so I can't get one of these from the boss or supervisor:

• I don't think it will work ... (boss, sometimes supervisor)
• I think we should use Oracle, it will solve all our vacuuming problems ... (boss, supervisor laughs at that - luckily)
• I read on XyZ website that it's better to ... (boss mostly)
• ...

What do you think about approach I'm planning - perhaps you have some better strategies? And what about the technical aspects I thought off?

Sorry for the long post and text intensive post ...

[moritz]

You could just ask them:

I'm fairly certain $technology could bring us a major boost in $area. What could I do to convince you? Make a nice presentation? Mock up a proof-of-concept? Get a testimonial from an expert?

Of course you can also try to find out on which bases previous technological decisions were made, and try to push into those directions.

[techcode]

I'm pretty much certain that he wanted Oracle since he saw Oracle partner logo on competitors website (well, not so much of a competition as they can eat us for breakfast). I'm not sure why the company switched from MySQL to PostgreSQL - I like PostgreSQL more - but for this particular thing, the lightweightesness of MySQL would probably work better. I'll actually throw in MySQL in the benchmark as well :)

Otherwise, it seems there wasn't that much planning of technical stuff - except we need X for yesterday.

I like your idea and wording - though basically I'm planning on skipping that part of asking, and just go for it. Wouldn't come up with asking expert though, can keep that in sleeve in case he doesn't bite the presentation and proof-of-concept.

[desemondo]

In my opinion, I think that having a boss and collegues who are good to work with make up at least 50% of my overall "job satisfaction". Some might say thats too high, but I've found that if the boss and collegues are good, it helps you get through the (hopefully occassional) dull and annoying periods

I've been particularly lucky to land where I am working currently; everything stacks up perfectly - boss is great and we get along well, coworkers all pull their weight and we see eye to eye on just about everything. When we don't agree there is a healthy measure of open debate and discussion on the issue(s), and the general consensis prevails. And finally the actual work itself is interesting and challenging...

If I were in your shoes I'd be heading for the exit door regardless of how good the 'work' or money is, simply because I wasn't getting along with the boss and that the boss doesn't repect your ideas and views...
And lets face it, hes the boss, not the technical engineer or architect, he should quite frankly p*ss off and stick to managing the people and business, and leave the technical details to the technical people (ie. you and maybe your supervisor)... But I can understand you not wanting to leave after less than 18 months for your resume, etc...

Update: ELISHEVA compiled some excellent info earlier this year, if you haven't read it yet take a look. Help in Tough Times

[JavaFan]

Seems the problem you're having aren't technical. If things aren't working well between your boss and you, and there isn't anyone in the company who can mediate, my advice is to leave the company and find a new job.

[techcode]

The idea is to leave at some point - but right now money is OK (could be better obviously). But it's also challenging work - in both technical terms and dealing with people.

So at this time - the idea is to stay and try to get as much as possible from the situation at hand. That being said, I don't think that staying for too long in such a place is good, but it also doesn't mean you should pack up on first sign of problems.

Few months ago the boss threw it all at me, he sent an email to me + address that forwards to everyone else in the company - and the summary of lengthly email was that I'm incompetent and that all the problems were cause by me, listed one by one. I reflected by hitting up reply to all (it was either that, or not replying at all) and answering to each of those. I basically told him that it's his fault because he wanted it that way (too much, too soon, with stubborn focus on wrong things and wanting some things done the certain not so good way) even though I advised another approach, and due to adhoc no-plan-whatsoever work.

Tomorrow morning, with sad looks in colleagues eyes looking at me as I walked into bosses office, I was told to increase the number of simultaneous connections, and perhaps it will then work better/faster...

[dHarry]

Dear techcode,

It sounds you need to work on your "softcode":P All the technical details you provide are completely irrelevant IMO. Somehow the message is not getting through. Are you sure you communicated it in the "right" way? Quitting your job as other monks suggest seems a bit extreme to me. I would try a course in communication skills first. The boss is the boss, nothing can change that. I think most of us had bosses like the one you describe;) What I find a bit surprising is that you worked for the company before as freelancer. You also knew what you would be getting. It seems there is some mismatch in expectations.

Cheers,

Harry