|Problems? Is your data what you think it is?|
snakes and laddersby Logicus
|on Aug 25, 2011 at 03:10 UTC||Need Help??|
This is going to be a long post, so if you don't have time or are not interested, please move along. Anyone making any personal references, trollish remarks, or throwing ad-hominems will be ignored.
About 5 years ago I got into perl programming when I reverse engineered "formmail.pl", which I found on matt's script archive. At that time I was completely ignorant of the world of Perl, but recognised the syntax enough from prior experience with C to start coding.
Over the following months, I found places like tizag.com/perlT which gave me enough information and enough answers to my questions to build a declarative programming environment which I called aXML. (the name is taken but I have yet to come up with a better one)
Over many successive iterations, and many many months of hard work, I built a system which allowed for the embedding of declarative tags into HTML, which fired off subroutines, SSI style. I overcame many design obstacles, one at a time by sheer brute effort and many many iterations.
Eventually I settled upon a syntax for the declarative tags which allowed for attribute parameters, and process order control, and was flexible enough to implement complex systems in, building them up from stacking the tags together using the ruleset.
The system worked. All code logic was separate from the display layer, and I was able to build viable applications from it which were stable and reliable.
There was just one problem... Efficiency. Whilst from an end developers view it is lightening quick to throw a template together with a few declarations which have already been coded to create something new, the system I built was just not fast enough.
I became concerned, perhaps a little overly so, with rendering time. I was sure there had to be a faster way to achieve the same results, and to use the same markup files unaltered to drive the thing.
It was about this time that I first discovered Perl monks, and the wider Perl community. Being new and by nature quite a nervous sort of person, I put up a node where I tried to explain what I had been upto, and the problem I was trying to solve, but I worded in a way that was not very well thought out.
Some people were very kind in their response, especially the user by the name of "grandfather", whilst others were lets say, less than friendly.
The problem was I didn't know how to describe the problem properly, I didn't know what "declarative" programming was, or anything much about the regular methods used as I had come into the world of Perl from a rather odd angle and was not a classical Perl programmer at all.
Coupled with that I also tend to take the words of strangers as being correct and knowledgeable, a personal bias which I am slowly overcoming as I discover more and I am therefore better able to separate the crap-talkers from those with real insight. Man it came as a shock to me just how many people actually talk out of their arses as if they are all knowing... but that's people I guess.
Anyway I digress.
So the system I lovingly built at lengthy effort and have used in many small production sites, was blown out as worthless. But it isn't worthless, it's just inefficient from a processor standpoint. It's very efficient from an end programmer standpoint as complex declarations are simple to build and just work, thus I can put together complex systems like a social network or a forum very quickly and easily.
So I went off on my own again, learnt more things, put together the huge and scattered puzzle that is Perl, and learnt just how much of a gap in efficiency my system has to others. I figured there must be a way of doing it more efficiently, so I started working on trying to figure out a compiler.
Now I know _nothing_ about compilers. I've got quite good with complex regex's, and the old adage comes to mind, if your only tool is a regex, everything looks like a string... or something like that.
I spent a long time playing around with different ways of trying to get it to work, but none of my attempts ever resulted in a system which had the same flexibility as the original. Something always got broken, there were always caveats which I just couldn't live with. After many many months of working at it I finally conceeded to myself that I needed help, that if I kept on trying to figure it out by myself I could be at it for another 10 years...
The one small light at the end of my dismal tunnel was that since I had started the project, the availability of processing power had inevitably increased. Most of the pages I was serving were being rendered in less than 0.1 seconds so I figured that eventually the efficiency issue would just disappear by itself.
I wasn't happy with that notion, I am 100% certain that the system can be refactored to be at least 30x faster than it is. I just don't know how.
So after 4 years of going round in circles, and constantly coming up against the same old problems, I decided the time had come to "eat humble pie" and return to perlmonks to ask for advice.
Apparently some of the best perl hackers in the world had a look at it, but only the tiny minority of responses contained any useful information, the rest consisted of a varying set of; insults, scorn, put downs, and suggestions that the whole thing was a gigantic waste of time, and that I ought to "know my place" as a novice and do as I am told.
Something sparked me off perhaps indignation at people's attitudes, the vague and aloof way they brushed off my problem without really looking into it, perhaps it was the way that they accused me of being delusional or of being incredibly egotistical.... no... I just have a system of declarative programming that I really like and need to make more efficient. That is all.
So I fought back, exchanged rudeness for rudeness, told people where to get the f*** off. Because you see, having had the experience of working within the system I built, I know it is both useful and powerful. It's just not efficient enough, and that is the only problem with it, and I don't want to change _ANYTHING_ else about it.
Then after several weeks of arguing, and upsetting bystanders who then also joined the fray and had the fun of telling me what a deviant I am that needs to eat humble pie and do as I'm told by my superiors and betters, I broke down. Literally completely broke down.
So now here I am at square one, back where I started, I either have to wait for moores law to solve my dilemma, completely start from scratch and learn something I don't want to learn just because we are still using stone-age computer systems, or I have to find an efficient solution to the compiler problem.
I'm not giving up. It is not heresy and I will not recant. One attitude that has always stood me well in the troubles of my life has been my indomitable will to keep on moving forward, despite any setbacks and problems.
Help would be very much appreciated.