Total speculation?

[BrowserUk]

Sitting back and reading the various PMDs over the last week or so, and then delving back into history a ways, I think that I could summarise the current status quo regarding PM developments as

1. There are some, a few or many doesn't really matter, good ideas for fixes, improvements and enhancements to the PM site and it's code base.
2. There is a glut of active PMDevers prepared to work on creating patches and fixes.
3. There is already a backlog of untested, unauthorised patches and fixes awaiting testing and approval.
4. There are, for very good reasons, a very limited number of gods, and apparently even less currently active gods, with the powers of approval/application.

It is the lack of time on behalf of the latter component, along with a lack of a safe test environment that is the bottleneck that prevents patches being tested in a timely manner, with the knock on effect that many of the would-be PMDevers fall by the wayside discouraged.

Making the current PM codebase available so that non-gods might set up an offline test server is not an option for reasons of security.

Using the available Everything-2 codebase for this purpose is not viable as the PM codebase has so many custom patches and fixes already, and is based upon a much earlier Everything engine anyway, that the value of testing with the available version would be minimal if measurable.

Would it be possible, a huge amount of work etc. to set up a band-width/cpu constrained, version of PM running on (one of) the existing server(s) that could have patches applied by lesser mortals than the gods, so that they might be exercised in a non-critical environment?

If they stand up to use by either a selected audience of users PMDevers+Power users+gods, or just whomever was around to see a link posted in the CB, "Please hit this link and tell me what you see", over a period of a week or so, and the associated logs didn't show anything untoward....

That might simplify the process, and ease the workload involved on behalf of the gods, and partially alleviate the bottlenecks?

[liz]

Sometimes I wonder whether it wouldn't be a better idea to look at the functionality that the Monastery currently offers and implement a new engine that would allow the same functionality, using todays knowhow and available software. And then at one point, close the Monastery for renovation for a few hours at most while the database is being converted.

And then I realize somebody/ies would have to do it in their spare time. And I wake up ;-(

Liz

[BrowserUk]

Personally, I don't think that there is that much wrong with PM.

There are some things I would like to see fixed and there's no doubt that any codebase that has been subjected to 'live maintainance' over an extended period is surely due for some remedial refactoring, and there undoubtably some good ideas (tye has mentioned a few in the past) that could be implemented to improve the performance etc.

I don't think I would throw the baby out with the bathwater. Too many times, essential fixes to the original codebase get overlooked in "rewrites from scratch", and then have to be re-invented when the new system goes operational and you rapidly end up with a new, but heavily patched system that exhibites all the same flaws as the original.

It comes back to the thoughts I expressed a while ago of always starting from a working base. Essential to the philosophy is the ability to have ideas and try them out without compromising the working system. A test system is pretty much essential.

It breifly crossed my mind that maybe something like thttpd -- a lightweight, throttleable server might be used to provide this, but I seriously doubt that it has a mod_perl environment available:(

[hardburn]

Perhaps an Apache virtual host would do it. I think a bare-bones throttling system could be set up with a filter. It could certainly be done with an Apache 2.0 PreConnectionFilter, though I doubt anybody is willing to upgrade PM to Apache 2 just yet.

[perrin]

Personally, I don't think that there is that much wrong with PM.

You're talking about features. Feature-wise, there is not much wrong except the performance. However, contributing code to PM is much harder than for most open source projects, and the site suffers because of it.

If PM had all of its code in CVS, and a clear installation procedure, and a test suite (with fake data) to tell you if you broke something, there would be a lot more people helping out with the code. I'm not sure there's any way to get there without a complete rewrite, as Liz suggested, and I share her doubts that anyone will manage to find time to do that. A migration to a later version of Everything might be easier, but I'm not certain of that.

[bart]

Well the prerequirement would be that the people rewriting the new engine should be intimitely familiar with the current implementation... Otherwise they'd surely make the same mistakes all over again. If you can't learn from history, you're bound to repeat it. Which brings us back to the original problem as brought up by theorbtwo... (here)

[AcidHawk]

many of the would-be PMDevers fall by the wayside discouraged
This is definately the case for me, for most of the reasons that you mentioned.

At one point I had a problem with seeing duplicate posts. I did not want to the that the node had been reaped by NodeReaper in Newest Nodes, I did not even want to know they existed.

Fortunately Mr. Muskrat had some code that he let me look at. He wanted me to try and patch it and see how I did it. I have never been completely satisified that I use the "best" methods of getting things done, but gave it a bash. My problem was that I could not test it. So I had no confidence that it would do the job. In the end we both came up with a very similar solution which was implemented.

Without being able to see how things sit together and interoperate other potential PMDevers will be less inclined to even try.

[Aristotle]

Would it be possible, a huge amount of work etc. to set up a band-width/cpu constrained, version of PM running on (one of) the existing server(s) that could have patches applied by lesser mortals than the gods, so that they might be exercised in a non-critical environment?

We have one of those already, but it too is gods only, because both the code and all data lives in the same database. I don't know how much of a problem it would be to separate the code so that exact functional duplicates of PM could be set up, but it too needs someone to do this work at the very least. (I might look into it later tonight or maybe tomorrow.)

Another discouraging problem (for newcomers anyway) is that it's not readily obvious if a node is in use or contains dead code, and we have a lot of the latter..

I'd also bet that besides a few gods, noone really understands more than a small portion of the interactions implications of the complete engine. tye in particular has spent a lot of time optimizing PM's engine flavour and pulling at its innards. I doubt there are a lot of people who can do remotely as much as him or a handful of other gods, which is why I tried to make a move for a tye fund.

Besides that, the most practical and most promising but also least feasible thing I can think of that we could now to alleviate as many of the problems as possible would be to try and migrate PM to a current version of Everything, so EveryDevel and PM can benefit from increased synergy. However, it's nearly as large a project as a full rewrite of a new engine..

Personally, (as I've said a bunch of times in the last couple of days) I'd really prefer if no code were stored in the database since that would allow working with standard tools (of which CVS or cousins would be very desirable), but it's not likely that PM is ever going to (be) switch(ed) to such. So being able to share work with EveryDevel seems like the most practical way to generate the synergies we'd need to bump along faster.

[BrowserUk]

The notion that keeps buzzing around in my head is COW! Copy on Write. Have the test server access the live database (readonly), but tweak the reads so that a look aside table is accessed first. If the request can be satisfied from there, it is, and any modified nodes would be trialled into the lookaside table.

If the requested node is not in the LAT, then it is taken from the live table. If the request results in an update(s), these are made to the LAT not the live DB.

Hmm. How easy is that to set up? Probably not at all, but the thought kept buzzing, so I mentioned it.

[Aristotle]

That's an interesting idea, but - where is authorization for node accesses done? If it happens on the development server (and I don't see how you could feasibly do it elsewhere, ie on the live site), anyone who can apply patches can patch the authorization code and so grant himself full access..

[demerphq]

Hi BrowserUK this is just a follow up to say that there now is a test server set up where we have much less conservative ideas of what should be applied. Furthermore at least two gods have been added since you wrote this (im pretty sure it is from before theorbtwo was deified.)

pmdev has picked up considerably of late and there are all kinds of new stuff on the go. If you are interested in participating then I think you know the procedure to get added to pmdev which will ultimately get you an account of the dev server.

Best wishes

[BrowserUk]

Thanks for the update demerphq.

Frankly, I was surprised when that old post came back to haunt me:) I'm also pretty sure that it pre-dates theorbtwo's deification, of which I was completely unaware. May I say, it couldn't have happened to a nicer bloke.

The test server idea I expressed is hardly rocket science. The limitation appeared (then) to be more to do with hardware and bandwidth rather than lack of desire. The hardware has been uprated once if not twice since I posted, so the applicability of the THTTPD idea probably lost whatever merit it once had.

Actually, I have no idea how to go about becoming a member of pmdev--probably because I've never really looked. I somehow doubt(ed) that there would be any enthusiasm for my particular brand of gung-ho pragmatism amongst that elite. Besides which, I've done little by way of webserver work, and as I said once before, when it comes to coding stuff that has got to be right, I'd personally employ you before me.

Good luck with your endevours.