Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Building a web-based system administration interface in Perl

by wazoox (Prior)
on Apr 07, 2009 at 19:17 UTC ( #756116=perlquestion: print w/ replies, xml ) Need Help??
wazoox has asked for the wisdom of the Perl Monks concerning the following question:

Yes, here we go again :) Here's my problem : we long used Webmin, and still use it, and still plan to use it for the years to come. But...

Webmin comes with everything including the kitchen sink and the coffee maker, so that's great. However Webmin sucks terribly, too : it's insecure as hell (everything runs as root, all the time), it's insecure as hell (try adding trailing characters to your password...), it's insecure as hell ( use strict? warnings? taint mode? Nah...). Did I mention it lacks security and is written in ugly Perl 4 style (it's slowly improving, but 1994 called and they want their codebase back)?

So... I made a survey of alternatives to webmin and I have to admit it's pretty poor. There's eBox, it's nice, it's pretty but it's far (quite far, actually) from complete. No user profiles, no error messages internationalisation, and a questionable backend relying exclusively on gconfd. I have nothing against gconfd of course, but the whole concept of "whatever manual edit to whatever configuration file will be overwritten mercilessly" bothers me. After all I plan to provide my customers with a nice Web UI, but I'll certainly go on using vi to edit configurations.

Let's recapitulate : Webmin works, but it sucks. eBox works, but it sucks. I found nothing else that would even approximately cut it. So I'll have to build a (simple) system with :

  • messages i18n (like webmin)
  • running as unprivileged user most of the time (like eBox)
  • offering user profiles (accessing different modules/options, like webmin)
  • written in 21st century perl ( including #!perl -T; use warnings; use strict;, like eBox could be )
  • using a standard templating engine ( like eBox );
  • working with an API that allows it to edit standard configuration files the One True Unix Way, with ed or cat or vi or emacs or... (like webmin mostly does)

Well that's about it. So before I start, if anyone knows of such a beast that would save me some weeks of coding, or of some fantastic and easy feature I forgot to mention, please don't keep silent!

Comment on Building a web-based system administration interface in Perl
Download Code
Re: Building a web-based system administration interface in Perl
by afoken (Parson) on Apr 07, 2009 at 21:05 UTC

    Hmmm, I knew that I'm not the only one who wants a "Webmin done right". It should ...

    • be really paranoid about all data coming from the network
    • survive a perl upgrade, both by an automatic system update and by a "cutting-edge-admin"
    • work with old perl versions (think "Debian oldstable", i.e. 5.8.8, or even older)
    • have few external requirements (don't require Oracle 11i just to store sessions, don't require half of CPAN)
    • be pure perl
    • bring its own web server
    • be able to use PAM for authentication and authorisation
    • work with all, or at least most Linux distributions, the *BSDs, Solaris, and whatever looks roughly like a Unix, even if it was relased five years ago
    • be modular, allowing to add plugins for my microwave oven and my server room laser defense system
    • be fast and lightweight. Webmins image-based headlines just SUCK!
    • not require Javascript, Java, Flash or other Plugins. Using those features to ENHANCE the interface is ok, but the basic interface should be plain (and valid) HTML or XHTML. Some plugins may be excepted from this rule, like an SSH oder VNC client in a Java-Applet.

    The interface should ALWAYS run as unprivileged user. A small, separate process should do the privileged work, and that process should not communicate with the browser. And that process is not invoked via the shell, but directly (i.e. NO system "command and parameters"), just to avoid nasty shell surprises.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
      • be pure perl
      • be able to use PAM for authentication and authorisation

      Somehow these two points look rather contradictory to me. So you want to have a pure Perl library that interfaces a C library (PAM)?

      I think that's possible with syscall and/or DynaLoader magic, but somehow I doubt that it'll be easier, more maintainable or less evil than the alternative.

      But I guess we all have our dreams...

        You are right. PAM and "pure perl" are nearly incompatible. If the tool would interface with PAM, it should do so using a classic XS-based module, preferably an already existing and well-tested one.

        Alexander

        --
        Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
      work with old perl versions
      That requirement sucks. Disk space is cheap, use it.

        I can see why that was posted anonymous.

        Operating with older versions of Perl isn't about disk space, it's about allowing a complex already-configured system to run an application without performing an operation (such as upgrade) that could render the rest of the system useless and unstable.

      • be really paranoid about all data coming from the network

      That's what "taint mode" is about, right? :)

      • survive a perl upgrade, both by an automatic system update and by a "cutting-edge-admin"

      Seems reasonable.

      • work with old perl versions (think "Debian oldstable", i.e. 5.8.8, or even older)

      I still use mainly perl 5.8.8, but I don't plan any compatibility with 5.6 or older. It may works of course, but I won't even try.

      • have few external requirements (don't require Oracle 11i just to store sessions, don't require half of CPAN)

      That's natural. It will be somewhat self-contained. However It will use CPAN modules and not some strange weirdo version packaged inside like webmin does.

      • be pure perl

      Mostly. I don't plan using anything but perl, but it will be using XS modules.

      • bring its own web server

      Sorry, no. I won't buy this one. There are too many fine webservers (even Perl ones). It won't be specific to any, but it won't carry one. For a start we'll work with SSL Apache2... I don't see the point of embedding a webserver, except maybe at a later stage for an easier quick and dirty deployment (but I'm too wary of "quick n' dirty" that lasts forever).

      • be able to use PAM for authentication and authorisation

      Yes, this is one of the first planned features :)

      • work with all, or at least most Linux distributions, the *BSDs, Solaris, and whatever looks roughly like a Unix, even if it was relased five years ago

      My main target right now is Linux. I don't see any reason why it wouldn't work on any distribution. For BSDs and Solaris, it will probably work but some modules (file sharing, network configuration, etc) will need an OS specific rewrite. However I'd rather have different modules for different OSes than "webmin style" modules, because it will make the code much simpler and cleaner.

      • be modular, allowing to add plugins for my microwave oven and my server room laser defense system

      Yes, this one is extremely important. We'll need an easy API to ease the way of anyone wanting to write a module. eBox is extremely good so I'll copy them shamelessly :)

      • be fast and lightweight. Webmins image-based headlines just SUCK!

      Well I'll do my best :)

      • not require Javascript, Java, Flash or other Plugins. Using those features to ENHANCE the interface is ok, but the basic interface should be plain (and valid) HTML or XHTML. Some plugins may be excepted from this rule, like an SSH oder VNC client in a Java-Applet.

      Right now it's still a matter of discussion with my team. I'm absolutely with you, but it's clear that making AJAX UIs that works flawlessly in links is a challenge. I still don't know how we'll sort this out; making a good, modern and easy to use interface is also a top priority.

      The interface should ALWAYS run as unprivileged user. A small, separate process should do the privileged work, and that process should not communicate with the browser. And that process is not invoked via the shell, but directly (i.e. NO system "command and parameters"), just to avoid nasty shell surprises.

      I'm eager to ear more for this one :) Right now I think the easiest thing to do is to use a "sudo" module, like eBox, but limits the unprivileged user (in /etc/sudoers) to execute one particular perl program, which will carry the privileged chores through an API (that limits somewhat what can and can't be done). Any other suggestion welcome...

        • Taint mode is just the first step. Each parameter should be validated as strictly as possible. The API could support this by requiring a "description" of the expected parameters to get access to the (validated) parameters. Data::FormValidator is a good start, but it has some legacy problems that I solved in my last project by writing a cleaner re-implementation (that is unfortunately not available to the public).
        • Perl 5.6 or even perl 5.005 should be no big problem unless you need Unicode. In that case, the minimum requirement should be perl 5.8.1.
        • "Own webserver" does not mean that you should write the 10.000th implementation of the server side of the HTTP protocol. But the HTTP server for the tool should be independant of any other parts of the system. Especially on a webserver machine, you don't want to mix public web access and system configuration. You would depend on the security of the public web server, misconfiguring that server would disable your tool, and especially the Apache web server has far too many features that could disturb your tool or give the public full access to the machine. I don't propose an embedded web server. The Web server could (and should) live in a completely independant process. It can be a very small and simple server, as it does not have to handle big loads, it should be able to deliver some static content (images, CSS, JS), and it should be able to pass requests to perl, using CGI, FastCGI or perhaps an equivalent of mod_perl. CGI and FastCGI have the advantage of an additional separation of web server code and application code. A fatal error in the application does not kill the web server, just the application process.
        • OS compatibility: Just don't expect any system to behave like a linux system, or a specific linux distribution, and most things should just work. If you need external tools, don't expect them to work like GNU tools. Don't expect them to be in /usr/bin, don't expect the default shell to be a recent version of bash. Using different plugins for different OSes is a good idea, but for components like an Apache configurator, one tool should be able to handle all OSes. That means that pluigins need a configuration space, e.g. to set the location of apachectl and httpd.conf.
        • AJAX is nice for some enhancements, like auto-completion or maybe a pure-Javascript terminal. Restrict yourself to use Javascript only as an enhancement, not for basic functions, and you will get a "modern" interface which still works nicely in links, lynx, NN3, TV internet set-top boxes, and old smartphones. Valid and semantic (X)HTML and valid CSS would also be good for this purpose.
        • sudo would also be my first tool for the privileged helper process, but as soon as we leave modern Linux distributions, sudo may disappear. On some distributions, sudo is still optional today. During the installation, the user would have to add rules to allow the unprivileged user to execute the privileged helper process. I think that a classic suid wrapper is easier to install. Unfortunately, this requires writing secure and paranoid C code. The Apache suexec wrapper security model could be a good starting point. You don't want any user to be able to edit /etc/passwd using the wrapper, do you?

        The privileged process should do as few tasks as possible. It should not start parsing third party configuration files, or rewrite tons of XML files. For some tasks, like reading and writing configuration files, you could get away with a stripped-down version of cat, chmod, chown, mv or cp, plus the paranoia code based on suexec. But when you want to start a service, the privileged process needs to execute an arbitary program with arbitary parameters as root, like apachectl, exim, smbd, nmbd, sshd, inetd, or the svc tool from daemontools. It would still run all paranoia checks, but it would give the tool full root access without requiring a password. Still better than running everything as root.

        Alexander

        --
        Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
      Kinda old thread, but I couldn't let this one slide (I'm a Webmin developer):

      "be fast and lightweight. Webmins image-based headlines just SUCK!"

      Webmin hasn't had image-based headlines in the default theme in years.

Re: Building a web-based system administration interface in Perl
by andreas1234567 (Vicar) on Apr 08, 2009 at 18:21 UTC
    Is rewriting webmin an option? You seem to have a clear view of what you want to accomplish, and if you play your cards well, you might even get others to help you out.

    I recently refactored 3000+ lines of perl 4 code myself. First I built a test quite for the main functionality. Then I refactored each part of the system, piece by piece. I imagine the same could be done with webmin, even if it may seem too big a task at first.

    --
    No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]

      Yes, it could be done. It is a huge load of work. Cleaning up the code is not the only job. Webmin needs to be splitted into a privileged and an unprivileged part. What about compatibility with existing Webmin plugins? Due to the nature of Webmin, testing all features requires access to a wide range of hard- and software. What about the licenses? Are we allowed to clean up Webmin and all plugins? Do we have a list of all problems in the Webmin code?

      Alexander

      --
      Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
        So, I know this is an old thread, but I thought I'd chime in on this (I'm a Webmin developer).

        "What about the licenses? Are we allowed to clean up Webmin and all plugins?"

        Webmin, Usermin and all core modules are licensed under a BSD style license. So, yes. You are allowed to do anything you like. We'd probably even accept reasonable patches that don't break compatibility.

      Just as a reminder of what a webmin rewrite represents (even a simple code audit), look at the numbers :)
      
      Total Physical Source Lines of Code (SLOC)                = 289,259
      Development Effort Estimate, Person-Years (Person-Months) = 76.80 (921.64)
       (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
      Schedule Estimate, Years (Months)                         = 2.79 (33.46)
       (Basic COCOMO model, Months = 2.5 * (person-months**0.38))
      Estimated Average Number of Developers (Effort/Schedule)  = 27.55
      Total Estimated Cost to Develop                           = $ 10,375,113
       (average salary = $56,286/year, overhead = 2.40).
      
      generated using David A. Wheeler's 'SLOCCount'.
Re: Building a web-based system administration interface in Perl
by jdrago_999 (Hermit) on Apr 08, 2009 at 18:49 UTC

    Interesting project idea!

    Having also used WebMin at a previous job for a few years, I agree with your points about it. I have not used eBox. Just looking at it now, it appears to be a special version of Ubuntu that would run in a virtual machine of some kind.

    Although it sounds like a lot more than a single weekend of hacking will produce, this wheel really needs to be reinvented. It would be nice to see this written on Catalyst or Mojo.

    Features:

    • Publish/Subscribe Eventing System:
      It would be nice to be able to "subscribe" to certain events, eg: "user.created" or "module.installed" or whatever.

    • Plugin Marketplace:
      The ability to buy/sell/give-away-for-free plugins to help manage/install things with this tool would be really nice. You could start with the ability to install packages with yum/apt-get/emerge and go from there.

    Probably loads more, but as long as it does what you need, you're done!

      DON'T use yum/apt-get/emerge/rpm to install plugins. Not all distributions use that tools, some use other tools, some don't use package managers at all. Distributing plugins should be easy. Having to generate 5 or 10 different package formats, each with 5 or 10 different packages for different distribution versions, ist just too much work. The Webmin way, a renamed and compressed tar file, works quite well. We could discuss if we want old or new style tars, with or without GNU extensions, or some other archive format, like arj, lha, cpio or zip.

      Alexander

      --
      Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
        DON'T use yum/apt-get/emerge/rpm to install plugins.

        They certainly have their problems, but reinventing the software-installation wheel might not be a solution.

        I like the idea of writing a simple plugin like this (thinking out loud here):

        Given httpd is installed When update is available Then notify me@my-email.com
      * Publish/Subscribe Eventing System: It would be nice to be able to "subscribe" to certain events, eg: "user.created" or "module.installed" or whatever.

      That's a good idea! I take notes :)

      * Plugin Marketplace: The ability to buy/sell/give-away-for-free plugins to help manage/install things with this tool would be really nice. You could start with the ability to install packages with yum/apt-get/emerge and go from there.

      Yup, this one is already on the map :) Actually it's even somewhat a core part of the whole project (which is bigger than an administrative interface)

Re: Building a web-based system administration interface in Perl
by girarde (Friar) on Apr 08, 2009 at 19:33 UTC
    edit standard configuration files the One True Unix Way, with ed or cat or vi or emacs or...

    Just remember that the One True Unix Way should usually be evaluated in list context.

      Just remember that the One True Unix Way should usually be evaluated in list context.
      LOL
Re: Building a web-based system administration interface in Perl
by Anonymous Monk on Apr 09, 2009 at 12:02 UTC

    Btw, the fact that Webmin runs as root has come in handy on a number of occasions for me, where due to a mistake I have been left out locked out of the box.

Re: Building a web-based system administration interface in Perl
by pileofrogs (Priest) on Apr 09, 2009 at 16:10 UTC

    I'm actually working on this problem right now. I'm making a web interface to allow users to do administrative tasks. And, if I may say so myself, I've had some good ideas.

    The best is, write your web app and your privileged run-as-root server as completely separate applications. Have them communicate by a very simple protocol. I use YAML. Then your privileged app doesn't need to handle the chaos of the internet and HTTP, it only has to handle your very well defined, specific instructions. Everything else it can ignore. Then the web app simply becomes an instrument to convert HTTP to your simple instruction set. You can test your privileged server with a simple client. You can write a library to talk to your privileged server. It's awesome.

    For the web app, I'm using Catalyst for the 1st time, and it's really as good as they say. Use it and kick yourself for anything web you've done that wasn't Catalyst. It's so easy to add a feature here or there. You never really know what you're building until you've built it, so being able to add stuff along the way is huge.

    Don't start with a list of features and try to make them all work. Make the most minimal useful thing you can, publish it and then add features. This has many benefits. One benefit I hadn't thought of until I saw it is, you get to send out a nice message to your bosses every week or so announcing a cool new feature, so they're always thinking how awesome you are.

    I've presently got 4 instances running on two servers. Two development and two production. The two servers are very different and I've written my code to work equally on both. No customization. I test in the two development instances and then roll out to the two production instances.

    I started with a server that can authenticate users and show them how much web space they've used. I've just added authentication against a bunch of new sources and the ability for other IT staff to check the quotas of other people. After talking to users, I'm adding a new feature so lab staff (I work at a college) can check the enrollment and fee payment of students. That's going to take me about a day. And it's gong to make a lot of grumpy people a lot less grumpy. And it's so easy to add features! I just add the capability to my privilged server, test, roll out. Add interface to web app, test, roll out. Email bosses, bask, repeat.

    I think I'm babbling... anyway, hope that's helpful.

    --Pileofrogs

      Have them communicate by a very simple protocol. I use YAML.

      We already built the "administrative core" for a previous project. It uses an XML protocol, which is as complex as necessary to carry all needed tasks, unfortunately :)

      You can write a library to talk to your privileged server. It's awesome.

      Yes, and I even plan to make it possible (and easy) to create interface modules in Python,Ruby or dog forbid, PHP! :)

      For the web app, I'm using Catalyst for the 1st time, and it's really as good as they say. Use it and kick yourself for anything web you've done that wasn't Catalyst. It's so easy to add a feature here or there. You never really know what you're building until you've built it, so being able to add stuff along the way is huge.

      I never managed to wrap my head around these frameworks... I don't know.

      I think I'm babbling... anyway, hope that's helpful.

      Thank you.

Re: Building a web-based system administration interface in Perl
by leocharre (Priest) on Apr 10, 2009 at 14:15 UTC

    Use CGI::Application.

    A development suggestion- code everything to work with a cli. Once the tests work, use the cli, once that works, then code the wui (web user interface) with CGI::Application. The plugins are great, it's easy to write your plugins, super useful way to organize code- into plugins. The plugins can define runmodes into your app etc.

    For some reason I have an aversion to CGI::Application::Framework

    Please do look at: CGI::Application::Plugins *

    When I first looked at cgiapp (apparently common nick) I thought it was silly, a big hairy chunk of code that did little. I think that's the same feeling we get when we are very young and see things like beer, cars, roads- we look at the adults and think to ourselves "really? are you serious? that's important to you?"

      Use CGI::Application.

      Thanks I'll have a look at it. BTW for the cli part, well it's already done and working (at least the hard part of it).

Re: Building a web-based system administration interface in Perl
by clinton (Priest) on Apr 13, 2009 at 19:37 UTC
      OK i'll have look at it, thank you!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://756116]
Approved by moritz
Front-paged by grinder
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (6)
As of 2014-09-20 11:08 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (158 votes), past polls