http://www.perlmonks.org?node_id=1231675

newbietocode has asked for the wisdom of the Perl Monks concerning the following question:

Hello everyone,

I want to create a bare bones, text only site classifieds site which is global. My site will be a niche one, but built like Craigslist and I want it to be able to handle robust traffic. I have seen Craigslist clone software to buy on the internet but they are usually modernised copies built on php or ruby and not really what I want. They have too many gimmicks.

I want to use perl to create my webste because that’s what Craigslist used (I understand it is built on perl and Mysql database). I will host it on a dedicated server.

I have created websites in the past for myself but I used Frontpage 2003. I don’t have a degree in computer science or anything like that, and my knowledge of computers is general (home, university, small business). But I am quick to learn.

I am not a programmer, although I have bought a Perl book to start learning. In fact, I got this website’s url from the book. My questions are:

1.Are there any sections of code here for building a classified site like Craigslist?

2.If not, is there a particular section I should start learning first? I am assuming I’ll have to build the database first, then code the html pages later?

Thank you in advance.

Replies are listed 'Best First'.
Re: I want to build a Craigslist clone
by Your Mother (Archbishop) on Mar 26, 2019 at 04:04 UTC

    Sounds fun. It's a weird situation because there are a number of us here focused on web development who could probably knock out the skeleton of a Craigslist clone in a work week—I knocked out a personal wiki in a three-day weekend a couple months ago—but for a beginner it's probably a three year project and will not be right, robust, secure for another two years.

    I don't mean to dissuade with that. I think you should go for it and I think Perl is arguably the best tool though there is a strong case for any high level language with a good web app framework and community support. Friendly, clear questions will get you terrific support.

    The issue with this kind of project is it's the death of a thousand cuts. HTML is easy. CSS is easy. HTTP is easy. HTTPS is pretty easy. Forms are easy. Automated email is easy. DB programming isn't hard. But you start to compound things and being a sysadmin AND a QA tester AND the programmer AND the designer AND the content manager… it blooms into 18 hour days quickly with little success to show. Finally, while most of that stuff is fairly easy at the roots, security is not; best practices are not; good testing is not; performance tuning is not; maintaining and defending a website is not; solid DBA skills are not. Without all those hard parts, you get what is colloquially known as spaghetti. High risk spaghetti if it contains personal data.

    Now constructive. You asked what to try to tackle.

    • Something that should always be in your mind is security. This is the best one-stop shop: https://www.owasp.org/.
    • You want to use a database. No one recommends MySQL at this point. Instead, consider PostgreSQL or MariaDB; in that order. SQLite is ideal for playing around. It has zero barrier to entry and is much easier to "just use." It's also built into a lot of devices and browsers so knowing its quirks/defaults is a good idea.
      • Try, learn DBI. Most of Perl's database solutions are built on it. There are many DB specific packages that go with it, like DBD::SQLite, DBD::Pg, DBD::MariaDB.
      • Once you have the basics of one or more I personally strongly recommend DBIx::Class but there are naysayers. For me, it removes a huge amount of the tediousness of DB development and affords, if you're careful, extremely semantic and easy to maintain code.
      • Try to find as many good tutorials on databases and database design as possible. Bad data design will complicate your work tremendously.
    • Avoid CGI.pm. There are 20+ years of Perl documents suggesting otherwise. Ignore them. It was Perl's killer app in the 90s. It's an albatross today.
    • Next, web framework. Do not try to roll your own framework. It's a rookie move many of us did that is an excellent teacher but it will absolutely backfire down the road if you hitch your wagon to it.
      • Mojolicious; a favorite among many with extremely smart testing and integrated web-related packages.
      • Catalyst; the 800lb gorilla since 2005 or so. It's what I'm good with. It runs a lot of big websites. It also has excellent testing. I wish it were tighter and smaller and not Moose based. That said, it's amazingly powerful and the steep learning curve, much as with DBIx::Class, makes things easy later. It's like Ruby on Rails but much freer.
      • Dancer2; another favorite. I'm not keen on it but it has a lot of fans who are good hackers.
    • Learn a web server. I strongly recommend nginx. Many still rely on apache; the everything and the kitchen sink of web servers. There are a couple other options but I think those two are the only worth considering.
      • You will probably need an application server eventually. uWSGI is amazing. The nginx guys have one called unit now. I have made the most cursory foray into playing with it so no review.
      • That leads to PSGI. CGI's modern replacement env/interface. Its best implementation is in the Plack space. All modern frameworks support PSGI.

    Lots more… That's a decent baseline to start. If you have specific questions, you'll get great help here. This is a great guide to using the site: How (Not) To Ask A Question.

    Afterward…DERP Afterword: I don't have a CS degree either, or even a BS. This stuff is accessible to anyone with aptitude who is willing to put in the time.

      Thank you for your kind reply, 'Your Mother'. I kind of had a feeling that it would be a tough job for a newbie like me, that is why I spent some time seeking out clones or similar software to buy first. I will chew over what you suggest. I will read the 'questions' link you posted too.l apologise if I did not format correctly.

        I didn't mean to imply you did anything wrong; you didn't. That page just has a lot of good advice on how to get the most out of the site. I hope you stick around.

Re: I want to build a Craigslist clone
by localshop (Monk) on Mar 26, 2019 at 14:01 UTC
    First off - welcome to the wonderful world of Perl. If you stick with it you will be well-rewarded.

    'Your Mother' provided a pretty good perspective and there's nothing there I'd strongly argue against - I'll add a couple of personal steering suggestions.

    • Set up a Github account, learn the basics and share your code. - you'll be surprised by how much help ppl are willing to offer if they can see your code.
    • I'd suggest starting with Mojolicious - The included Morbo and Hypnotoad will provide a server and get you coding straight away without worrying too much about web servers and middleware. There are a heap of other reasons including some excellent step-by-step tutorials that will get you up-to-speed quickly. There's some good modern best practice baked in to this path.
    • (re)-Learn Javascript and start off your architecture without the thinking of rendering HTML in your backend. I'd suggest picking a modern framework such as VueJS and think of your backend primarily as an API to render in your VueJS application.
    • MariaDB and PostgreSQL are great and a relational DB will ultimately serve you well for this type of application but don't be afraid to start out hard coding some JSON structures to start off. A non-relational DB isn't a terrible idea to start ( MongoDB etc ) as it will shift the focus away from perfecting your table structures/schema.

Re: I want to build a Craigslist clone
by bliako (Monsignor) on Mar 26, 2019 at 15:41 UTC

    In my world, data is everything. Processing data (e.g. searching, filtering, analysing, cleaning it) is second most important (or vice versa). Representation, e.g. viewing and on different devices, with different themes and different capabilities is by far the last item to my list. *In my world*, such websites would provide only processed data (e.g. as a result of a search) in very simple, text-based format. Like JSON or a Perl data structure. All requests should also be sent in some similar way, REST or by means of a form sent by POST.

    Of course the question arises: how do you make money without inserting those pesky ads in between your content, or coercing users to follow this or that link, or analysing the way users "navigate" your site in order to "personalise" the content you feed them? I don't know and I won't waste our time giving you advice that I have no clue about.

    You insist on "text-only", "barebones" website. Great! Then put lots and lots of effort to design a solid DB for your data instead of wasting effort on writing HTML which is like being in stone-age. I would even hire someone just to create the DB schemas, the tables, how they are linked and how efficient search is done to them with what they call "indices". You will have authors, readers, guests or logged-in and each will have some privileges in reading and writing that data. You have to figure that out on paper first and have it reviewed by some DB expert.

    Privileges: you need some form of authentication. In browsers this is achieved by an authentication cookie/token which is set at user-side (e.g. in browser) once user's password is verified. Then all subsequent requests to the site mention that cookie and user attains the privileges its user-status entails. This is easier than it sounds but obviously a very critical part of your site so be careful there.

    Then you build your DB and some very high-level functions to insert data, retrieve, search by that or the other, delete data, delete users, send message to users, login or logout a user. These functions should not be concerned whether a webapp calls them or you call them at the command line. They should know nothing about web and should not output html or anything like that. Just JSON or Perl data structures.

    Then you create a mock-data generator and start testing your DB and DB functions. At this stage you must also get a book which will guide you to testing your programs and make you understand how important testing is especially when you will be alone on this project. Appreciating the importance of testing is like transgressing onto another programming level. You will be a different person after that.

    Now you have a DB, perl functions to manipulate data in that DB and users, functions to check on user privileges, etc. It's time to make some types of listener to handle user requests and spawn the different actions required and finally send back to caller results. That's covered best by replies from fellow Monks. Again, you will be writing lots of test cases to test the listener.

    You are done. You have created an entropy-reducing tool and the world will be grateful for organising their space, time and lifes. You can do the most powerful searches and therefore users would love it when the responses match their expectations and more but in the right direction.

    Now start thinking for getting something material, in return for all your efforts. Well, maybe you can create a very simple app which will be like a craiglist assistant. It sits in the background (Edit: I mean it runs on user's mobile phone or computer), checking on user's requests or wishes, searching occassionaly the website, shows the results in a nice non-overwhelming manner to user, even goes and buys something if a deal is OK. Or you can sell craiglist data to other websites who have all those designers and personnel to actually build a frontend (the views, the CSS, the html, the javascript) for interacting with your data. Like embedding youtube videos or weather from 3rd-party providers. You will be that 3rd-party provider.

    bw, bliako

Re: I want to build a Craigslist clone
by harangzsolt33 (Chaplain) on Mar 27, 2019 at 00:18 UTC
    I'm a beginner myself, and when I saw your question, I couldn't help but wonder why you would want to do this. I mean I have no doubt that you CAN do this eventually. But why?

    From a marketing perspective, a website is valuable only if LOTS of people know about it and use it REGULARLY. The really difficult part of launching a project like Cragislist is getting people to use it.

    If you ask people, where do you go online to buy or sell stuff, the first thing that comes to mind is EBAY, AMAZON, CRAIGSLIST, and yourwebsite. In order for your website to receive the kind of publicity and fame and recognition so that it's where people turn to, it has to be REALLY GOOD, REALLY SECURE, and POPULAR. (Popular means lots of people are using it already.)

    There have been many other similar projects such as Backpage.com and several others which failed, because they were unable to attract enough users. or they were unable to monitor what was being posted, so people started selling drugs and all sorts of things, so this project you are trying to start is a really big project.

    I don't want to discourage you, but from a motivational/behavioral perspective, if you want to be able to keep your momentum in this project or any project, you have to be able to make small successes, or otherwise you're going to get discouraged and quit. You either have to break up this project into a million small projects each of which has a clearly defined beginning and end, and you can tell if you succeeded or not. OR you have to pick a project that can be achieved in a lot less time, because we tend to give up. You might not give up right away, but maybe 10 years from now? Before we start something, we have to assess how much time and money it's going to require. If you have no clue about how much investment this will require, you may start with great excitement and then abandon the project less than half way through, because as you start working on it, you realize how difficult it is.

    See Also: other Craigslist-like sites

Re: I want to build a Craigslist clone
by karlgoethebier (Abbot) on Mar 26, 2019 at 19:10 UTC
    "...Craigslist...handle robust traffic...used Frontpage 2003...knowledge of computers is general...not a programmer..."

    Very ambitious. Solution stack knowledge required. Why don't you start with something like WordPress? It's hard enough. And nothing is easy. Best regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

    perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help