OpenStreetMap API : Worker and Plugins architectureby bliako (Priest)
|on May 15, 2019 at 08:47 UTC||Need Help??|
bliako has asked for the wisdom of the Perl Monks concerning the following question:
I am currently interested in fetching OpenStreetMap (OSM) data using Overpass API (see https://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide). This is data regarding roads and cities: traffic lights, roundabouts, amenities, residential addresses. The wealth of data is amazing, its correctness is also amazing judging from what I can verify with my locality. (I can't really thank enough those who came up with the idea and implemented it and collected the data)
I soon came to the, possibly wrong, conclusion that learning the API is hard. What I noticed is that a lot of people just ask how to implement a specific task, e.g. how to fetch all traffic lights data within a bounding box. And so, I soon gave up on implementing a monolithic package which abstracts the queries, do the fetching and cleans the data or exports it. Instead I wrote it so that it does not know of how to do specific queries (e.g. fetch traffic lights) but will do one if user supplies the query content (via a plugin).
I thought of the following architecture (all modules are in PerlOO):
Implement Geo::OSM::Overpass which deals with all the low level communication with OSM server and fetches data -- provided one supplies it with a query text (queries can be in XML or in their own format - QL).
Implement various plugins which take in an Geo::OSM::Overpass object, through which one can run a query or do an export on its last fetched data. Each plugin will know of its specific query content and just that. E.g. how to fetch traffic lights. It may have parameters for cleaning the data afterwards e.g. selecting only traffic lights on highways (or just implement another plugin to do that). The plugins base class will be Geo::OSM::Overpass::Plugin. Any plugin must use this as its parent class and implement the run() method.
For example, I have implemented the Geo::OSM::Overpass::Plugin::FetchTrafficLights to fetch traffic lights data. And also Geo::OSM::Overpass::Plugin::ParseXML to parse the last-query-result (XML) and convert it to a hashtable.
With this model I can get away with my minimal knowledge of Overpass API and let more knowledgable people create their own plugins for fetching this or that and submit them to CPAN without my mediation. At the same time, I have satisfied my basic and urgent need: fetch traffic lights information, but also keep the code expandable and maintanable for the future rather than making a one-off script.
Contrast this model with the model where all queries are implemented in a single module by the current maintainer via user feature-requests (and then scraping on the forums any information on how to convert it to Overpass query which probably will be sub-optimal anyway if you are not an Overpass whizkid *).
Here is an example:
I am looking for comments and suggestions regarding this approach.
*) After my initial success with fetching traffic lights, I pushed my luck to fetch roundabouts. And there all the weaknesses of the underlying OSM model broke: there are roundabouts, mini_roundabouts and circulars. These are nodes. However, the actual roundabouts are ways: a collection of nodes circling the roundabout structure. So, one has to know that or keep asking (see https://forum.openstreetmap.org/viewtopic.php?id=66178 on how the information comes out bit by bit it was like an interogation) and put the puzzle pieces together. Eventually I found out how and also how to ask for the centre coordinate. The kind bureaucrats on the forum are still debating what a roundabout is ...