Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: Parsing output

by 1nickt (Abbot)
on Aug 02, 2019 at 21:47 UTC ( #11103786=note: print w/replies, xml ) Need Help??


in reply to Parsing output

Hi there again,

Any ideas?

  1. Take everything you've learned so far about working with Perl data structures and put it back in your newly-expanded toolbox. You'll never not need it, good job.
  2. Throw out most of your program so far.
  3. Set up a relational database for your data. It's time.

For the type of query that you want, being able to say select * from sub where subname = 'foo' order by date_modified desc limit 1 or similar simple SQL, is about 95% easier than trying to morph nested Perl data structures from one format into another.

I understand that you are working with a Mongo DB project, but I am not familiar with Mongo and it looks like you are storing nested structures and yet are not able to query them? If that's so, and you don't just need to read some more Mongo doc, set up an SQLite DB to crunch your data.

You can even set it up in memory in your process, populate it as you read in data, query it to analyze the data and make reports, and throw it away when the process is finished. I use exactly this technique to build nightly reports from a commercial website, reading in raw data from two or three places, building a temporary SQLite DB, and producing the reports from there.

The time it takes to figure out how to do that will not greatly exceed the time it takes to build up a huge ugly Perl program to make the same queries. Your code will be a tenth as long and 100 times more maintainable, readable, and robust.

Hope this helps!


The way forward always starts with a minimal test.

Replies are listed 'Best First'.
Re^2: Parsing output
by ovedpo15 (Monk) on Aug 03, 2019 at 11:42 UTC
    Thanks for the suggestion, we already use MongoDB and it will take a lot of time to change it to an SQL DB because we have a lot of reports already in the MongoDB.
    I will still try my luck here in waiting for a Perl suggestion.
    Thank you for this post and all of the other as well!
      You will probably have more luck, narrowing stuff down to pieces you have problems with into short example(s). We are here to help with the language and programming problems, not to solve your business logic.


      holli

      You can lead your users to water, but alas, you cannot drown them.
        What do you mean by "business logic". I'm just trying to understand from more experienced Perl programmers on how to approach this issue from Perl. I have added my "starter" code suggestion and where I stuck.

      Well, good luck. I'll just note that I was not suggesting dumping Mongo. In fact I mentioned that you should be able to query your structured data there (https://docs.mongodb.com/manual/tutorial/query-embedded-documents/). I suggested marshalling your data for insertion into Mongo using a temporary SQL DB instead of the complex and hard-to-maintain Perl data manipulation routines you are struggling with.


      The way forward always starts with a minimal test.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11103786]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (3)
As of 2019-11-12 04:23 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Strict and warnings: which comes first?



    Results (64 votes). Check out past polls.

    Notices?