Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Same thing as last night, Pinky, try and take over the world!

by blue_cowdawg (Monsignor)
on Sep 30, 2011 at 14:30 UTC ( #928825=note: print w/replies, xml ) Need Help??

in reply to Philosophy: Got the Data, Now what?

      I'm curious at a higher level how one goes about managing data like this once they have it.

My first reaction is that you are probably over thinking this thing a bit.

First off, a few things you need to consider no matter what path you take is

  1. What are your short and long term goals for gathering this data in the first place? Answering this question may help you decided on a proper means of persisting the gathered data now that you have it. Flat text files? Sybase? MySQL? NoSQL? Perhaps a combination of the two.
  2. What is the end product or products that are going to come out of this?

Once you have decided a strategy then go forward with it. If you are queasy at the thought of working with SQL then I'd partner with someone skilled in it and let them figure out your table schema and even write queries for you. Take the queries and encapsulate them in a Perl module and forget about them once they are working the way you want them to.

Peter L. Berghold -- Unix Professional
Peter -at- Berghold -dot- Net; AOL IM redcowdawg Yahoo IM: blue_cowdawg
  • Comment on Same thing as last night, Pinky, try and take over the world!

Replies are listed 'Best First'.
Re: Same thing as last night, Pinky, try and take over the world!
by raybies (Chaplain) on Sep 30, 2011 at 16:26 UTC

    Well the programs run a large complex "machine" (keeping it generic). Each piece of the machine has a number of realtime programs that run them, seperate control systems, and they communicate with a central computer system/display.

    I've collected all the messages that come from each of the subsystems that communicate with this central display process (by examining the actual C sourcecode of each program used in each machine, because that's where the messages originate from). They're scattered through a dozen independent realtime systems with interchangeable components, etc.

    The users of the whole system want to know why some of the messages (often error messages) appear on the display even when there isn't a problem and other times they simply don't know if the messages are important.

    Many of the messages have been misclassified, so part of this job would be to analyze the messages, and rank them according to how severe they are.

    I may eliminate some of these messages directly in the code.

    Some may need to be made more severe.

    There are hundreds of possible messages coming from each machine, and the users have no idea what all of them mean--as many are very cryptic.

    So part of the job will be making the messages more user friendly.

    And I will want to make notes on these items.

    Finally, I need a way to report on all the messages of a given severity or by which piece of hardware they originate--probably in a table.

    Essentially providing the users of the system with documentation (because they were given none from the developers)...

    I guess it gets complicated because the data I've collected may affect the source. the source is where I originally generated the data, and because every message is identified according to its position in the source code, that position could change if someone adds a single line of code to a file with multiple messages in it.

    We don't have a lot of DB experts around here... they're mostly engineers with very little experience in software architecture. I'd be in the same boat, but I've picked up most of my "good" habits from my love of Perl. it just kinda makes you think about doing things in a better way.

    I appreciate the feedback. I know I sound a bit distracted and scattered--and I admit the vague nature of this is somewhat more of a question of software architecture, than one of Perl--but like I say, the smartest minds I know in computers tend to be here. And Perl just makes things that much easier.

      Since you can edit the messages in the C source, I suggest adding a unique but easily distinguished error ID code to each one. For example:

      "Something bad happened. (err0r_aaaa)" "Oh crap - you're hosed. (err0r_aaab)" "The thingamajig dumped. (err0r_aaac)"
      Then a simple $ grep -rin err0r * in the top-level source directory will result in something like:
      some/dir/something.c:42: dump_err("The thingamajig dumped. (err +0r_aaac)"); some/thisthing.c:32: log("Something bad happened. (err0r_aaaa)"); thatthing.c:41: return "Oh crap - you're hosed. (err0r_aaab)";
      which you can split into path/filename, line#, and error message to update whatever you use to track the error messages.

      Bonus, whenever someone receives an error message you can use the (idcode) to determine the exact source of the message, even if different .c files have otherwise identical messages.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://928825]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (5)
As of 2018-06-21 01:35 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (117 votes). Check out past polls.