Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: Perl script is getting stuck for no reason

by davido (Cardinal)
on Jul 31, 2014 at 17:17 UTC ( [id://1095778]=note: print w/replies, xml ) Need Help??


in reply to Perl script is getting stuck for no reason

"Perl script is getting stuck for no reason"

There's always a reason. An apparent lack of reason is an indication of inadequate investigation.

"...it is getting stuck at random..."

That is almost impossible; computers are useful because they are deterministic. Of course you're probably aware that computers don't behave randomly, or you wouldn't be asking for assistance with the behavior you're seeing. But the sooner you abolish all notion that something is happening randomly, the sooner you'll be on your way to diagnosing the problem yourself.

The code you posted doesn't compile. But you're not asking about why it doesn't compile, you're asking why your code seems to hang. So that tells me that the code you provided for us to look at is not exactly the code you're running. That makes it difficult for us to know what's wrong.

Fixing the right curly bracket, I then get three warnings. Have you investigated why you're getting warnings? Are you getting warnings, or is that something that only happens in the code you provided us?

We can't debug this for you; we don't have the data, or the real code. But let me offer these suggestions, which with a little diligence, will probably give you an answer. For what it's worth, these are steps I would probably take.

  • Add "warn" statements that tell you where you are within your code, and where you are within loops.
  • Run this with "top" going in another terminal. Watch for memory growth. The "warn" statements previously mentioned will help you to see what parts of your script are consuming memory, as you follow the runtime with top.
  • Test with a much smaller subset of your data sets. This will allow for quicker iterations of the debug/test cycle.
  • Even with your much smaller subset, don't be satisfied with any substantial growth in memory usage as the script progresses. Keep in mind that you are dealing with very large files. If it becomes too costly to hold intermediate data in memory, put intermediate results into files or a database, and process those files rather than trying to hold the entire computation in memory at once.

If it seems like I'm talking a lot about memory, it's because without a running script and small test data set, I can only go with instinct, and my instinct is that when I see some data structures persisting throughout your script, and read that you're dealing with 800MB, 16GB, and 500MB files, and when I read that your complaint is the script grinding to a halt, you are consuming too much memory and bringing your system to its knees.


Dave

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1095778]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others musing on the Monastery: (5)
As of 2024-03-19 09:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found