calling awk one liner from perl

devi has asked for the wisdom of the Perl Monks concerning the following question:

I have large data set that has to be processed. when i tried with small data set, awk one liner seem to work. however i have a problem with memory issue and it is getting killed. How to include it in perl script for qsub?

Comment on calling awk one liner from perl

Replies are listed 'Best First'.
Re: calling awk one liner from perl by igelkott (Priest) on Apr 22, 2013 at 08:23 UTC
Consider using a2p to help "upgrade" your awk command to Perl. Actually, I'd convert this by hand but `a2p` might help you get a first approximation.	[reply] [d/l]
Re: calling awk one liner from perl by 2teez (Vicar) on Apr 22, 2013 at 07:12 UTC
why would you want to call 'awk' one liner in perl? Have you tried solving the problem in perl at all? However, if you insist, you can check perl function system and it's several cousins... If you tell me, I'll forget. If you show me, I'll remember. if you involve me, I'll understand. --- Author unknown to me	[reply]
Re: calling awk one liner from perl by Anonymous Monk on Apr 22, 2013 at 08:34 UTC
on awk and converting it to perl automatically grep lines from log for last 10 mints (calling awk from perl, awk to perl, a2p) get value from a awk system command Perl equivlanet of simple awk perl awk gawk to perl Script to manipulate data is not giving me any output Awk and Perl unique features Awk usage in perl Convert from awk to perl UNIX 'command' equivalents in Perl sed awk to perl help Multidimensional Hash implementation/usage trying to convert from awk to perl using variable in search and replace Extracting data from each line that matches a email address from a Log file (Tab delimited) awk like question in perl awk...ward awk/perl? Selective substitution: not in Perl? AWK and Perl on UNIX chomp a newline in awk After looking at awk a second time, I don't think I'd ever use it :) I'd stick with perl, see perlintro Tutorials: Pattern Matching, Regular Expressions, and Parsing perlretut perlrequick split perlfaq6 - Regular Expressions FMTYEWTK About Mass Edits In Perl Uncommon* but Useful Perl Command Line Options for One-liners Re^2: perl -pi -e s'/^\s+//'g $file Re: perl -pi -e s'/^\s+//'g $file Re^2: perl -pi -e s'/^\s+//'g $file Re^3: perl -pi -e s'/^\s+//'g $file Re^4: perl -pi -e s'/^\s+//'g $file Beginning Perl (free) Chapter 6: Files and Data Modern Perl: Chapter 9: Managing Real Programs > Files use autodie open strict warnings perlintro perlretut perlrequick Beginning Perl (free) Chapter 6: Files and Data Modern Perl: Chapter 9: Managing Real Programs > Files	[reply]
Re: calling awk one liner from perl by hdb (Monsignor) on Apr 22, 2013 at 07:30 UTC
If your awk command runs out of memory, calling it from Perl does not help as it will run out of memory as well. You probably need to translate it into Perl and then subdivide it in smaller, less memory-intense pieces. As always, people will be helpful, if you provide details and show what you have tried so far.	[reply]
Re: calling awk one liner from perl by Rahul6990 (Beadle) on Apr 22, 2013 at 07:12 UTC
Hi vroom, sed and awk cannot handle large amount of data, because they try to load the entire file at once into the memory and then apply the changes on them. Even if you call a awk command from perl script same thing happen. But here is an example that you can try: `$command = <awk command>; system($command);` [download]	[reply] [d/l]
Re^2: calling awk one liner from perl by runrig (Abbot) on Apr 22, 2013 at 15:33 UTC
... because they try to load the entire file at once into the memory and then apply the changes on them. No, they don't. Perhaps you're saving too much data into an associative array? Anyway, that's enough guessing games w/o any actual examples.	[reply]
Re^2: calling awk one liner from perl by hippo (Bishop) on Apr 22, 2013 at 13:51 UTC
sed and awk cannot handle large amount of data, because they try to load the entire file at once into the memory and then apply the changes on them That is quite some assertion. I'd be interested (and very surprised) to see any evidence which backs this up.	[reply]
Re^2: calling awk one liner from perl by Anonymous Monk on Jan 09, 2015 at 13:09 UTC
... sed and awk ... load the entire file No. Quite the converse. You talk about "sed" but don't know it is short for STREAM editor ... RTFM.	[reply]

Back to Seekers of Perl Wisdom