Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re: Problem in creating process

by BrowserUk (Patriarch)
on Nov 23, 2015 at 10:21 UTC ( [id://1148380]=note: print w/replies, xml ) Need Help??


in reply to Problem in creating process

The trick to optimising IO-bound processes is to: a) serialise your disk usage; and b) overlap processing of one file with the reading of the next.

To devise an optimal strategy you need to know some basic information:

  1. How many files of each type? (Average numbers per run is fine.)
  2. How big are those files? (Again average sizes in bytes and lines would be most useful.)

With that information and a suitable strategy, it may be possible to reduce the overall processing time significantly.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^2: Problem in creating process
by ravi45722 (Pilgrim) on Nov 23, 2015 at 10:33 UTC

    Here is the data of one file. Always SMSBcastCDR is less. But we cant say how many files for each type. It's totally depend upon the traffic. But for yesterday It 92 files for SMSBcastCDR & 46 files for SMSCDR 

     "SMSCDR_POSTPAID_151106160000_10.xx.x.xx_RS10.log" [dos] 1182594L, 49
+7460662C


"SMSBcastCDR_6020151056461598_2015110615_10.xx.x.xx_Submit_uni_bihar.l
+og" [dos] 443437L, 72184936C

    [download]
[reply]
[d/l]

      How big (in records) are these files?

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

        Please consider this sample data. Because after ending of the day (23.59.00) only we get the complete data. 

        
-rw-r--r-- 1 root root  7866731 Nov  6 16:21 SMSBcastCDR_1202015106118
+1132_2015110615_10.64.5.80_Submit_Uni_upe2.log
-rw-r--r-- 1 root root  4750700 Nov  6 16:21 SMSBcastCDR_1202015106121
+6352_2015110615_10.64.5.80_DND_Uni_upe2.log
-rw-r--r-- 1 root root        0 Nov  6 16:21 SMSBcastCDR_1202015106121
+6352_2015110615_10.64.5.80_Mobile_Uni_upe2.log
-rw-r--r-- 1 root root 76461510 Nov  6 16:21 SMSBcastCDR_1252015105612
+2846_2015110615_10.64.5.67_Submit_uni_bihar1.log
-rw-r--r-- 1 root root        0 Nov  6 16:21 SMSBcastCDR_1252015105613
+18464_2015110615_10.64.5.67_Filter_uni_bihar1.log
-rw-r--r-- 1 root root  5841881 Nov  6 16:21 SMSBcastCDR_1252015105613
+18464_2015110615_10.64.5.67_DND_uni_bihar1.log
-rw-r--r-- 1 root root     1557 Nov  6 16:21 SMSBcastCDR_1252015105613
+18464_2015110615_10.64.5.67_FloodControl_uni_bihar1.log
-rw-r--r-- 1 root root        0 Nov  6 16:21 SMSBcastCDR_1252015105613
+18464_2015110615_10.64.5.67_Mobile_uni_bihar1.log
-rw-r--r-- 1 root root 77645323 Nov  6 16:21 SMSBcastCDR_1312015105529
+24142_2015110615_10.64.5.80_Submit_uni_upe3.log
-rw-r--r-- 1 root root        0 Nov  6 16:21 SMSBcastCDR_1312015105529
+56928_2015110615_10.64.5.80_Filter_uni_upe3.log
-rw-r--r-- 1 root root  3790221 Nov  6 16:21 SMSBcastCDR_1312015105529
+56928_2015110615_10.64.5.80_DND_uni_upe3.log
-rw-r--r-- 1 root root        0 Nov  6 16:21 SMSBcastCDR_1312015105529
+56928_2015110615_10.64.5.80_Mobile_uni_upe3.log
-rw-r--r-- 1 root root 32571126 Nov  6 16:21 SMSBcastCDR_1422015105591
+3149_2015110615_10.64.5.67_Submit_uni_upw.log
-rw-r--r-- 1 root root  9525488 Nov  6 16:21 SMSBcastCDR_1422015106013
+30764_2015110615_10.64.5.67_Submit_uni_upw.log
-rw-r--r-- 1 root root  1516225 Nov  6 16:21 SMSBcastCDR_1422015106069
+518_2015110615_10.64.5.67_DND_uni_upw.log
-rw-r--r-- 1 root root        0 Nov  6 16:21 SMSBcastCDR_1422015106069
+518_2015110615_10.64.5.67_Mobile_uni_upw.log
-rw-r--r-- 1 root root  6995588 Nov  6 16:21 SMSBcastCDR_1422015106069
+518_2015110615_10.64.5.67_FloodControl_uni_upw.log
-rw-r--r-- 1 root root  1534798 Nov  6 16:21 SMSBcastCDR_1422015106117
+41776_2015110615_10.64.5.67_DND_uni_upw.log
-rw-r--r-- 1 root root 16925004 Nov  6 16:21 SMSBcastCDR_1422015106117
+41776_2015110615_10.64.5.67_FloodControl_uni_upw.log
-rw-r--r-- 1 root root        0 Nov  6 16:21 SMSBcastCDR_1422015106117
+41776_2015110615_10.64.5.67_Mobile_uni_upw.log
-rw-r--r-- 1 root root        0 Nov  6 16:21 SMSBcastCDR_1432015106819
+54593_2015110615_10.64.5.67_Filter_Uni_upw2.log
-rw-r--r-- 1 root root  3240265 Nov  6 16:21 SMSBcastCDR_1432015106819
+54593_2015110615_10.64.5.67_DND_Uni_upw2.log
-rw-r--r-- 1 root root        0 Nov  6 16:21 SMSBcastCDR_1432015106819
+54593_2015110615_10.64.5.67_Mobile_Uni_upw2.log
-rw-r--r-- 1 root root 41856138 Nov  6 16:21 SMSBcastCDR_1432015106819
+54593_2015110615_10.64.5.67_FloodControl_Uni_upw2.log
-rw-r--r-- 1 root root 67635181 Nov  6 16:21 SMSBcastCDR_1432015106848
+11991_2015110615_10.64.5.67_Submit_Uni_upw2.log
-rw-r--r-- 1 root root 41543266 Nov  6 16:21 SMSBcastCDR_1612015106127
+19882_2015110615_10.64.5.80_Submit_uni_upe5.log
-rw-r--r-- 1 root root  2225808 Nov  6 16:21 SMSBcastCDR_1612015106128
+40925_2015110615_10.64.5.80_Submit_uni_upe5.log
-rw-r--r-- 1 root root 19191619 Nov  6 16:21 SMSBcastCDR_1612015106129
+36767_2015110615_10.64.5.80_Submit_uni_upe5.log

        [download]
        
-rw-r--r-- 1 root root 484574875 Nov  6 17:35 SMSCDR_POSTPAID_15110615
+0000_10.64.5.102_RS4.log
-rw-r--r-- 1 root root 482043589 Nov  6 17:35 SMSCDR_POSTPAID_15110615
+0000_10.64.5.73_RS11.log
-rw-r--r-- 1 root root 481948282 Nov  6 17:35 SMSCDR_POSTPAID_15110615
+0000_10.64.5.78_RS12.log
-rw-r--r-- 1 root root 552695748 Nov  6 17:36 SMSCDR_POSTPAID_15110615
+0000_10.64.5.85_RS1.log

        [download]
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?

www.com | www.net | www.org
Node Status?
node history
Node Type: note [id://1148380]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? • Last hour • Other CB clients
Other Users?
Others avoiding work at the Monastery: (8)
As of 2024-04-19 09:44 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found