Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

Hi all,

currently i have doubt regarding how to merge two big files and 1 config files. the 1st files is around 20 Mils records and 2nd one is 400K and 1 config files around 50 records.

due to this, need your help whether we can use perl (using hash) for merging files or using sql database (pl sql) to make it faster.

logic :

1. get the 1st, 4th and 6th columns from file 1

2. compare the 4th and 6th columns with config files for the mapping

3. if the mapping is found from 2nd step, compare with file2 by :

a. comparing the 1st column on file1 and 2nd column on file2

b. comparing the 2nd column from config file and 1st column on file2

4. merge into new files by :

a. replacing the 4th column on file1 with 1st column on config file

b. replacing 6th column on file1 with 5th column on config file

c. add the 3rd & 4th column from file2

first files : customer, subscriber,cust_code,custplan,status,plan_rate,remaining_pla +n,quote 623192729079,510993192729079,19,22,0,90,282051608, 623192728769,510993192728769,19,22,0,210,84950715, 623192729901,510993192729901,19,70,0,82, 623192609007,510993192609007,19,70,0,85, 623416771429,510993416771429,19,1,0,95, 622319309157,510992319309157,19,22,0,30, 623192724581,510993192724581,19,75,0,4, 622319381619,510992319381619,19,70,0,5, 622198575655,510992198575655,19,1,0,5, 623192724589,510993192724589,19,70,0,6, 622743581281,510992743581281,19,71,0,8,
2nd files : prod, subscriber,startdate,expdate, SUPERG30^623192729079^20121225120308^20130124120308 SUPERM30^623192728769^20130105185053^20130204185053 SUPERB30^623192729901^20121209150053^20130108150053 SUPERC30^623192609007^20130102111208^20130201111208
config files : ccode prod, custplan plan_rate new_plan 5 SUPERG30 22 90 100_1d_unlim 10 SUPERM30 22 210 1g_15d_unlim 15 SUPERB30 70 82 700_30d_lim10 25 SUPERC30 70 85 1g_7d_lim5
expected results : customer, subscriber,cust_code,ccode,status,new_lan,remaining_plan,quo +te,startdate,expdate 623192729079,510993192729079,19,5,0,100_1d_unlim,282051608,20121225120 +308,20130124120308, 623192728769,510993192728769,19,10,0,1g_15d_unlim,84950715,20130105185 +053,20130204185053, 623192729901,510993192729901,19,15,0,700_30d_lim10,,20121209150053,201 +30108150053, 623192609007,510993192609007,19,25,0,1g_7d_lim5,,20130102111208,201302 +01111208, 623416771429,510993416771429,19,1,0,unlim,,,, 622319309157,510992319309157,19,22,0,unlim,,,, 623192724581,510993192724581,19,75,0,unlim,,,, 622319381619,510992319381619,19,70,0,unlim,,,, 622198575655,510992198575655,19,1,0,unlim,,,, 623192724589,510993192724589,19,70,0,unlim,,,, 622743581281,510992743581281,19,71,0,unlim,,,,
please advice about this

In reply to file merge with config(parameter) files by hyans.milis

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    [Discipulus]: somene can explain me Anonymous's irony?

    How do I use this? | Other CB clients
    Other Users?
    Others musing on the Monastery: (5)
    As of 2017-11-21 09:00 GMT
    Find Nodes?
      Voting Booth?
      In order to be able to say "I know Perl", you must have:

      Results (297 votes). Check out past polls.