Handling japanese file & comparison

[CSharma]

Hi PerlMonks,

Information:
1. A Japanese compressed file(tab separated), Say A having products information (iphone etc), columns: name, description, image, uniqueid, rid1, rid2, url
2. Another file, say B having mid, rid, ntid (column separated).
3. rid1, rid2 => A & rid => B are numerical values.

What I want:
If rid from B matches rid1(from A), get that line form A into third file, Say C. In case, rid is null then rid(from B) to be matched with rid2 (from A). If rid1 & rid2 don't have values, leave that line.
Also I've to add some values to the url (from A) & then it's to be pushed to 3rd file C after rid comparison.

The main problem I've are:
1. How to handle the japanese? Do I have to use any modules? as saving the japanese character to a perl variable turn to junk.
2. There are 30 files like file A, each having 1 million of lines in them. What would be the efficient way to do this in perl?


Help is much appreciated!

csharma

[perlron]

Items i dont think you're gonna find via this post on PerlMonks.
• someone who is conversant with the Japanese script who will evaluate your data file. I would suggest you try with a local Perl Mongers group like Tokyo.pm
• a decent perl developer to help you write the code, if you say there are 30 files, with millions of lines in each.
Items for your reference..
• You have to take the support of CPAN modules like Unicode::Japanese.
• A related query posted earlier. To convert the unicode entity (症) to japanese character 在
• I'm sure monks can point you in the right direction, but for that can you please provide a more accurate dataset in your Original Post, along with some code (or pseudo code/design if you are new)

[graff]

Do you know what character encoding was used to write the "Japanese compressed file"? Is it Unicode (utf8, utf16)? (Shift) JIS? EUC? Something else?

If you don't know how to answer that question, your first task is to figure out how to answer that question. If you don't have some reliable reference or documentation for your data that answers the question, use Encode::Guess for that. Try it, and if you can't figure it out, show us what you tried.

As for the rest, it sounds like you want to read file B first, store its "rid" values as hash keys (and if there's anything else you need from B in your output, store that as hash values).

Then read each A file in turn, and output whatever you want when there's a suitable match to one of the hash keys from file B. Your description is not clear, but I gather that the logic would be something like:

use strict;

my %B_rids;

# ... open file B, read it to fill %B_rids hash, then: 

my @A_files = ... # do something to get list of A file paths/names

for my $A_file ( @A_files ) {
    unless ( open( my $afh, "<", $A_file )) {
              # update: you'll add encoding on 2nd arg, e.g. "<:utf8"
        warn "Open failed for $A_file: $!\n";
        next;
    } 
    while (<$afh>) {
        chomp;
        my @fields = split /\t/;
        if ( exists( $B_rids{ $fields[4] } )) { # does rid1 exist in B
+?
            # do something
        }
        elsif ( exists( $B_rids{ $fields[5] } )) { # does rid2 exist i
+n B?
            # do something else
        }
    }
}

(updated to fix typo in list of encodings)

[CSharma]

Thanks a lot graff for the help!

1. file encoding is : utf-8
2. How to open zipped file with encoding?

I've written my code, review it & let me know my mistakes.
1. I've read cat file into %catHash
2. opened the main file from which I've to extract line if exists in cat file.
3. if $line17 doesn't have value, $line14 value to be used; if both don't have, then read next line. Am I right below?
4. $line17 & $line14 in main file, some have leading 0's but not in that value in cat file. for example: cat file value: 101010100 & main file has: 0101010100
What do you suggest to handle this situation. I've tried replacing these leading 0's.

I'm new to perl & programming both, please help.



#!/usr/bin/perl

use strict;

my $f = "vc_20151003_1.zip";
my $cat = "5426085.csv";
my $file = $f;
my $file =~ s/.zip/.csv/;

## Read the cat file into hash
%catHash = ();
open(CAT, $cat) ||      die "$cat couldn't be opened\n";
while(<CAT>)    {
        @line = split /,/;
        $catHash{$line[1]}++;
}
close(CAT);

open(OUT, ">".$file)    ||      "$file couln't be opened for write\n";
open(IN,sprintf("zcat %s |",  $f)) || die "Could not open pipe for $f 
+: $!";

while(<IN>)     {
        chomp;
        my @line = split /\t/;
        $line[17] =~ s/^0+//;
        $line[14] =~ s/^0+//;

        if(exists($catHash{$line[17]}) || exists($catHash{$line[14]}))
+  {
                print OUT join("\t", @line) . "\n";
        }
        else    { next; }
}
close(IN);
close(OUT);

[CSharma]

Well, I've done this myself.
Thanks for help!

~Csharma