Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Arabic Encodding Problem

by malak (Initiate)
on Feb 09, 2014 at 11:59 UTC ( #1074106=perlquestion: print w/ replies, xml ) Need Help??
malak has asked for the wisdom of the Perl Monks concerning the following question:

Hello, I need to extract Arabic text from HTML files. However, some files worked well, while the others I get non-readable characters like "العام". Here, is my code:

#!C:/STRAWB~1/perl/bin/perl.exe -w use Data::Dumper; use HTML::TreeBuilder; use HTML::TreeBuilder::XPath; use utf8; #------------------------------------------ if (!$ARGV[0] || !$ARGV[1]) { print "\n\n htmlParser-V.1.0.pl: Error: missing input and/or outpu +t directories.\n\n"; print "Example of use: htmlParser-V.1.0.pl <input-path> <output-pa +th>\n\n"; exit; } my $inputpath = $ARGV[0]; my $outputpath = $ARGV[1]; #================# Read the files from the inputpath opendir(DIR, $inputpath) || die "\n - ERROR: Can't open the input path +: '$inputpath' !!"; my @files = grep { -f "$inputpath/$_" } readdir(DIR); closedir DIR; #================# First, check if outputpath exists. If not, we creat +e it. if (not -e $outputpath){ mkdir $outputpath; print "\n - Output path created!"; } #===================================================================== +=====# foreach $file (@files){ open (FILE, '<:encoding(UTF-8)', "$inputpath/$file") || die "can't +openfile $file: $!"; print "\n - Scanning file '$file' ... "; my $root = HTML::TreeBuilder->new_from_file("$inputpath/$file"); $root->parse("$inputpath/$file"); #=======First Attribute======== my @div_class = $root->find_by_attribute("id","maincontent"); #_by_attribute("id","ctl00_ContentPlaceHolder1_FormView1_Label +2"); foreach my $node (@div_class) { open (OUT,">>$outputpath/Bank\-$inputpath\-$file\.txt") || die +("\n - ERROR: Can't create the output file !!"); print OUT $node->as_text, "\n"; print OUT "\n"; }###foreach }#foreach print "OK"; close FILE; close OUT;

Comment on Arabic Encodding Problem
Download Code
Re: Arabic Encodding Problem
by moritz (Cardinal) on Feb 09, 2014 at 12:54 UTC

    There are two problems with your code. One is that you decode UTF-8 at the script/input level (with use utf8; and open with :encoding(UTF-8)), but you don't encode at the output level. A

    binmode STDOUT, ':encoding(UTF-8)';

    should help. read more.

    The second (potential) problems is that you open all files as UTF-8, but if some of them aren't actually UTF-8 encoded, you'll get Mojibake.

    Before you decode a file as UTF-8, you need to find out its character encoding. If you have no additional meta data that can help you find out the character encoding, you can look for clues inside the document, or use something like Encode::Guess to auto-detect the character encoding. (But beware that these methods are also error-prone).

      Thank you for replying, in fact all the files I can open them in IE using utf-8, but in the same script there is nothing about the encoding. I have used Encode::Guess but still not solving the problem. On the other hand I used UTF-8 at the output layer but in this case all files will be unreadable. I appreciate any help.

Re: Arabic Encodding Problem
by graff (Chancellor) on Feb 10, 2014 at 01:51 UTC
    It's strange that you use open on each file in your list, but then you pass the file path string to HTML::TreeBuilder->new_from_file(), which causes the module to open the file again - probably without the ":encoding(UTF-8)" option.

    Have you tried calling "new_from_file" with the file handle (FILE - which probably should be changed into a my lexical variable) instead of the path string? That might make a difference.

    If that doesn't help, maybe you could post some sample HTML data that demonstrates the problem...

      This is very good point. From HTML::TreeBuilder documentation:

      When you pass a filename to parse_file, HTML::Parser opens it in binary mode, which means it's interpreted as Latin-1 (ISO-8859-1). If the file is in another encoding, like UTF-8 or UTF-16, this will not do the right thing.

        The point is some files managed well and can be presented as normal Arabic text. So I guess it is not the HTML::Parser problem. Any way do you recommend any solution to test?

      Thanks graff for replying, I attached HTML code of the file caused the problem. Thanks in advance.

      <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <html> <head> <title> - : </title> <LINK REL="SHORTCUT ICON" HREF="/favicon.ico"> <meta name="keywords" content=" "> <meta name="description" content=" "> <link rel="stylesheet" href="/styles/ab_templatear.css" type=" +text/css" media="screen"> <link rel="stylesheet" href="/styles/navigationar.css" type="t +ext/css" media="screen"> <script language="javascript" src="/scripts/scripts.js"></scri +pt> </head> <body> <form name="Form1" method="post" action="/ar/waysbankarabiguid +e.aspx" onsubmit="ValidatorOnSubmit();" id="Form1"> <input type="hidden" name="__VIEWSTATE" value="4/DMyr6D5wLBkibw80jmu6v +jhMP8MbvT9MWqqebu3DKgSm8p1B4rt+uhnOxQYCCxHTTycouuC+2qaahX76bwKAwaDIoe +6DGTFXqrQcYDW/c7r9nriohJfy6tiBWC59nvWpwu6FEM2RhoY7/PrvyJSdIBZaogxCw8l +jOKD0vi7TkXUOLAFjE+kBXow0do5KrACrT6CUanTUTUjaCEEWTTgYRn78ic2k2ToiXD4+ +AWjelzFuNSkFdes7CNjfkzp/MQkfpWQVtGUlQvYGFD2eztnm3ThD2m/+kjvhW1lC1n4kq +LdHJUgr50FWaw/uurYhA+mcklINgrxe5WdaeWC+fh/owUOEDca937KWY9TgHoJGRzMEK1 +rZhYrEggkcf+r0n6/e13fd8MFCeS4TBlFVYg3hNAklXP0obam7cSKay/ZB333N5T/hgWo +OXWRkms2H80+6SFqIjGaQ2Tv3KIVNphiImW+7u3vqV0yHiniFOizstlwi3G/6kcZRj6Tg +GGDiYWpzHmOza/CpeU2rpbLC9/b5PryvVYhww/d7Kc4r0IzDjk1Z7eiskzqQj5JgWdwQs +jFq0Q1xENYG5JZV85tsfCNqsIBH4y40Tgt/BVmw1uC3fFQJiJjgnLl5u3t/g91kXsgvNr +9Vj3PdT4YBBJZzZKZmENsCJ8ZnW8QRyKtz7+jhaAjWFbJw0qfCacF4l4MpRTtHBdDUstn +ZRUCuXAyiLmblr8MBFhSFaPbazyR/3Fyil5i3B/RiGeGsO/hUN3wq7veqkU/ugu4sgBd6 +HFW7ElLqpTULpvVJybVIdNYs+Fv4eZmfOQFEUGMfJ8nE6SsrR4lOD8D63cnW+nedO+AKN +kEcGT14PupHMtrmxPFqkJzgsbaTX/dIVT0z3KslMNjEOQ7LQPFJyy72D9ALBd10LO0WAK +Qk5ju/DP55+Tk8G+oevh/F8Af3POe+sGedUVIr0og68yZAOt4GJYd9wLceRiMo9AK2mSA +yRHF1n/eSZbSVywZdSEavCQiGqXok/PMCx309tz6C/kZglqmnfmd24AKNiHDhOYHGyfvD +77+jorq3T3QrX50e35fKFQPhKtMPl9LDfTdSrpby8+AetyW2WB8Kn/9fobasL+JWiZwUZ +J+z4uwcWMIitLl495ty8oH8BT4lh1pL/wkApFGgH5TT535s7rDWJGxd57SsABuFfe8zn4 +z1pCdUyA5RJvnNTCTrJ3HQ4/ORSXB7oCDGbmDKcBj5cgDiwKj8mT4Od8lWZKDMGj762Rs +52bOpVx84OM2SUMQwA8uBLq5FPWAS5hH9GzwXwJ+R3K5CS/GiMFkWyhNkUqL1zFRizZh+ +bY6IqWxO4U6PK1OVCwXiNNiia4EwXc2x/bd0vqgQxXvSUvZrqNzaTSnmJrsjmwrqNOh64 +XwREMkn/kfIKuC+kzxBLQtHv0FiSdt1cteSnHn6Ps+0UwE1OKP5UEh0HpwN/blfh1IW7s +b851OdhbzqNmRlEhgbzSbDq7eX4AINPTDUkfXW5vCUHsVk/L1NaVfYiZnqlcIYzePWI/V +b3rXa7c2GezMV52T3arzzOrJ3EVibs4xJzzTN+4+CLZQt/QhgOyDJFKEUCNIHEJpzSX+j +z+qorBAKAd+Po2Q+YiKEKkz0+8vMq8k/wFitB2aqOw6QM0Tr8Ga9v/NYcnbNVJQ6m2Nhb +vyJBjk3krD1us0CljKvEihor7hP+RzUeCQiVIaXDytoXM71Zm1y0L846JIbQI337H2sxx +Rqncm0606TU/56d6O4ztbuVYGBUPvgvIq4qaRsbb0OD5XHGZUlsKF+ICsZDOmai08fqR0 +Rys+e9zBYyF3ZV8yy6Agf5o+6qNrX6R5YEAgVgKaP4EPdLOOu89Yg772sIrJ6Mo4HsAru +G+kSXcZlHYC3EGJr+nVQaUNAGpRUTX+kR3o21NyxX3jXvLxDmrPlu8m32Mm/Cu/qSP/SV +Kt0qHi86/LyyYEXB/zeSpqLL/Kt6O1dYCKqRtjKBBv2qSWN+G7Hvhh8+47TMWaAmbY5UR +zl7E0MOV0nbyIYTUUR7dyaSri4c020HPJCX4GmcRDJukzAkvK9fMxjiW3LFzn3eoJmfJq +G7aM51Sx86tIVdiauqRuIrCWJ/y7jmQ4guFUCNSkcB8YbuatrbfXUmss/VZld0c67zkhf +h2oh12eAd4Y98QdfzVbvFdCPzRQA9prYnEUiIuBBSGd8qQ7dkP3RE4HATmq0TAshtgqvH +L/EPk3jDtYEqoyo9kNpKjuQUGjjKbwePK+ibxgOZiJ3a3LQBA+bLDLmUuM5M6EPp/L61C +OjrztbO33vhvUWheYfJF38qUo5NF9rFOZUtMkrixfAlv6W6YKXZdFWVnzJKoV+z0VvzFN +jBnRuPk3WshHWckrAPEvwmOyO+gZYqi0aLoM4A7+c85pp7++tTbDrV+AhmZFMSNqlDVLr +ClWpGDFo9BDbNiNTcW7iCxZJLl8lKX6TWqa8V63j2lpW8qGubNGZRUq94juxQTvjf+smj +79FwJe+9sbrx+/o5WjlnsxVGTp2vrq2uVJgqxmmIlrFWYSiPIWXja8i2CCiCsVcM8Toj+ +VLnsozE3ZgUQwV6+vGHntLRw6z3kq4V/IRJ0Pz8+lxZ1jAEU4b2gPtTPC6yaV0AIncbz0 +0GwWxaKOZ31J5UZ+ioLfLxphKLiyPplYwHD76rbWgdgqN+oWRzGZGeITJKCCdAr7IvIWE +31UNLpnlMUN1KgclPteae1F9pMlaeSUYFezpuvlfadwTyYqTLC4fRv/gqog58Ox68BO8n +/Q9FWkDawzyem6jlY0F1Voo4X1tHikPQEjN0HyYmKbOkGUaP4JoAVKi1PlIIuQf7k1hCr +w2LxYcj9YV91ppvm7dLOdkK1GAh5AdpNxb5/oAL7MdrIKm+qhHMpzGi8iJaGaOs2RM0pO +sDzWZkh3r723831XhrLBdtZ5rORNXh9wAQbht+kjqEpgIV9v/JrpL5tWicZbwKiAHHwpv +4yC/A21bltDAwuiOeQU+gB437eczU6Urp8B2RK3PWnF7eGcPTJzI66F/3KtfdVAd3M/VB +l2IRlPR8+g9RaZa4dsA3qcjd8z4iE52ad+GhqOCfvCnTNF9hfAF0/d8hgi9l2EsfZP5+9 +Cw6Fwks2x1IH4DYjXurKFdeUYlaXMf6XY4aFZXwiDj/elsI4iCjktXg4Mi4QJJ0Eo6o6i +Fp4actaPifPHBJJAe+QpSA5hqWF70HfhSaDvHW4Vp3pxFCilBUPQoXmdB/KnKQwfNgEMG +X9aLajDCdcHCHackryPUn3Pkf2VrRxJXTIt7ecSa342n7RMzGuTs9x7yve3fNKiMZZHKv +66zNN3DS4L/lbzrV0dote5VxL7VA4GlWTdQu96xpB2n4eOD7UdzFuxB1TWxCwdaB6UpJ6 +vvZJUNGc9mOCnJ1KybbixvyMnbXceuTgRfeOIX3/eshBnlRvw6vd2tsXAIL2XdfjY5jeK +Oe1lIod7iQiE5nDLgOf6ubCG5ZZmunIGaQvfiKmPchCIKMENV928U9YYgq7S+0iiDM1Pm +jJKpJEtvXmVXrRhTu80FLqrN6JUDvlisdtN0sOJJVyS7gcpHZbFVJ03WWByAxmMF3/jd1 +uzDFaz9yDlOk7a8OQKRo9w5bgGByfpeqhyLlNsrnZ6ym/zWx3QSeb+IJIeJJ0NM/G2I25 +Hsz8UDLiSrVSJfxAxdiYcQqDB+cbEqic8BFDDqqgpRoBoMV4DBCCOpAeFdQYAzfUxgQ19 +Gbn8C+Db7XV7pVfuV+0PDttEZfmeRCCyVyo2+T84AamQFg2bBR4AHSKNB+bOQ17Vt3r/B +oBXA4PdIDFW1lllLjmD5DfKd86UtvKeiBEYeWjZFxM7qNGIXheYkac/FIHr04Mk85WrAZ +INZnZZhnlJXu+iQB7TYKIKrRbOI2ejy4J1N+c1PyGIpatispMRa+Mp71NUdMujWTlHBLB +sY68OSmlNeF8thn7DWm6ao7hICnYWETv9SZJhQYaU+drrVOndpmRRrCcpXt/0UtUe7gu9 +bESAtHJu2nvdfUyfUCb4Xao08aI8b6fqU+zFoZ6KGnMwDhfexyYrjMJ4SewkSmC7oZBIQ +X3u/sdeJ50yYClTnShuNxW1QM/XhRtoiyL/a5U2Fir0pGdGJGfb8Rp70XcvL/SeVJDJKY +5EFn0Sw/ZYQoqqWbQCToTobSr/JHZ7mb8Zi38k/O1YcbbjX8uefKfRILglTAvowFmrw6y +fa2261rINnxWPC7gEYJQbu9UGPE2EQYCcb6kzVzKABnz3gdD+vwfoaZJ+daLshh9Dab8z +4vU7e9TovRdEGvf4I+XDA3NPkJeKXVlhL5dagoMD6UwTi0FyXS+tCgqu+/lWDAEg++2ft +WxtGW8ZS8LhhsYU/j2iURvqKPu8MA85spVeG4Dl5PRqdi5IYURH/l24fEJ215FReo122M +cN5X+8U5KR1csPTm5upgAiK1t8x5pa16dclzd+CwVFoFKPyvdqtwPVGgZw60vYCRM2Vlb +Cc/vGrK+lhcr6E6BG6gd7YhKdRekWjxxG9A6wMQU0YlqVD+my9yRJ4RhTTD+okMy2XYos +WB7hK9MCudTPUE9tjJpeNSX/mY98ZTwi/bE5weHDUmqzv/m2bv5qkr6RoSxqdZjFjAi+F +I28RHIMeegoItgGAtBr+dpXsVF0s3eDmgA/NCVdY9UY3VtlQ0cMPn1gldTgV1/UQ7Va6M +cdiGW2HU8GuOqUndz95k+w2f8p/g/O090jcE5H+ktpa8BucCGix0nVjKafdxr9YuTz/PG +zpGuO76vxIfHCG2l6yr47KNYmmrQs/FGe3qshDnpMETvyi/SuwxP4k8Pq74t/zDyvYQFW +z3FIxZgA1kIK0dItkGm3RTr7dg0eQOU8vFuHjKZXHnIu4DsKjzRTGP7D3Av0xYITtIeqP +Q7JV0wsR25eNJtla0GlvUxqHwK/9P6zW86YcM8RH+ZF2AOVm6qv7dDmfKX/SJyl+cyDfJ +EdYGXPsw3yfSqTu2Wy3aDWO2Wksz1gj50NwurvSBydufvl6XW7thjkeuI9jXsqtYK/oHx +5hjbehhESYBJ6mAFiVdCy8hHIs9EU6LJTfP1GlVuh0j56Egc42qJiAbNh3FGC6rG9o2yo +AVFJwgm+NdPV4EMn4Ps4Ij70LmFTUZLJNrxHo17hAvhN9tNtx9ytQ0kQmUxsO4hH2KP4j +9nNkabrj/AEIFdCV4fD9sumT+MJiVSXlJAOfiQw0tIqYcK7/HZFJOex5mZRhjuJnmeZu7 +S993qaIfgOGR/BKOF/hAQnhTUwWrYYQxPAV9CnQPknUo8+MDuVC9eG7t/N12DXw1qGEzD +hFdBxll8l/1ZBYAMGx/KTF7crOiRxoHOgfpWFMlcRpT1pn+Ii1Dfxzp1+QgyfWGTbj4qH +48RUMZDx7Bv4WSzJzxts2A0v2i1N1l06OfEwMi+tPpwcdfJgkmaft1/QhE81+9eLj0nNU +SvE+A2+lSvTllUqOifSX/RHWm16TdmnRTPWDgo0Js3+QcdjXVli9gPckxRX06Fi6I68U9 +3iL1klTQvdmy56BZc1nTYqzVKnKRuH9MCAvJq8l6z+FvqlAP7Yl88yELyiNDo1ojyL5Xg +hFkJgcyfwyr8VkHSXrIeZahclICOfhOB77baV4+X5cr809G3Y9Dk+VvUOfln4MiuIjbCe +Roi2gXZ2rPTImMlQMx8jjRM8Nt/WfpQnBEROj3XLHQZ6ExvKi8ai/XFH1U/xJS+/7C8q1 +BLrxNti18BDZYGlhj1iQXaiI2wcpmHhjEW8wRow005siVENp9oKGXvzm1qpD6Hy1usQDg +fjx+mRugKt5hX0IjC21ElL4Dswzk2b4kBVmniTqK3GMgeJJk1ZBnzXjj9xkuhoItaINkp +cwX/YbO41pfju6EjvF1cLQESV0OoQy5LNskxALmfmuYDuj0vuufQ/hXFAocVAZqiD+SXc +7JoF4tszNljgZ1tSimwUibJmh4BnDatQL8ptcPmB2ZE3ZTowasihS1jecbWShpouzh51Y +YbjPZgTeaRIuQPdTfORlYA56B/yVinuyaryXMtoEpSpIp4I8Zr/n0Gqrd0aOCnjz/BDaR +IpFYGq4ui9mxPosvEiB1+NQBRzJfW3+Fpv2IL1JexQwfVclBzdHKSAXowLU/Ekjlw31WW +jYycXXMGMMESCVyoEcsV1EiP81sD6p//Rg9GODJCXGfKJCHlfRfSxEBNtlku8En+mxtF4 +z3u1FBUKMKpS8Yf4slNgen1Z1tT0dW4k5BzSoMWCcI6LCjQ7h7FIbViNJBzaj40mDZTVX +7xrcmqfsbowWHTbyr7gYAvWoWm+qaui8I2dYwKoST/YGNTa/5w8T4pw1agRokRFCA05Dz +JVPPbweOYamjBZOBHlIjaBkLVCvO+MMYOzP3c6OhyMoPXCtt8RD1PRTb2+WVLjp2bzjot +dsw9a1rsMU+/0BUq/6Fs8h36133RyTfxgCWh6lXFQBrUIPzFKwmMUbHOLsTT8pBTmF2un +U2xAqSxgCHyycuK0sX0O5bInYrCNZr878vzdiFvrqdX7cjnmWd1j0p3S/fTcSYSaABvFU +pb1aFV6r2mI8nEvG6sKzrh86KBTVoUTOXMZaRYGDhD/1Q/AS6wARwZ70ekZsgZ8TeOesl +0bWTaPSM6E0MiNMD+DQ3d+9RofNrjoueN2Wurfs1WW4DTwKve1mg3BcYqukYlzCHtGxe9 +DcXO2OB4TEyTZqo47ypPykw1zOtKdiHxdTh1lMHbBGosWEzgYLc7UjJkStUPAgozjedOq +rrWzE4H21xqYw1A4lx+K8KCnNQSYDS6R/P1yDX48TgJD60wOoXrBCMO1xzAUKpY6J+0GG +RNwW7o11sKb6Kc6iceBYPrOS45JHK1gyNwcS6LA0XZFBq2x6IhxIDqQmk1ovBSBvczH8F +T/DysbM4Abu0N0r+Ex4pdju5ViekR9JP5HX6qG/eHo3NP7CE8rw/cMoI682CYbvOid5Bj +2j9vhwgvE1V/XmqUFqPmXLH+3KJ8I7utPgUmgtMj3OOPm8Dm0mp+GYizMhZF+KVWvoyFO +WwymyZ9Of1iICix5ELr6wpkM8j4GWrrlPQHTaE5cCUjSeXMom6IqRj2AWTnTCo11SBGS2 +6OL+7PT6jLT4iz6hZy4ZQCLsNfahjTrQT71uU/rpTNhnCiM5+uJ1AOLVQRDTZNnuw8I/o +WpcLqPlA4t4L/zLAUng8wOTyNIohYUM+YQGoLtdH9t9SJGQlTn3qQv/IlclRMtswWTKeF +9ZFiCxwyCHX4F5YR1uAyzD6TPzEt+BYonVei8cwOSegvaQpIyiBFs4yQgS65L72WVKuAw +kMHnWeuMV1nQu0LNWBKiX/Vn/1vFmuEHxL71oNPE0jgR1gphVVThKaRawG8hat9inVM14 +FeTVPVXNuoyDDM9Ua5RfdOkDm3Gc6KwDVN1LSAm8NHV/7mv9H1THGcV7FBZe+qGkACgFQ +hUZzgN24uWBgDfhfLhgKzEuxas4vLRAXrhNTDeDgMRbuKodxT5nb6fiqRVAF4wDUy5t1y +gHPNxIQBpSNXihjDrpwmtB5uVTywvZhPARyhxIUxD5xj9jQsAMrjjexiMQHeVVI+nOnxg +wxukQq+beA/OlUccVPA75ro0pJw8JbwQtjuZQtA1jiy7PZlEMkmX3EUrTOv56U6ztMVaZ +A6k+NUJxoZ2OeqBUaUCE5E7eR3CrpKph1RRbTuux71USuQb4GlhTJc81glS91E2NMNz1u +F1rU5dyyYZHvT5wc2+KKzPLblqDEioE8e4/V7IFdiQuXXXSkQu2UYpbs4jiJy7pmETErK +mLmL6tM/BDLbAETOzw+b/05/9zHyUKZiRC9XMv/NBuM+WVqrM8aWxuoyyDdnyWIRaa1ki +C8BEBZVX0T8WKrAGxGPTtUXv2sGtbwbEMWMeyP7djcHKqFan6VaCfCxyKfmdGM03xqX9N +cnb054Y2CZdFgI5qpH7YghXsySgU3RK1NjdZTFQtx9vBQI/IDOAbhtNt6V0aQ6avRdzFC +xxaDJqdGMEA/UkvvQSljKSXf+oHo8nutEEMlO5Jsy+CDeq5Mq2fjvVRpZNUpkP61eBM1k +ltzjjRUmEFFRn2EOHXyfVbxp8U4rsnmZVgNBFNsKP7QjBAXQtlf9XYGmCbUw1WGk4ZMrR +A771WBQo02s3n3XeR0mWRsZimCuz8brLhIwYo9Lzur8MZhrs6gkFRfxbv6ijVED5xA95l +G1K9Grvm5Q6qDZpiYOJAx1kAaalaPvfli2ROK4GO8LgZPSJjlnCIcVFejnVDJAOSfcUpj +eSkXN4qpibfPhMA6vtO8Lh3Eoopve+QOPnSvHNpLyM7NFDGyKeWk5mJ2LB+iSmJTDIQRF +H0SzW1ezJvmFWlJ8qkIVtpDuH6FN7/4HIPur+m9fqI88kut+KZEqZsFJ7Dyf2jX7hQQ0I +kaAwfn25SussXUIDlbPPwFzE22beL4dk9AnEL6eTbSSh5CZYQqwq4n5a9SARDQdIBNJq1 +4DeIjYm6zBME8Sr+kfIOVZMeJPZb27uf0AAmjdzAHLFsupqnWrefLV5Miwsda0XPONv79 +ms1vVQX86kuW/mvbOUM0mhZzCsPApCNnmC2KX0tbuSfZKMeYKFq1ILh+x+4Mcf8CySlme +iMVKjccAIcutN0+tUmEjjeDpr63Moq+ptSovxlFFSe4J1UrPRQFrx+GebrIntW2OrnHuT +cOH4YZqn6GEDR3op6dVCykwHFVz1JeKpj/kBso9fH+K0BcPDZ6QaJMTHWDvYzS8kHNVJZ +rmScDmXWRG34Dg5OmR85tvfQBMZXmD2gMMj2/lCFJb4NVHAwJxFhZABGa5ixKt1ndrPC/ +gVaRjHJPuUvDBTTt5a4YLXpv47nSqanbsLyB1i5YfavPRiTf1arbQATDc/fidyXhkAEuc +UoezENb2UeXd6NwWgWeQOtJR9qiaygGmJGCJ1+7fUGarSxIFAJ6S90v1PHhhGAZVJ95P2 +EgsiEnGcpzfBvDSHD8jhEbMVWbW2IKXJ/y5gsBhNoyoyDW0/9sKiNCuHW1mYe6d+625mC +irRBb0DHOBhOYjhZeQxXgEJotjropw2DSp83GU3vKSW+l1pSpZBOmsna/1hXa6/9q0bGj +JLd/Ebdxig91/RAW3ScFOSYrrhIxE0e9qCN54+BIPEJyimQjzLDmyDE3qZC3yFeXd6H/j +kMzz9lhxS2kEryWLbr33qaq/VQzY8R/Zo7O3QVNq6UkGVRyOfcDCWfIjsudKQArsdxs5o +iNhWIWv/ZQChCCHIkYPLHYNebKxLMnc44ZTnzjWRlvyHcQdlQiSBi52HLutCcUL04SXrB +mUw40fQ0v8X7Db4okWNfbcebEtTTaPnBjlVYWcFx4SWwix4GDuGCMsIM/Arioodzr8SWl +pyrm5QXJQc35Qyjqyd9L/rQDCFWLHhh2TTFuKjz8J/LidXXnyW+FZkb74Q80FxW2NGarL +9REQyCh6HAji1VT84upIVVbDgO6iKUE7naDL9oU5IBqmWEvrzs/mkHD61l6bhMbAhoeDR +H/CTFCWXlE1P082+I8qSC77FshKpFMPAeyWVTp1ftAM0AT8IwwZkINqcWrMt9NBQguaKL +nld60Rse6TTnsx8GNfnuksxUATuEehkZRpGjBKTSjzaHXKxCX8mZAuCzH+jGlnI3fCzpC +NbmI53WZMG6My7qxTrvyzTy59s3on5T7wUPayFNNM+lS07Vxw/A8uJ5SfvQA6tsO22JIX +GSKvjlBAFhsZ1ohit1tZKOwlFlKp6J0PeHNFWkqWxgdOUeqaOYFn8Yh3xhDHxaN7BVk4M +WuOFdWx72ZKwSFGKlNSnP31ttXV88l6gy1vSxrrL+ckv+kdBiTB4+e6aud5WuVgtTM6xu +Daok7j/OWFKhpCzwV7AjVwRs6/B74MHXDTNkPJYbZm/+kQmaP/L4XzmZMAn7Ay7GUdAaV +nv5t+p5PvKTEUEIIDA9lBXPup7OlH4SnkjlEnWmpggXnZQSwV1lhkCQ2LAETYhtidhr36 +8ebqiPUsLT9njaiTAh+kEFr6l/11waOUXyWIFTZjs964+5OxFU/FLrEm4hp65mRwfpiIQ +uWwz93Bc2NFlf3w8FXZDYO7DL7KGjhcmNVe1MvRKRQX0EfSKuvv/NZIMcqrz7Mh6qQyms +Dhlbcd3dOgLoma49yO/SGwbKfg5pw4hOqeClREuLUvGdC7zE/40ghfm/zGzagma7B+qlT +PbmTBHQOqsqOr55Xs0Vvw4pJw3VUtP6irH9GCe/z0qCYptkmv3QvgmSW/8ZdkoohrHM5Z +75pVVWo53vy9nA7osD/30XiBI32/45DNAPrakUQsuP6nAhOeVR1s5ecj2Assts7+AdPZg +6cJR+ozqWHG2xzfX/0Vyh8baUxbhRReB+PKcoqRQcm+hFljUXX3WWIPUYWHesR/LntIOg +6aMuapSqAsC7nsj6vi1UHmDO7gH35hn7mowlpLw1B0bQt29vblEb7CEXPALAPVeKqhRkM +dfYiESnHKSeSOmGjGbOZW0uBA0wgboAn2CXIe3yN6zyXIx8PtwK/Smj1O0FSkgfH8J68y +f7iaKd8kCNvJjdeZKk/DwcNjJ+Hhe7+AGEwQpV/1H7wfe/BVTEJcHrS83iUBUff8zzzbd +2JAss52gVwAcU05l/UaujNma4NL3v0MYKzkWcK0ehEsaagmD5eQA4Di/bS+KBq2DQWfFp +TOloDF1t4WWxIGkbr1Mmp/LsNYNWmhBffmKVR0yrhvOkwN83Ot7K6Jtx24Yiw/KImV3qt +9yPzA56IuzvyE0mXEa6cskZWEBFIgpnCk6zSQp6Jp0Yk3s5vMyLtubKdueVc8OImxsmIN +7dBHXmT2wAIjytPVQEOk0c8AqmeMUCt1i/XRGIevFq6faBQuacsOGAnMB93AC3okNhZQz +40WR9cRSqn7kTPOQI1Wszb2n1nRaPEgyOJKTuNLXbuCL/AjElyqRfcXyHYhNaChza9aUt +nJ3H9zeLLrUrP4zHZl3Umt1bKpFgoq+7vdKPLEhtQ3g0WUSSW6FjlqDkV6NZ2fSjdNGL6 +kPWuGPAHmZV+C1BzxG9djKNv4EweBpUPnFqQ97RQLXS+sJJpE2vVHXN7WaDLhXZs3HvxU +6ULdoOsfuylStmcpJaNYZmx034oegxQleHbZR8qRWMOIGyugCIX//lGDdwHT2DNRXKOl1 +yavgIUIaw0iJk2gLMvltD6GdJ8UpfHzsFsTZHFpraBW1H9qKG9Lx/06JjtFm9/rmQbKty +tv6ZcJb5TvkMo7bVXlVNH8A5LFfvbf9Ptt2DeQK0Bv3tmWGEyMcjaPH8e/2qjkSjjw7a4 +RklnYerQNk9ygsFEDzknlBAxx950aPqQmYMB8a+ObKMSRF31GWcweZmuu1sJtcdf4r6E7 +xNaVnAmtfbEdvc3SThomjFXnvUt0+66YtPrOiosvYkBvsTTmCb020JJQAA84zE2/V4UqO +qN+Ok/k64dspXIFspKlzzCEkBgC8QtrtkCWuI6xJGBa69tuCXzep5CByqyikl/CBZpWUf +4pdgqVQU+MgKbbooAJwOvOz4iIkPO69ntQOX4DHQNhVK9sbNs87cJ3w3tNEwtLkfwT2Iw +xBcRsH7qN5WBJNJprHh+TF4K8/Enk82aPeUhVsCKznib5s3uvVtt/Of76WwdZDhtHi4Ny +Ncu1PK7l+AgW5xVg7YVeIujMMed3l+tJ7dOAdTzaMLeVOfoGJ8pWpPcULwaVIlMidilr/ +1dD6zblOLc8ua879HlQh8/zPw/clEtw66KkG7I8Srj2k/iBsCFgKNRMWNlbHRDIeLdPxP +3PoJTES5M57iA/ju98wJQfur7lcqTDbYYkRGHGJyWrAhTz2JUONcfiHziUiZa0b6SG" / +> <div id="container"> <table height="100%" width="100%" cellpadding="0" cellspac +ing="0" border="0"> <tr> <td> <div id="topcontainer"> <table cellpadding="0" cellspacing="0" bor +der="0" width="100%" height="107px" dir="rtl"> <tr> <td width="50%">&nbsp;&nbsp;&nbsp; +<a href='/Default.aspx'><img src="/images/shared/logoEng_ar.gif" bord +er="0"></a></td> <td valign="bottom" align="left">< +span class="SiteLang"><a href="/fr/waysbankarabiguide.aspx" title="Fr +ench"><img src="/images/shared/Francais.gif" alt="French"></a> <a href="/en/waysbankarabiguide.aspx" title="English"><img src="/image +s/shared/img_top_english.gif" alt="English"></a> </span></td> </tr> </table> <div id="topnavigation"> <div id="topnavigationlist"> <ul id='ULtopnavigationlist'><li><a href="/ar/default.aspx"> + </a></li><li><a href="/ar/aboutus.aspx"> </a>< +ul><iframe scrolling='no' frameborder='0' style='position: absolute;f +ilter: progid:DXImageTransform.Microsoft.Alpha(style=0,opacity=0);lef +t: 0;top: 0;width:160;background-color:transparent;z-index: 0;display +:inline;'></iframe><li><a href="/ar/messagefromchairman.aspx"> +</a></li><li><a href="/ar/history.aspx"> </a><ul><ifra +me scrolling='no' frameborder='0' style='position: absolute;filter: p +rogid:DXImageTransform.Microsoft.Alpha(style=0,opacity=0);left: 0;top +: 0;width:160;background-color:transparent;z-index: 0;display:inline; +'></iframe><li><a href="/ar/avisionfulfilled.aspx"> </a> +</li><li><a href="/ar/trustandcommitment.aspx"> +</a></li><li><a href="/ar/catalyst.aspx"> </ +a></li><li><a href="/ar/nationalization.aspx"></a></li><li><a +href="/ar/steadyexpansion.aspx"> </a></li><li><a href="/ar/ar +abbanktoday.aspx"> </a></li></ul></li><li><a href="/ +ar/profile.aspx"> </a></li><li><a href="/ar/boardofdirect +ors.aspx"> </a></li><li><a href="/ar/ourvalues.aspx"> +</a></li><li><a href="/ar/humanasset.aspx"> </a><ul><iframe + scrolling='no' frameborder='0' style='position: absolute;filter: pro +gid:DXImageTransform.Microsoft.Alpha(style=0,opacity=0);left: 0;top: +0;width:160;background-color:transparent;z-index: 0;display:inline;'> +</iframe><li><a href="/ar/topmanagement.aspx"> </a></li> +<li><a href="/ar/ourpeople.aspx"></a></li></ul></li><li><a href +="/ar/financials.aspx"> </a></li></ul></li><li><a href="/a +r/persbank.aspx"> </a><ul><iframe scrolling='no' framebor +der='0' style='position: absolute;filter: progid:DXImageTransform.Mic +rosoft.Alpha(style=0,opacity=0);left: 0;top: 0;width:160;background-c +olor:transparent;z-index: 0;display:inline;'></iframe><li><a href="/a +r/persloans.aspx"></a><ul><iframe scrolling='no' frameborder='0 +' style='position: absolute;filter: progid:DXImageTransform.Microsoft +.Alpha(style=0,opacity=0);left: 0;top: 0;width:160;background-color:t +ransparent;z-index: 0;display:inline;'></iframe><li><a href="/ar/hous +inglaons.aspx"> </a></li><li><a href="/ar/smeloans.aspx" +> </a></li></ul></li><li><a href="/ar +/persaccount.aspx"></a><ul><iframe scrolling='no' frameborder +='0' style='position: absolute;filter: progid:DXImageTransform.Micros +oft.Alpha(style=0,opacity=0);left: 0;top: 0;width:160;background-colo +r:transparent;z-index: 0;display:inline;'></iframe><li><a href="/ar/p +ersaccountcurrent.aspx"> </a></li><li><a href="/ar/persac +countsave.aspx"> </a></li><li><a href="/ar/persaccountfdjo +.aspx"> - </a></li><li><a href="/ar/persaccountfdfor +eign.aspx"> - </a></li></ul></li><li><a href= +"/ar/persremit.aspx"> </a><ul><iframe scrolling='no' fram +eborder='0' style='position: absolute;filter: progid:DXImageTransform +.Microsoft.Alpha(style=0,opacity=0);left: 0;top: 0;width:160;backgrou +nd-color:transparent;z-index: 0;display:inline;'></iframe><li><a href +="/ar/persremittrans.aspx"> </a></li><li><a href=" +/ar/persremitfx.aspx"> </a></li><li><a href +="/ar/persremitdrafts.aspx"> </a></li></ul></li></ul></li +><li><a href="/ar/corpbank.aspx"> </a><ul><iframe scrolli +ng='no' frameborder='0' style='position: absolute;filter: progid:DXIm +ageTransform.Microsoft.Alpha(style=0,opacity=0);left: 0;top: 0;width: +160;background-color:transparent;z-index: 0;display:inline;'></iframe +><li><a href="/ar/corploans.aspx"> </a +><ul><iframe scrolling='no' frameborder='0' style='position: absolute +;filter: progid:DXImageTransform.Microsoft.Alpha(style=0,opacity=0);l +eft: 0;top: 0;width:160;background-color:transparent;z-index: 0;displ +ay:inline;'></iframe><li><a href="/ar/corpoverdraft.aspx"> +</a></li><li><a href="/ar/corprevolv.aspx"></a></li></ul></li>< +li><a href="/ar/corpinttrade.aspx"> </a></l +i><li><a href="/ar/investloan.aspx"></a></li></ul></li><li><a h +ref="/ar/treasury.aspx"></a></li><li><a href="/ar/waysbank.asp +x"> </a><ul><iframe scrolling='no' frameborder='0' + style='position: absolute;filter: progid:DXImageTransform.Microsoft. +Alpha(style=0,opacity=0);left: 0;top: 0;width:160;background-color:tr +ansparent;z-index: 0;display:inline;'></iframe><li><a href="/ar/waysb +ankarabi.aspx"> </a></li><li><a href="/ar/waysbankbranchn +et.aspx"> </a></li></ul></li></ul> </div> </div> <span class="TopNavGray"></span> </div> </td> </tr> <tr valign="top"> <td height="100%" width="100%"> <DIV id="leftnavigation"> <h2 id="_ctl2__ctl0__ctl0_pt" class="titleBlue"><span> +</span></h2> <script type="text/javascript"> var baseopacity=0 /*function test() { xx=document.getElementsByTagName("div"); for(var l=0;l<xx.length-1;l++) if((xx[l].id)=="tet") { alert(xx[l].id); document.getElementById(xx[l].id).className="Normmenu2"; alert(xx[l].className); } }*/ function RedirectPage(thetext){ location.href=thetext+".aspx" } function showtext(thetext){ if (!document.getElementById) return /*document.getElementById(thetext).style.visibility="visible"; document.getElementById(thetext).style.fontSize=12;*/ if (document.getElementById("DivId").value=="") { document.getElementById("DivId").value=thetext; } else { document.getElementById(document.getElementById("DivId").value).style. +display="none"; document.getElementById("DivId").value=thetext; } document.getElementById(thetext).style.display="block"; textcontainerobj=document.getElementById(thetext) browserdetect=textcontainerobj.filters? "ie" : typeof textcontainerobj +.style.MozOpacity=="string"? "mozilla" : "" instantset(baseopacity) /*document.getElementById("tabledescription").innerHTML=thetext*/ highlighting=setInterval("gradualfade(textcontainerobj)",100) } function hidetext(){ cleartimer() instantset(baseopacity) } function instantset(degree){ if (browserdetect=="mozilla") textcontainerobj.style.MozOpacity=degree/100 else if (browserdetect=="ie") textcontainerobj.filters.alpha.opacity=degree else if (document.getElementById && baseopacity==0) document.getElementById("tabledescription").innerHTML="" } function cleartimer(){ if (window.highlighting) clearInterval(highlighting) } function gradualfade(cur2){ if (browserdetect=="mozilla" && cur2.style.MozOpacity<1) cur2.style.MozOpacity=Math.min(parseFloat(cur2.style.MozOpacity)+0.2, +0.99) else if (browserdetect=="ie" && cur2.filters.alpha.opacity<100) cur2.filters.alpha.opacity+=20 else if (window.highlighting) clearInterval(highlighting) } </script> <div id="_ctl2__ctl0__ctl1_cldiv" class="cLVertical"> <input type =hidden id="DivId" name ="DivId"> <div id="waysbankarabi" > <a href=javascript:RedirectPage('waysbankarabi')> + </a> <input type="hidden" value ="waysbankarabi" id="val2"> </div> <span id ='waysbankarabi' class=tabledescription><div id=' +waysbankarabiguidesub' class=SubMenuSty><a href ='/ar/waysbankarabigu +ide.aspx'> </a></div><div id='waysbankarabiadvicesub' cla +ss=SubMenuSty><a href ='/ar/waysbankarabiadvice.aspx'> < +/a></div> <input type="hidden" value ="waysbankarabi_waysbankarabigu +ide_waysbankarabiadvice_" id="val1"></span> <div> <table> <tr> <td height="0px"></td> </tr> </table> </div> <div id="waysbankbranchnet" > <a href="/ar/waysbankbranchnet.aspx"> </a> <input type="hidden" value ="waysbankbranchnet" id="val2"> </div> <span id ='' class=tabledescription> <input type="hidden" value ="_" id="val1"></span> <div> <table> <tr> <td height="0px"></td> </tr> </table> </div> </div> <script type =text/javascript > var Parant; var sPath = window.location.pathname; //var sPage = sPath.substring(sPath.lastIndexOf('\\') + 1); var sPage = sPath.substring(sPath.lastIndexOf('/') + 1); var x=document.getElementsByTagName("input"); sPage = sPage.toLowerCase(); sPage=sPage.replace(".aspx",""); //a=a.toLowerCase(); for(var i=0;i<x.length;i++){ if( ((x[i].id).indexOf("val2") != -1)){ var x3=new String(x[i].value.toLowerCase()); //alert(x3) if (x3==sPage) { //document.getElementById(x3).className=""; document.getElementById(x3).id="Normmenu2"; }else { document.getElementById(x3).id="Normmenu"; } } if( ((x[i].id).indexOf("val1") < 0) || (x[i].value == "")) continue; var str=new Array(); var x2=new String(x[i].value.toLowerCase()); str=x2.split("_"); for(var j=0;j<str.length-1;j++){ a=str[j].toLowerCase(); if(a == sPage){ showtext(str[0]); //alert(document.getElementById(a+"sub").id); if (document.getElementById(a)==null){ document.getElementById(a+"sub").className="SubMenuSty2";} //else{ //document.getElementById("Normmenu").className=""; //document.getElementById("Normmenu").id="Normmenu2";} break; i=x.length+1; } } } </script> <div class="clear"> </div> </DIV> <div id="wrapper"> <div id="maincontent"> <h1> <br><img src ='/images/GrayHeaderAr.gif'></h1> <div ID="genCntnt"> <p align="justify"><strong> :</strong></p> <p>&nbsp; . </ +p> <p><strong> :</strong></p> <div> <ul> <li> <div align="justify"> ɡ + . </div> </li> </ul> </div> <p><strong> :</strong></p> <ul> <li> <div align="justify"> <a href="http://w +ww.arabbank.dz">www.arabbank.dz</a></div> </li> <li> <div align="justify"> &quot;&quot; + </div> </li> <li> <div align="justify"> ɡ + </div> </li> <li> <div align="justify"> + . + </div> </li> <li> <div align="justify"> ɡ + </div> </li> </ul> <p><strong> : </strong></p> <ul> <li> <div align="justify"><strong> ޿</strong></div> <ul> <li> <div align="justify"> + ˡ . </div> </li> <li> <div align="justify"> ҡ + + . +</div> </li> <li> <div align="justify"> . </div +> </li> </ul> </li> <li> <div align="justify"><strong> / +: <br /> </strong> : </div> <ul> <li> <div align="justify"> &n +dash; </div> </li> <li> <div align="justify"> </div> </li> <li> </li> </ul> </li> </ul> <p><strong> :</stron +g></p> <p> ɡ &quot;&quot; + . + . </p> <p><strong> ɿ<br /> </strong> &quot;&quot; + . </p> <p><strong> +ǿ</strong></p> <ul> <li> <div align="justify"> + . ǡ + + . </div> </li> <li> <div align="justify"> + . ǡ + . </div> </li> <li> <div align="justify"> + +. ɡ . + </div> </li> </ul> <p><strong> </strong></p> <ul> <li> <div align="justify"> + . + + . </div> </li> <li> <div align="justify"> ɡ + + ɡ + . </div> </li> <li> <div align="justify"> + ɡ +. </div> </li> <li> <div align="justify"> ޡ + . + ʡ .</div> </li> </ul> </div> </div> </div> <div class="clear" style="HEIGHT: 10px"></div> </td> </tr> <tr> <td> <div> <div id="footer"> <table cellpadding="0" cellspacing="0" + border="0" width="775" ID="Table1"> <tr bgcolor="#b2b2b2" height="19"> <td width="20%"></td> <td align="center"><a href='/a +r/privacypolicy.aspx'> </a></td> <!--<td align="center"><a href +='/ar/faqs.aspx'> </a></td>--> <td align="center"><a href='/a +r/sitemap.aspx'> </a></td> <td align="center"><a href='/a +r/contactus.aspx'> </a></td> <td align="center"><a href='/a +r/securitystatment.aspx'> </a></td> <td align="center"><a href='/a +r/legal.aspx'></a></td> <td width="20%"></td> </tr> <tr bgcolor="#e5e5e5"> <td colspan="8"> <table cellpadding="0" cel +lspacing="0" border="0" width="100%" ID="Table2"> <tr id="subFooter"> <td> + &copy; <!-- token:year --> <a href='http://www.arabbank.com'> +</a> </td> </tr> </table> </td> </tr> </table> </div> </div> </td> </tr> </table> </div> <script language=JavaScript>if (document.getElementById +("maincontent")!=null) document.getElementById("maincontent").style.w +idth='100%'; </script> <script language="Javascript"> topnavigationlistFN = function() { if (document.getElementById("ULtopnavigationlist")==null) {return; +} var sfEls = document.getElementById("ULtopnavigationlist").getElem +entsByTagName("LI"); for (var i=0; i<sfEls.length; i++) { if (sfEls[i].getElementsByTagName("UL").length>0){ sfEls[i].onmouseover=function() { this.className+=" sfhover"; } sfEls[i].onmouseout=function() { this.className=this.className.replace(new RegExp(" sfh +over\\b"), ""); } } } } if (window.attachEvent) window.attachEvent("onload", topnavigationlist +FN); </script> </form> </body> </html>
        It's better to use <pre> tags instead of <code> tags if the data contain unusual characters. If the sample is that large, a <readmore> is also a plus.
        لսႽ ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ
        The non-ASCII content in that HTML data is also non-UTF8. Treating it as CP-1256 will probably yield suitable results.

        If there are a bunch of HTML files like this (and also a bunch that really are utf8), and you don't want to waste too much time sorting them out, you can add a subroutine like this to your program:

        use Encode; sub check_encoding { my ( $inp_name ) = @_; open( my $fh, '<:raw', $inp_name ) or return "$inp_name: open fail +ed: $!"; my $str = ''; until ( $str =~ /[^[:ascii:]]/ ) { $str = <$fh>; } if ( $str =~ /^[[:ascii:]]+$/ ) { return "ascii"; } eval { $_ = decode( 'utf8', $str, Encode::FB_CROAK ) }; if ( $@ ) { return "cp1256"; # We assume Arabic only, so if not utf8, the +n cp1256 } else { return "utf8"; } }
        (update: removed hyphen from "cp1256")

        Call that subroutine for each file name, and it will return the string that you should use for the encoding spec when you open the file for parsing. If you handle data for any language other than Arabic, and encounter the same problem, you'll need to tweak this to return some other non-unicode encoding, depending on the language.

        You'll want to read the man page for Encode, especially the part about "Handling Malformed Data".

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1074106]
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (7)
As of 2014-12-25 00:10 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (159 votes), past polls