|Just another Perl shrine|
loading csv files into MySQL databseby zimvoter (Novice)
|on May 16, 2008 at 17:27 UTC||Need Help??|
zimvoter has asked for the
wisdom of the Perl Monks concerning the following question:
Please help. . I am based in Zimbabwe and have volunteered to put the voters roll in a database for the opposition government in waiting. The ruling dictator Mugabe does not offer the database in electronic format. We are to have a second round of elections in the next 4 weeks and lots of our supporters have been killed and beaten by the Mugabe regime. Despite all this we are confident that the will of the people will prevail via these elections. We need to have a database so that displaced people and others can know where and whether they are registered to vote. I have a young programmer who has been working in trying to load the data into an My SQL database. We have all the data in excel or csv files , We have 210 constituencies with total voters registered about 5.9Million.
We are looking for a solution that will quicken and make the process of loading into MySQL database easier We would need some help in someone who knows about php and MySQL to assist my associate by the name Green Below is his description of the problem he is facing and an example of the error messages.
We are working with PHP and MySQL to developing a database front-end to manage data we collected from around the country on a windows operating system. We got the date files in PDF format and they are being converted via an OCR process of pdf2XL so that we can extract the data to input in the database. We have created the CSV’s for uploading to our MySQL database and we have faced a problem of timeouts and with apostrophe’s/single quotes contained in the names and the addresses. Is there a way of making MySQL ignore the single quotes/apostrophe’s when uploading the data? The data we have to upload runs into millions of records and we have limited time to have the database up and running.
Example error message
Attached is the sample csv file mtp4date.csv
On the issue of timeouts, we plan on changing the default in the PHP ini file from 300 seconds so that we can upload as much data as we want. What is the best solution regarding timeouts?Regards