I rememberred that we had this problem with our automatic job processing too. We will kick off processing when certain file has arrived from FTP. We came up with several solutions:
By periodically checking the size of the file coming in, and if the file size has stopped growing, then we would assume
that the file transfer has stopped. However this does *NOT* work, at least not reliablly! We had one perticular case where the FTP has paused / died and the system thought that the file has been received properly, and started to process the file. It made a total mess that took many days to resolve.
A better approach than the first one is to modify the system to receive the data file first, followed by a trigger file. The system will act on the arrival of the trigger file. This approach is more reliable than the first one, however, it assumes that the sender works properly. We had a case when the sender program/script sent half the data file, and then somehow sent the trigger file without checking the completion of the data file. This of cause caused another mess.
The best senario is when the data file has an integrated verification mechanism, like a ZIP file. You can be certain if the incoming ZIP file has arrived completely by periodically testing the integrity of the ZIP file with the zip -T
switch. This method works 100% of the time.
The 3rd method, with a self validating file format, is the preferred method, if the client/sender can produce such format. If not possible, then fall back to the 2nd method with additional trigger file. If this is not possible, then fall back to the 1st method and pray. :-D
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.
| & || & |
| < || < |
| > || > |
| [ || [ |
| ] || ] ||