Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Limit to Line Length?

by Kozz (Friar)
on Jan 26, 2001 at 21:23 UTC ( #54573=perlquestion: print w/ replies, xml ) Need Help??
Kozz has asked for the wisdom of the Perl Monks concerning the following question:

My dear monks:
    Is there a limit to the length of a line in an plain text file?
    Assuming that my record separator is "\n", and the only application reading from/writing to the file is a perl script, would there be any possibility of encountering a limit to the number of characters in a single line when written to this text file?
    I've got a file which is being read from/written to by a perl script which I'm sure is using all the right methods of file locking. Yet after extensive use over a period of many months, suddenly this datafile is missing several records... Any insights would be appreciated.

Comment on Limit to Line Length?
Re: Limit to Line Length?
by mr.nick (Chaplain) on Jan 26, 2001 at 21:34 UTC
    Not that I could find. I created a text file with two lines, each approximate 665,000 characters in length. Then I did a
    open IN,"<bigline" || die; while (<IN>) { print; } close IN;
    and it seemed to have worked just fine. No doubt the line length is limited only by available memory.
Re (tilly) 1: Limit to Line Length?
by tilly (Archbishop) on Jan 26, 2001 at 21:35 UTC
    There should be no limit to the length of a line.

    OTOH there are a lot of subtle ways that file locking can go wrong, and most of the time when I see people offering advice on how to do it they get it wrong. Therefore, your confidence notwithstanding, I would suspect the locking routines.

    For a discussion of what can go wrong, see RE: RE: Flock Subroutine or Don't use unflock (flock 8). Ever.. See Simple Locking for a reasonably good example of how to do it.

    Incidentally if you can, I would suggest moving this to a database...

      Incidentally if you can, I would suggest moving this to a database...
      Yep. That's very good advice. At the time in which I put the scripts together, I had no knowledge of databases. Now that I've got some books, read up, and have had moderate experience with them, I'm prepared to move it to a true database. I need only to pray for a visit from the magic free-time fairy, and I can convert things over. ;) Thanks again for your advice.
Re: Limit to Line Length?
by wardk (Deacon) on Jan 26, 2001 at 21:35 UTC

    While I don't directly know the answer to this, and is likely operating system dependant, this is a great opportunity to use perl to find out. Write a script to create successively longer lines, prefixed by the length. if the lines are missing due to length issues, you should be able to ascertain what that number is for your platform pretty easily. the max length should be length of the last line + 1

    Are you *sure* the missing lines are related to a logic issue? or perhaps a limitation in your text editor?

      Good idea! I just wrote the following and am executing it as we speak (it's at 1_400_000 right now):
      #!/usr/bin/perl my $l=600_000; while (1) { print "\rTesting $l..."; open OUT,">biglines.txt" || die; print OUT "x" x $l; close OUT; open IN,"<biglines.txt" || die; my $r=<IN>; close IN; if (length($r)!=$l) { print "Oops! A linelength of $l fails!!\n"; exit; } $l+=5000; }
      It doesn't have good resolution, but it would take it years to try all the possible lengths.

      (12:01pm EST) Addendum: well, it's up to 25_000_000 characters on a line and still no problems!

      (2:28pm EST) Well, I finally stopped it from executing at approximately 65MB per line :) At this point I'd say that there is no theorectical line-length-limit, at least under Linux. I would be interested, though, to see if Perl under NT lacks the same limitation.
      Don't bump by 1 -- Bump by 1024, then start backing off by 1 when you fail. The test will run a lot faster.
Re: Limit to Line Length?
by sierrathedog04 (Hermit) on Jan 26, 2001 at 22:17 UTC
    Apparently perl has no limit on line length.

    If, however, you are passing parameters to or from perl through the command line then there can be an OS-imposed limit to the total length of the values of the parameters that you pass.

    Last year I went through some tsouris (pain) when my perl database program was dropping without warning the last couple of fields that I was passing through to the command line. Someone told me that UNIX systems often limit the length of command-line parameters to some small number, maybe 512 bytes or something like that.

Kozz Re: Limit to Line Length?
by Kozz (Friar) on Jan 26, 2001 at 22:18 UTC
    This is good advice, and good questions.
    No, I'm not sure that it's not a logic issue. It's running on Linux, Perl 5.005_03. And the text editor doesn't have anything to do with it, since when I'd grep the text file for a pattern which is SUPPOSED to be there, it wouldn't show up. And again - the only thing touching these files were perl scripts.

    There are, however, a few things going on in the script which could have some consequences... primarily, my former bad habit of open,flock,read,close followed by open,flock,write, close rather than rewinding the file, writing, truncating, and closing.

    Further research leads me to realize that the dates contained within the records indicate that nothing past a certain date was appearing, and since new records are simply appended to the file, you might indeed be right that this is a flocking issue.

    Notwithstanding my confidence in my ability to use FLOCK, I still think I made some mistakes, and I appreciate Tilly's soft criticism which made me re-think my methods.

    I think I'll go look at and fix some code. I humbly thank all of you for your research and insight. :-)
Re: Limit to Line Length?
by ColonelPanic (Friar) on Jan 27, 2001 at 01:50 UTC
    A file is just a bunch of bits. The way it is arranged in memory has nothing to do with the newline character. Any limits are arbitrarily imposed by outside applications. Perl, to my knowledge, would have no reason to care. Perl does not specifically care about lines either, they are only a string of data ended with whatever character is in $/. Scalar variables are limited only by memory, and have no problem with reading an entire file if you change $/. Just my two cents.

    When's the last time you used duct tape on a duct? --Larry Wall

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://54573]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (7)
As of 2014-12-25 12:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (160 votes), past polls