Re: Ignoring not well-formed (invalid token) errors

[Laurent_R]

If the error is always showing the same pattern, maybe you could preprocess the file to remove the offending line(s). I know that the idea of preprocessing 13 GB is not very attractive, but sometimes you have to bite the bullet.

[bitingduck]

If it's looking for a simple pattern it might be doable in a reasonable amount of time. There are extractors for the Open Directory Project and Wikipedia dumps, both of which are in the many GB range, that can process very quickly, even on relatively old machines. I was pulling all of the music content out of ODP in less than a few minutes some 10 years ago on a mac laptop that was reasonably current then, and I don't recall how long it took to pull all the music topics out of Wikipedia, but I think it was quite reasonable.