Unless you can guarantee that the CSV you are dealing with is absolutely free of fields with embedded newlines, CSV parsing cannot be threaded or parsed in parallel.
The OP only deals with fields 7 and 31, which are unlikely to contain new-lines, but we have no idea what the other fields may hold.
Enjoy, Have FUN! H.Merijn
| [reply] |
We do have some idea of what the other fields may hold, in that the original post states the CSV files are firewall logs. I'm not aware of any common firewall log format which includes data that might contain embedded newlines, so it's probably not an issue in this case. But OP would know the specifics of the format they're dealing with better than I do, of course.
| [reply] |
| [reply]
[d/l] |
Right, there are no new-lines in there and it's pretty uniform data overall.
| [reply] |
