Re^2: Command-line utility to apply perl expressions to CSV files

[gpvos]

No, I don't think I did. From a cursory look, these tie the entire CSV, or maybe even load the entire thing into memory. Their documentation says that they have speed problems for some operations. My utility processes the CSV as a stream and only holds one line at a time in memory. Because of that, it is generally lightning fast. Also, I often use it as only one step within a pipeline.

For cases where it is necessary to load the entire table into memory, I tend to use something like csvsql from the (Python) csvkit. If I just need to delete or reorder some columns, I tend to use csvtool (from OCaml). But my tool shines for the cases in between those two when you need to do simple data transformations, like converting a column into hexadecimal, or adding a column that is the sum or concatenation of two or more others.

It is mostly intended to be similar to awk, but then reading and writing CSV instead of text (and offering the flexibility of Perl). In fact, that's why I added the -b and -e options, to have something similar to BEGIN and END, even though I haven't yet used those options in a real scenario.

The horrible way I use tie just allows you to address the field values in the current line by their column name using a hash called %F, even though those are actually stored in the array @F which you can also directly use. The tie just passes through the actions after applying a mapping from field name to array index. But you can also work with only the array and ignore the tied hash, which was in fact the only possibility until two weeks ago.

[Fletch]

In the vein of other tools for an interactive approach check out visidata which gives a tty spreadsheet (pretty much anything csv, tsv, or xls that python can read) editor.