Experiments with command line programs for streams processing in the D programming language, particularly focusing on TSV and stringified-JSON formats. note that these are Unixoid tools: reading their inputs from STDIN, writing their outputs to STDOUT, and writing usage and error messages to STDERR.
tsv2json [-c | --compact]
reads a tsv file from stdin and writes a json file to stdout
assumes there is a header record to use as JSON keys
memberInfo.sh twitterUserHandle listname
(sometimes I hate markdown) uses
<a href="https://github.com/sferik/t" title="t commandline tool">t</a>
to get users who are members of a particular list, then
<a href="https://github.com/eBay/tsv-utils-dlang" title="tsv utils"> csv2tsv</a>
to pre-process into tsv, and finally **tsv2json** to produce output as reasonably-compact stringified-JSON.
there is an incredible amount of data which can usefully be thought of as a stream, and properly-written streams-processing programs can take arbitrarily-large data files as input... no more Mister Memory-Constrained for processing data.
because I am largely a Webbish sorta person when it comes to communications, and JSON-form data is useful and fast and easy to work with in a browser context.
csv, because of all the
rant removed because of the number of people it would offend
data capture that happens into spreadsheet form, often just because all they have is a hammer and everydamnedthing looks like a nail
another rant removed for the same reasons
and tsv because it is easier to work with to sanitize out all the cruft the Multiverse guarantees will show up
third and final rant redacted, writer cut off
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
