Decompression & Processing Pipeline

[MoRevolution]

We have wrapped up our EDA, and we now need a streamlined pipeline to decompress, preprocess, and search through the whole Reddit Submission dataset. So here is a preliminary list of tasks we need to complete:

• Write a script to decompress each of the files in the Submission dataset-preferrably use stream decompression from the zstandard python library.
• Create a pipeline to preprocess and search through the dataset for, using the same principles as in our EDA notebook.

[MoRevolution]

I just added a decompression script in the "src" directory. It works, but it's slow since I didn't add any form of multiprocessing. It should be fine for now, but I'll incorporate multiprocessing after we validate the output of the full 2018 dataset.