Step 0: Data ingestion Setup

[enjalot]

We want to be able to add more error handling and feedback before fully ingesting a dataset.

We can also flag potential embedding columns (columns that are uniform dimensional arrays) so that they can be brought in as embeddings in parallel with the dataset.

This also gives us a chance to show the user what we know about the columns of the dataset, including any columns we deem "categorical" which allow for filtering.

It may make for easier debugging of poorly formatted data as well.