Introduces load_from_disk datasets

[nickmitchko]

This addition allows for a user to load a dataset directory they crafted locally using the save_to_disk functionality of HuggingFace Datasets. If you pass in a dataset with a directory name (ending in / or \ ) it will be treated as a local huggingface dataset

https://huggingface.co/docs/datasets/process#save

[artidoro]

Ideally huggingface does the parsing for us. We should stay away from deciding what is local and what is on the hub.
Also isn't this handled by load_dataset?
https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.load_dataset

Please provide more details for why this would be needed or if it's already included in the repo.

[nickmitchko]

Ideally huggingface does the parsing for us. We should stay away from deciding what is local and what is on the hub. Also isn't this handled by load_dataset? https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.load_dataset

The impetus of this change was that load_dataset did not handle my dataset directory correctly. I can lookup the exact error but handling this use case (directory slash) works.

Please provide more details for why this would be needed or if it's already included in the repo.

You can't load a local dataset directory (one not published on the hugging face hub) without it. I have data that can't be published on the hub and this is why I need this functionality.

[nickmitchko]

If you check out the load_dataset method, it only opens csv, json, parquet, and others based on the python script provided in the directory. save_to_disk from here doesn't save in this format. Thus if you concatenate a variety of datasets into one, and then save to disk, you can't use the current load_dataset method.

[nickmitchko]

@artidoro thoughts?