-
Notifications
You must be signed in to change notification settings - Fork 88
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide base class for dataset loaders #59
Conversation
This will allow for the future addition of an eager dataset loader
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great @cthoyt !
any chance we would load , train and essentially test our own dataset ? |
@aminemosbah I would suggest looking at https://chemicalx.readthedocs.io/en/latest/api/chemicalx.data.LocalDatasetLoader.html#chemicalx.data.LocalDatasetLoader for loading your own dataset that's already in the right format in a given directory |
thx, but i have smiles to predict in a csv file to predict locally , any quick snippet ? |
@aminemosbah I have in mind a solution for what you want (which is the obvious realistic use case) but this it is blocked #50 and #58. @benedekrozemberczki would love to get your input on #50 ;) |
need to hack the dataloader to make it work for local data |
Summary
This PR abstracts the essential components of the dataset loader into a base class to allow for future implementations of eager datasets (e.g., all parts of the dataset are already in memory) and for other lazy local dataset loaders.
Changes
RemoteDatasetLoader
Next steps
The following shows an implementation of an eager dataset, which might be more useful for local datasets.