-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
load torch tensors in OGBDatasets #107
Comments
Can I work on this? |
Sure. I don't remember for which specific dataset this was needed though |
This problem can be seen in |
Been inactive due to Uni. exams, will start working on it today. |
Some problems have been overcome here, including loading ".pt" format using Pickle.jl and have been discussed with @chengchingwen : https://github.com/yuehhua/GraphMLDatasets.jl/blob/65d6a2bb02d31569a64b47004a0c4b192739a066/src/preprocess.jl#L391 |
Split tensors appear for edge-level tasks in OGB Datasets. The dataset loading for LinkPropped Datasets differs from GraphPropped or NodePropped. We might need a change of OGB-Dataset APIs.
data = OGBDataset(name, split; dir) But this has one obvious problem: loading any split eg. train would involve computation of the other two splits (val and test) given the intertwined nature of how the data is stored.
train_data, test_data, valid_data = OGBDataset(name; dir) Can be ambiguous for non-split datasets and does not exactly match with other dataset APIs.
data = OGBDataset(name; dir)
train_split = split(data, :train) # this may weird way to do
# maybe something like
train_split = data[:train] Representation for link tasks in OGBDataset will differ from Node or Graph tasks. |
Also, API for splits should be consistent for different data sources. eg: Cora and OGBDataset access training masks using different APIs. |
Some of the features of the OGBDataset are downloaded as torch tensor stored in the ".pt" format. They are currently ignored at the moment, but we could load them using Pickle.jl (e.g. see this comment)
The text was updated successfully, but these errors were encountered: