Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce a property to fix the output_type_id column data type across the hub #87

Closed
Tracked by #92
annakrystalli opened this issue Jul 1, 2024 · 0 comments · Fixed by #88
Closed
Tracked by #92
Assignees

Comments

@annakrystalli
Copy link
Member

Background

For a hub to be successfully accessed as an arrow dataset, column data types should not change from round to round.
Generally many task IDs that are covered by our schema shouldn't change data type in further rounds as that's somewhat fixed by the schema. Custom task IDs however, which are beyond our control, and the output_type_id column have the potential to change and this could indeed cause problems downstream. This is mainly a problem for parquet files (but has a small chance to cause problems in csvs too).

This is why early on when we had lots of discussions about this we allowed for a hub to override the automatic detection of the output_type_id data type in the hubData::create_hub_schema() function, used to determine the overall hub schema from the tasks.json config file. To future proof the output_type_id admins could set the value of output_type_id_datatype to the safest, most future proof data type, i.e. character.

Set value of arg output_type_id_datatype in config

To enable hub admins to be able to configure and communicate this setting at a hub level, I propose an output_type_id_datatype enum optional property at the top level of tasks.json config (i.e. sibling to rounds and schema_version) that would accept valid hubData::create_hub_schema() output_type_id_datatype argument values. If this property exists in the config, the default behaviour (unless specifically overridden when calling) of hubData::create_hub_schema() would be to set the output type id column to the data type specified in the config.

This would give admins the ability to future proof their hubs by setting the column to character if they are unsure whether they may start collecting an output type that could affect the schema.

@annakrystalli annakrystalli added this to the robust-hub-schema milestone Jul 1, 2024
@annakrystalli annakrystalli moved this from Todo to Up Next in hubverse Development overview Jul 17, 2024
@annakrystalli annakrystalli self-assigned this Jul 22, 2024
@annakrystalli annakrystalli moved this from Up Next to In Progress in hubverse Development overview Jul 22, 2024
@annakrystalli annakrystalli moved this from In Progress to Ready for Review in hubverse Development overview Jul 22, 2024
annakrystalli added a commit that referenced this issue Jul 24, 2024
Add output_type_id_datatype property to v3.0.1. Resolves #87
@github-project-automation github-project-automation bot moved this from Ready for Review to Done in hubverse Development overview Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging a pull request may close this issue.

1 participant