You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For a hub to be successfully accessed as an arrow dataset, column data types should not change from round to round.
Generally many task IDs that are covered by our schema shouldn't change data type in further rounds as that's somewhat fixed by the schema. Custom task IDs however, which are beyond our control, and the output_type_id column have the potential to change and this could indeed cause problems downstream. This is mainly a problem for parquet files (but has a small chance to cause problems in csvs too).
This is why early on when we had lots of discussions about this we allowed for a hub to override the automatic detection of the output_type_id data type in the hubData::create_hub_schema() function, used to determine the overall hub schema from the tasks.json config file. To future proof the output_type_id admins could set the value of output_type_id_datatype to the safest, most future proof data type, i.e. character.
Set value of arg output_type_id_datatype in config
To enable hub admins to be able to configure and communicate this setting at a hub level, I propose an output_type_id_datatypeenum optional property at the top level of tasks.json config (i.e. sibling to rounds and schema_version) that would accept valid hubData::create_hub_schema()output_type_id_datatype argument values. If this property exists in the config, the default behaviour (unless specifically overridden when calling) of hubData::create_hub_schema() would be to set the output type id column to the data type specified in the config.
This would give admins the ability to future proof their hubs by setting the column to character if they are unsure whether they may start collecting an output type that could affect the schema.
The text was updated successfully, but these errors were encountered:
Background
For a hub to be successfully accessed as an arrow dataset, column data types should not change from round to round.
Generally many task IDs that are covered by our schema shouldn't change data type in further rounds as that's somewhat fixed by the schema. Custom task IDs however, which are beyond our control, and the output_type_id column have the potential to change and this could indeed cause problems downstream. This is mainly a problem for parquet files (but has a small chance to cause problems in csvs too).
This is why early on when we had lots of discussions about this we allowed for a hub to override the automatic detection of the
output_type_id
data type in thehubData::create_hub_schema()
function, used to determine the overall hub schema from thetasks.json
config file. To future proof theoutput_type_id
admins could set the value ofoutput_type_id_datatype
to the safest, most future proof data type, i.e. character.Set value of arg
output_type_id_datatype
in configTo enable hub admins to be able to configure and communicate this setting at a hub level, I propose an
output_type_id_datatype
enum
optional property at the top level oftasks.json
config (i.e. sibling torounds
andschema_version
) that would accept validhubData::create_hub_schema()
output_type_id_datatype
argument values. If this property exists in the config, the default behaviour (unless specifically overridden when calling) ofhubData::create_hub_schema()
would be to set the output type id column to the data type specified in the config.This would give admins the ability to future proof their hubs by setting the column to character if they are unsure whether they may start collecting an output type that could affect the schema.
The text was updated successfully, but these errors were encountered: