You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In #343, we're currently using pickle to save the metadata dataframe. This has the advantage of allowing us to save anything that the user inserted into metadata (including arrays, functions, etc.), but is brittle: pickle sometimes fails to load if the version of e.g., pandas used when saving and loading differs.
Given that I think we don't need the full flexibility of pickle (why would the user insert a function as metadata), we should instead use json to save the dataframe. That would require us to check that whatever metadata value the user sets is json-serializable. It looks like
returns '{"0":{"0":1},"1":{"0":{}}}' (converting the lambda function to an empty set), so we'll need to come up with some other way to check whether the object is really json-serializable. We could check that df == pd.DataFrame(json.loads(df.to_json())), though this might be a bit slow.
Also, to_json forces ascii by default, so if we want to support unicode characters in metadata, will need to change that.
The text was updated successfully, but these errors were encountered:
In #343, we're currently using pickle to save the metadata dataframe. This has the advantage of allowing us to save anything that the user inserted into metadata (including arrays, functions, etc.), but is brittle: pickle sometimes fails to load if the version of e.g., pandas used when saving and loading differs.
Given that I think we don't need the full flexibility of pickle (why would the user insert a function as metadata), we should instead use json to save the dataframe. That would require us to check that whatever metadata value the user sets is json-serializable. It looks like
returns
'{"0":{"0":1},"1":{"0":{}}}'
(converting the lambda function to an empty set), so we'll need to come up with some other way to check whether the object is really json-serializable. We could check thatdf == pd.DataFrame(json.loads(df.to_json()))
, though this might be a bit slow.Also,
to_json
forces ascii by default, so if we want to support unicode characters in metadata, will need to change that.The text was updated successfully, but these errors were encountered: