-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix NaNtoMissing #116
base: master
Are you sure you want to change the base?
fix NaNtoMissing #116
Conversation
@@ -269,16 +268,12 @@ function expand_categoricals!(df::DataFrame,categoricals::Array) | |||
return expand_categoricals!(df, categoricalidxs) | |||
end | |||
|
|||
# convert NaNs to NAs | |||
# isnan(x::NAtype) = false | |||
isnan(x::AbstractString) = false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this was type-piracy.
Fixed the problem that it was avoiding via adding isa Number
check below
# convert NaNs to NAs | ||
# isnan(x::NAtype) = false | ||
isnan(x::AbstractString) = false | ||
isnan(x::Union{T, Nothing}) where T = isnan(x.value) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was a bad find and replace from a previous PR that replaced Nullable
with Union{Nothing, T}
It was always nonsense.
idk why it wasn't breaking tests earlier
|
||
# same functionality as above. | ||
function NaNs_to_Missing!(df::DataFrame) | ||
m,n = size(df) | ||
for j=1:n | ||
df[!,j] = [ismissing(df[i,j]) || isnan(df[i,j]) ? missing : value for (i,value) in enumerate(df[:,j])]; | ||
df[!,j] = [(value isa Number && isnan(value)) ? missing : value for value in df[!,j]]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is just a bit simpler we only need the value
, and adds the isa Number
check
bump |
running the tests was erroring with:
Looking at the code for
NaNtoMissing
it was a bit funky.I made it less funky.
Now tests pass without errors for me