Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resample/interpolate doesn't handle columns with dot in name #361

Open
ghormann opened this issue Sep 9, 2023 · 1 comment
Open

resample/interpolate doesn't handle columns with dot in name #361

ghormann opened this issue Sep 9, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@ghormann
Copy link

ghormann commented Sep 9, 2023

ISSUE

While maybe not the best practice, databricks does allow columns to have "." in the name. When doing a resample with interporlate, this results in a cannot resolve column name error

How to reproduce

  1. Create a TSDF with columns that include a "."
    image

  2. Attempt to resample and interporlate with

resample_tsdf = base_tsdf.resample(freq="30 seconds", func="mean").interpolate(method="ffill")

An error is produced

AnalysisException: Cannot resolve column name "Bundler.Status.CurMachSpeed" among (site, line, ts, Bundler.Status.CurMachSpeed, Bundler.Status.MachSpeed, agg_key); did you mean to quote the Bundler.Status.CurMachSpeed column?

Workaround

Rename the columns before resampling / interpolate

@tnixon tnixon added the bug Something isn't working label Sep 12, 2023
@tnixon
Copy link
Contributor

tnixon commented Sep 12, 2023

Thanks for bringing this to our attention @ghormann - we'll look into it and see if we can get a fix out soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants