You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe. ToUnixTimestamp and GetTimestamp or more operators require a format parameter, and they are time zone aware operators. Refer to Spark link
val formatter = formatterOption.getOrElse(getFormatter(fmt.toString))
formatter.parse()
I think our GPU implemetation currently does not support non-utc TZ:
def parseStringAsTimestamp(
When TZ is Asia/Shanghai, to_timestamp("1970-01-01 00:00:00", "yyyy-MM-dd HH:mm:ss") get negative 8 hours instead of zero.
Describe the solution you'd like
Expose the timeparts structure, refer to the following code.
Then rebase local time in a time zone to UTC time. Alfred is woring on this.
Update:
After sync-up with @NVnavkumar.
Maybe we have a simple solution via GpuTimeZoneDB directly.
We first get the microseconds from 1970-01-01 00:00:00 in UTC which are called instants.
Then use GpuTimeZoneDB to rebase the microseconds according to the timezone.
For example: Parse("1970-01-01 00:00:00", 'yyyy-MM-dd HH:mm:ss') when session time zone is Aisa/Shanghai.
First get the microseconds, here it's 0. This means get microseconds from "1970-01-01 00:00:00" in UTC TZ.
Then use GpuTimeZoneDB.fromUTC(cv, tz) or GpuTimeZoneDB.toUTC(cv, tz) to rebase the microseconds to what we want.
Is your feature request related to a problem? Please describe.
ToUnixTimestamp
andGetTimestamp
or more operators require a format parameter, and they are time zone aware operators. Refer to Spark linkI think our GPU implemetation currently does not support non-utc TZ:
When TZ is
Asia/Shanghai
,to_timestamp("1970-01-01 00:00:00", "yyyy-MM-dd HH:mm:ss")
get negative 8 hours instead of zero.Describe the solution you'd like
Expose the
timeparts
structure, refer to the following code.Then rebase local time in a time zone to UTC time. Alfred is woring on this.
https://github.com/rapidsai/cudf/blob/v24.02.00a/cpp/src/strings/convert/convert_datetime.cu#L399-L401
We have limited supported format
We can add more supported format in the future.
Or we can put the code into this PR: Add timestamp parser to parse timestamp string with time zone #1539
legacy format: link java.text.SimpleDateFormat
new format: java.time.format.DateTimeFormatter
When
spark.sql.legacy.timeParserPolicy
is EXCEPTION, refer to linkThe text was updated successfully, but these errors were encountered: