Datetimes all become NaT

[wilcovanvorstenbosch]

I'm trying to synthesize a dataset with various columns for dates.
I'm using the Gaussian Copula Synthesizer, and the following transformer for these columns:

OptimizedTimestampEncoder(
            enforce_min_max_values=True,
            missing_value_replacement='random',
            missing_value_generation='from_column',
            datetime_format=None
        )

The columns are of type <M8[ns] or datetime64[ns]. They already contain a lot of missing values (NaT).
After synthesis, the sampled data contains ONLY missing values (NaT).

The dates are of format yyyy-mm-dd. I tried setting the date_time format to '%Y-%m-%d' , but to no avail.

What did I do wrong?

[srinify]

Hi @wilcovanvorstenbosch 👋

1. Out of curiosity, does the NaT issue disappear if you use the default transformer, without updating to OptimizedTimestampEncoder? I'd be curious to know if the default workflow results in a roughly similar proportion of NaT values in the synthetic data.

2. I'm also curious to know if you're hoping to generate synthetic data with the same datetime format as your real data, or if you're hoping to generate synthetic data with a different one? In SDV land, the datetime_format value should describe the format in your real data, not the aspirational format for the synthetic data.

3. Do you also mind sharing your SDV code? This would include how you're instantiating the synthesizer, customizing the synthesizer's behavior using parameters and methods, any constraints you're using, etc.