Incompatible change in internal string representation (encountered with Rust bindings) (?)

[clin1234]

Not sure if it's the right place to ask, but here it is.

I'm attempting to introduce 3.14 support in pyo3 (PyO3/pyo3#4811). I've encountered test failures involving UCS and refcounts targeting Windows and Linux. Invoking cargo test --no-default-features --lib --tests, one example of failure is here: https://github.com/clin1234/pyo3/actions/runs/12834317749/job/35791414368. The Python version used for testing 3.14 is 3.14.0a4.

However, running the same command with Python 3.14.0a3 running on Fedora Rawhide locally, I have encountered no test failures whatsoever.

And yes, I have pinged @davidhewitt on this matter.

[sobolevn]

Hi, thanks for the report! Can you please try to come up with a reproducer that does not include rust, but only C-API?

[davidhewitt]

I don't think it's a CPython problem.

PyO3 has to replicate the C API at the ABI level, so it's not a surprise when bumping python versions that if structures change we need to make adjustments.

The PyUnicode data access macros are a known problem for PyO3 and to be honest I think the better option might be to just remove our mapping of them. But we can discuss that downstream.

[ZeroIntensity]

I suspect the "offending" change would be #128196, but yeah--we won't guarantee any stability with the internals, especially in the alpha phase.

Though, I am having some thoughts about whether or not the structure of PyASCIIObject is actually internal, because we do technically expose the structure in the non-limited API.

cc @encukou for the stability rules here.

[encukou]

By my reading of PEP-387, backwards-incompatible changes to PyASCIIObject needs deprecation period (or SC exception). However, that would be C API changes: rewriting that header in Rust is, unfortunately, not covered by any guarantees.
IMO, CPython should still try to keep more stable than promised. But this is a bit of a tight spot regarding backwards compatibility and free-threading support.

---

I see #128196 preserves C API compatibility but introduces an implementation-defined C feature (short bitfields) to do it. I fear that this will bite us in the future.

[corona10]

By my reading of PEP-387, backwards-incompatible changes to PyASCIIObject needs deprecation period (or SC exception). However, that would be C API changes: rewriting that header in Rust is, unfortunately, not covered by any guarantees.
IMO, CPython should still try to keep more stable than promised. But this is a bit of a tight spot regarding backwards compatibility and free-threading support.

If we have to guarantee object layout stuff, one possible negotiation point would be just change for the free-threaded build because it is kind of an experimental build, so some of the breaking changes will be acceptable. (And we already break ABI compatibility between the default build and the free-threaded build)

cc @colesbury