Garbage collector's warnings while running uvicorn on Debian 11

[AlexanderPodorov]

We're trying to deploy FastAPI application on Linode under Debian 11. We're using SQLAlchemy and asyncpg for DB communication. We use Python 3.10.2.
We're constantly getting GC warnings like that:

The garbage collector is trying to clean up connection <AdaptedConnection <asyncpg.connection.Connection object at 0x7fbfa2b6e6c0>>. This feature is unsupported on async dbapi, since no IO can be performed at this stage to reset the connection. Please close out all connections when they are no longer used, calling ``close()`` or using a context manager to manage their lifetime.
sys:1: SAWarning: The garbage collector is trying to clean up connection <AdaptedConnection <asyncpg.connection.Connection object at 0x7fbfa2b6e6c0>>. This feature is unsupported on async dbapi, since no IO can be performed at this stage to reset the connection. Please close out all connections when they are no longer used, calling ``close()`` or using a context manager to manage their lifetime.

We noticed that every request to the app creates a brand new DB session and that session never gets disposed. So that we quickly run out of connections:

asyncpg.exceptions.TooManyConnectionsError: remaining connection slots are reserved for non-replication superuser connections

We are using the latest versions of FastAPI, SQLAlchemy, asyncpg and uvicorn.
We have a managed postgres DB on Digital Ocean with disabled pgbouncer, because we're using asyncpg connection pool.
We suspect that the problems could be there (this is idiomatic way of getting DB session in FastAPI):

async def get_session(settings: Settings = Depends(get_settings)) -> AsyncIterator[AsyncSession]:
    async with settings.create_session() as session:
        yield session

settings.create_session() essentially calls sessionmaker() and then the session is yielded. Since we're using context manager here async with, the newly created session must be closed automatically at the end.
We run the uvicorn in its simplest form: uvicorn main:app.

We were tested it on Windows 10 Pro and macOS 12.3 Monterey with either the local or remote postgres. Everything worked fine: no GC warnings and DB sessions were properly re-used and we never hit TooManyConnectionsError. But we do get warnings and errors on Debian 11.
We replaced uvicorn with hypercorn on Debian 11 and all issues gone away.
We are not sure is that issue is some kind of uvicorn limitations/issues particularly on Debian or we just missed some required uvicorn configuration. Do you have any suggestions?

[euri10]

os, python version, I think have no impact in your issue (running several instance myself on same os / same python but on aws), it seems to me it's more tied to the way you initialize your db connection.

What is weird though is that you have the issue only with uvicorn and not hypercorn: I don't know precisely the difference we have here in handling lifespan events with hypercorn but typically that's on the startup event you might want to handle the db connection initialization.

Unfortunately I'm not familiar of how to handle that through sqlalchemy, maybe someone can provide another snippet, but with just asyncpg it boils down to something like:

def create_startup(
    _app: FastAPI, app_settings: AppSettings
) -> Callable[[], Coroutine[Any, Any, None]]:
    async def startup() -> None:
        logger.info("Starting database")
        _app.state.dbpool = await asyncpg.create_pool(app_settings.db_uri)
       
    return startup
app.add_event_handler("startup", create_startup(app, app_settings))

the use get_connection in the endpoints

def _get_db_pool(request: Request) -> Pool:
    return request.app.state.dbpool


async def get_connection(
    pool: Pool = Depends(_get_db_pool),
) -> AsyncGenerator[Connection, None]:
    async with pool.acquire() as conn:
        yield conn

hope it helps

[AlexanderPodorov]

Thank you for suggestion!
Unfortunately working with low-level asyncpg API is not an option for us.
SQLAlchemy does that work for us. We create the engine this way:

engine = create_async_engine(
    url=...,
    pool_size=...,
    max_overflow=...,
    pool_timeout=...
)

And we're seeing that this is working properly.
Your get_connection function is essentially the same as the one we're using with SQLAlchemy API.
We just create (or get from the pool) DB session per request, and then the response is sent, we just return this session to the pool.
Yes, we create the engine and sessionmaker on API startup.
We sure that there is no bugs in our code, since it's not working just with Debian 11.

[euri10]

then a minimal reproducible example that triggers the bug is welcomed to go further

[AlexanderPodorov]

@euri10 , we've built a small test app, but unfortunately were not able to reproduce it. It seems the issue is deeper than we thought. I will come back here if I have any thoughts on how to reproduce the issue.

[Kludex]

@AlexanderPodorov Does this still happens on FastAPI 0.74.0+? See https://fastapi.tiangolo.com/release-notes/#0740

[AlexanderPodorov]

As far as we know yes. The latest fastapi version where we didn't have those issues was 0.68.1.
At the time we tested it the latest version was 0.75.0.
We switched to hypercorn several months ago due to this and suspended our research on that.
We decided to check if it is solved somehow periodically.

[AlexanderPodorov]

Sorry, forgot to update everybody here. The issue was related to SQLAlchemy. It works well with SQLAlchemy 2.0.19.

[hulahoo]

Hello!
Faced with the same issue: SQLAlchemy==2.0.23
Fixed error with providing arg to:

async_sessionmaker(close_resets_only=False)

The reason is:

def _close_impl(self, invalidate: bool, is_reset: bool = False) -> None:
        if not is_reset and self._close_state is _SessionCloseState.ACTIVE:
            self._close_state = _SessionCloseState.CLOSED

Here _close_state has three values: ACTIVE, CLOSED, CLOSE_IS_RESET
By default from newly versions, arg close_resets_only by default affects _close_state to CLOSE_IS_RESET , that's why even after calling .close() the state of session instead of CLOSE remains with CLOSE_IS_RESET therefore checks failed and session didn't close.
Hope this help

P.S:
Maybe it works for me, let me know if this helps

[chausler]

This helped us a bunch by throwing an actual exception, with a real traceback from somewhere deep in our code where we were accidentally exiting the connections with context before we were finished using it.

Thanks for the tip, was super helpful

[hulahoo]

I'm glad to help
I've spent 2 hrs to detect what the heck is going :)