v4.0.0 Brainstorming: Concurrency, Thread/Fiber Safety, and Multi-DB Support

[mnovelo]

Hi everyone,

For v4, I’d like to focus on improving concurrency and thread/fiber safety in the Apartment gem. With Rails supporting multiple databases and async queries, there are increasing demands on how we manage tenants across threads, fibers, and connections. I believe v3 has significant limitations in this regard, particularly with connection pooling and multi-threaded contexts.

Current Challenges

1. Concurrency Issues Across Databases

• Schema-based tenancy (PostgreSQL):
  In PostgreSQL, we rely heavily on SET search_path to manage tenants. However, we lack a robust way to track or reset the search_path per connection, causing issues with tools like RDS Proxy (#302). This also creates problems in tests where shared connections across threads lead to unpredictable tenant state (#239).

• Database-based tenancy (MySQL):
  For MySQL, where each tenant is its own database, switching tenants frequently creates new connections, which hampers connection pooling efficiency. This is especially problematic in setups with read replicas (#275).

2. Async Query Support

Rails 7’s load_async queries fail to respect the tenant context when executed on async thread pools, resulting in queries falling back to the public schema (#199).

3. ActionController::Live

Threads spawned by ActionController::Live bypass tenant state, requiring manual intervention to maintain the tenant context (#304).

4. Schema Dumping

The schema.rb dump currently excludes create_schema calls for non-tenant schemas, which is frustrating for apps with shared or analytics schemas (#303).

---

Proposed Direction

1. Leverage ActiveSupport::CurrentAttributes

• Use CurrentAttributes to track the “current tenant” uniformly across threads and fibers.
• Track tenant state within each connection, ensuring new connections inherit the correct tenant.

2. Integrate with Rails Connection Pooling

• Enhance Apartment to work seamlessly with Rails’ connection pool by overriding key methods in ActiveRecord::ConnectionHandling and ActiveRecord::Core.
• For schema-based tenancy, reset and manage search_path per connection.
• For database-based tenancy, manage tenant state efficiently during connection creation and pooling.

3. Support Async Queries and ActionController::Live

• Ensure tenant state is propagated to threads spawned for load_async and ActionController::Live.

4. Configurable Schema Dumping

• Add a configuration option to include non-tenant schemas in schema.rb dumps.

---

Feedback Requested

I’m open to thoughts on the following:

• Are there better ways to manage tenant state across threads, fibers, and connections?
• How should we prioritize these issues in v4?
• Are there edge cases or additional concerns that need addressing?

Let me know what you think or if you’d like to collaborate on specific areas!

[ryanb]

@mnovelo This sounds great to me.

Enhance Apartment to work seamlessly with Rails’ connection pool by overriding key methods in ActiveRecord::ConnectionHandling and ActiveRecord::Core.

If methods need to be overridden, perhaps hooks can be added to Active Record to make this part of the API. Worth raising a Rails issue for it anyway.

Are there better ways to manage tenant state across threads, fibers, and connections?

If the tenant ever has a risk of leaking outside its scope then that can be a disastrous bug leading to huge security issues. I don't know how the Apartment gem works internally, but I'd love to see additional guard rails in place to prevent this from happening. Maybe check ActiveRecord::Base.connection.schema_search_path before every query if not done already?

Ensure tenant state is propagated to threads spawned for load_async and ActionController::Live.

Instead of solving load_async and ActionController::Live directly, I wonder if there's something that can be done at the lower level to solve both. My thinking is there may be other cases that pop up through Rails or gems and if both can be solved through one solution then it may solve those cases too.

[mnovelo]

I appreciate your feedback, @ryanb ! Here’s where my current thinking aligns with your points:

Hooks in Active Record

I’m particularly interested in the checkin and checkout callbacks defined for ActiveRecord::ConnectionAdapters::AbstractAdapter. My current idea is to ensure every connection is aware of the tenant it’s tied to, which would allow us to validate and reset state as needed during connection lifecycle events. I’ll explore whether extending these hooks or proposing new ones in Rails would be beneficial.

Guard Rails for Tenant Leaks

I completely agree that tenant leaks pose a serious security risk, and it’s something we’re highly cautious about as well. However, we also need to balance this against performance concerns. For example, I recommend turning off tenant_presence_check in production due to its overhead.

Another limitation is that while we can prevent accidental tenant context leaks, we can’t stop queries that explicitly target other tenants, as mentioned in rails/rails#43500. This means schema-level locks or checks aren’t a complete solution either. I’d love to hear if you have ideas for lightweight safeguards that could complement these limitations without adding too much overhead.

A Unified Solution for Async Queries and Live Controllers

I think you’re right that a lower-level solution for propagating tenant state across threads and fibers could address most use cases. One caveat is ActionController::Live, which I think bypasses the controller middleware and likely needs a custom mechanism to set the tenant for the current request. However, once the tenant is set (whether for a live request or an async job), all connections should inherit that tenant state unless explicitly switched. I’ll explore if there’s a unified way to handle both cases while keeping room for edge cases like ActionController::Live.

---

Again, I really appreciate your insights—they help sharpen the direction for these features!

[macfanatic]

This is a great overview and I think a great direction for a solid v4 that would be an awesome update.

Do you have any thoughts on versions of rails itself you would want or need to continue to support, to also help drive any API or dependency decisions? If you wanted to support for example only rails 7.0 and newer, or rails 6.0 or newer, that could drive some decisions perhaps.

My other initial thought would be that indeed using ActiveSupport::CurrentAttributes would be the current rails way of thread specific state, and build upon some newer community standards. I don't imagine it would help solve #304 directly, perhaps indeed a new rails API to expose hooks would be helpful here to modify behavior upon spawn, but without that, the workaround listed in the issue by the author would be the best solution indeed, that we could build upon.

[mnovelo]

Thanks for the kind words and thoughtful questions @macfanatic!

Supported Versions of Rails and Ruby

For v4, I think we should align with Rails' new maintenance policy (Rails Maintenance Policy) by supporting the last three major versions of Rails that are actively maintained. Since Rails 7.0.x will only receive security fixes until April 1st, 2025, I’m comfortable dropping support for it in v4. This means v4 will support:

• Rails versions: 8.0.x, 7.2.x, and 7.1.x
• Ruby versions: 3.2, 3.3, and 3.4

With Rails now releasing a new version every six months, this feels like the right time to refactor Apartment in a way that keeps pace with these updates. Supporting the three most recent major versions ensures we can take advantage of new Rails features without being constrained by legacy compatibility.

ActiveSupport::CurrentAttributes and ActionController::Live

I agree that ActiveSupport::CurrentAttributes is the Rails-standard approach for thread- and fiber-local state and will hopefully provide a solid foundation for managing tenant state. As you noted, it wouldn’t directly address the issues with ActionController::Live because it appears to not use the standard controller middleware so I agree that that we'll still need a workaround similar to the one proposed in #304, where the tenant is explicitly set for the live request.

[macfanatic]

All of this sounds great to me 👍