Rails::Sharding

Simple and robust sharding gem for Rails, including Migrations and ActiveRecord extensions

This gems allows you to easily create extra databases to your rails application, and freely allocate ActiveRecord instances to any of the databases.

Accessing shards is as simple as:

  # creating a user to a specific shard
  new_user = User.using_shard(:shard_group1, :shard1).create(username: 'x')

  # retrieving a user from a specific shard
  loaded_user = User.using_shard(:shard_group1, :shard1).where(username: 'x').first

You can also use the block syntax:

  Rails::Sharding.using_shard(:shard_group1, :shard1) do
    # All statements inside this block will go to the selected shard

    # Do some queries
    new_user = User.create(username: 'x')
    loaded_user = User.where(username: 'x').first
    billing_infos = loaded_user.billing_infos.all
  end

You can also pick and choose which models will be shardable. Non shardable models will be retrieved from the master database, even if inside a using_shard block.

Compatibility

Gem version 1.x.x:

• Rails 5.0, 5.1 and 5.2
• Databases: MySQL, MariaDB, Postgres

Gem version 0.x.x:

• Rails 4.2
• Databases: MySQL, MariaDB

Installation

Add this line to your application's Gemfile:

gem 'rails-sharding'

And then execute:

bundle

Creating Shards

To start with the rails-sharding gem, run the command

rails g rails_sharding:scaffold

This will generate a config/shards.yml.example like this:

default: &default
  adapter: mysql2
  encoding: utf8
  reconnect: false
  pool: 5
  username: <%= ENV["MYSQL_USERNAME"] %>
  password: <%= ENV["MYSQL_PASSWORD"] %>
  socket: /var/run/mysqld/mysqld.sock

development:
  shard_group1:
    shard1:
      <<: *default
      database: group1_shard1_development
    shard2:
      <<: *default
      database: group1_shard2_development
...

Rename it to config/shards.yml and change it to your database configuration. This example file defines a single shard group (named shard_group1) containing two shards (shard1 and shard2).

A shard group is a set of shards that should have the same schema.

When you're ready to create the shards run

rake shards:create

Migrating Shards

Go to the directory db/shards_migrations/shard_group1 and add all migrations that you want to run on the shards of shard_group1. By design, all shards in a same group should always have the same schema.

As of now, there is no generator for migrations. You can use the regular rails generator and move the migrations to the shards_migration folder.

For example, add the following migration to your db/shards_migrations/shard_group1:

# 20160808000000_create_users.rb
class CreateClients < ActiveRecord::Migration[5.0]
  def up
    create_table :users do |t|
      t.string :username, :limit => 100
      t.timestamps
    end
  end

  def down
    drop_table :users
  end
end

Then run:

rake shards:migrate

All the shards will be migrated, and one schema file will be dumped for each of the shards (just like rails would do for your master database). You can see the schema of the shards in db/shards_schemas/shard_group1/, and it will be something like:

ActiveRecord::Schema.define(version: 20160808000000) do

  create_table "users", force: :cascade do |t|
    t.string   "username",       limit: 100
    t.datetime "created_at"
    t.datetime "updated_at"
  end

end

Other rake tasks

The rails-sharding gem offers several rake tasks analogous to the ones offered by ActiveRecord:

rake shards:create
rake shards:drop
rake shards:migrate
rake shards:migrate:down
rake shards:migrate:redo
rake shards:migrate:reset
rake shards:migrate:up
rake shards:rollback
rake shards:schema:dump
rake shards:schema:load
rake shards:test:load_schema
rake shards:test:prepare
rake shards:test:purge
rake shards:version

They work just the same as the tasks rake:db:... but they operate on all shards of all shard groups. If you want to run a rake task just to a specific shard group or shard you can use the SHARD_GROUP and SHARD options:

rake shards:migrate SHARD_GROUP=shard_group_1
rake shards:migrate SHARD_GROUP=shard_group_1 SHARD=shard1

Gem Options

Running the rails g rails_sharding:scaffold will create an initializer at config/initializers/rails-sharding.rb. You can pass additional configurations on this initializer to control the gem behavior. You can see below all available options and their default values:

# config/initializers/rails-sharding.rb

Rails::Sharding.setup do |config|
  # If true one connection will be established per shard (in every shard group) on startup.
  # If false the user must call Shards::ConnectionHandler.establish_connection(shard_group, shard_name) manually at least once before using each shard.
  config.establish_all_connections_on_setup = true

  # If true the method #using_shard will be mixed in ActiveRecord scopes. Put this to false if you don't want the gem to modify ActiveRecord
  config.extend_active_record_scope = true

  # If true the query logs of ActiveRecord will be tagged with the corresponding shard you're querying
  config.add_shard_tag_to_query_logs = true

  # Specifies where to find the definition of the shards configurations
  config.shards_config_file = 'config/shards.yml'

  # Specifies where to find the migrations for each shard group
  config.shards_migrations_dir = 'db/shards_migrations'

  # Specifies where to find the schemas for each shard group
  config.shards_schemas_dir = 'db/shards_schemas'
end

Wiki

Want to know more? How to integrate with RSpec, Capistrano, etc? Take a look at our wiki.

Development and Contributing

After checking out the repo:

1. Run bundle to install gems

2. Create your spec/fixtures/shards.yml based on the example on this same folder (you need MySQL and Postgres)

3. Create your .env based on the example on this same folder.

4. Run rake db:test:prepare to create the test shards.

5. Run rspec to run the tests.

Bug reports and pull requests are welcome on GitHub at https://github.com/hsgubert/rails-sharding.

License

The gem is available as open source under the terms of the MIT License.

Acknowledgements

This gem was inspired on several other gems like: octopus, shard_handler and active_record_shards.