Ruby has so many awesome, easily accessible gems, but their ease of use can lull you into a sense of complacency. This is what happened to us at Kenna this past year and we learned the hard way just how important it is to know your gems.

The Problem

This past year at Kenna we sharded our main MySQL database and we choose to do it by client.

So all of a client’s data lives on its own database shard. This means every time we are looking up client data in MySQL, we need to know what database shard to look at. Our sharding configuration,

{
    'client_123' => 'shard_123',
    'client_456' => 'shard_456',
    'client_789' => 'shard_789'
}

which tells us what client belongs on what database shard, needed to be easily accessible because we needed to access it every single time we make a MySQL request.

We first turned to Redis to solve this problem because it was fast, and the configuration hash we wanted to store was not very large. But eventually, that configuration hash grew as we added more and more clients.

13 kilobytes might not seem like a lot of data, but if you are asking for 13 kilobytes millions of times it can add up. Soon we were reading 7.8 MB/second from Redis which was not going to be sustainable as we continued to grow. We had to figure out a way to access our sharding configuration that did not involve Redis if we had any hope of scaling.

Finding a Solution

When we started trying to figure out how to solve this problem, one of the first things we did was take a close look at ActiveRecord’s connection object. This is where ActiveRecord stores all the information it needs to know how to talk to your database. Because of this fact, we thought it might be a good place to find something we could use to help us store our sharding configuration. So we jumped into a console to check it out. What we found was NOT an ActiveRecord connection object at all...

(pry)> ActiveRecord::Base.connection
=> <Octopus::Proxy:0x000055b38c697d10
      @proxy_config= #<Octopus::ProxyConfig:0x000055b38c694ae8

Instead, we found an Octopus Proxy object that our Octopus sharding gem had created.

This was a complete surprise to us! After finding this, we immediately started to dig into the Octopus gem source code to try and figure out what this class was doing. When we finally found that Octopus Proxy class, much to our delight, it had all these great helper methods we could use to access our sharding configuration

module Octopus
    class Proxy
        attr_accessor :proxy_config
        delegate :current_shard, :current_shard=,
                     :current_slave_group, :current_slave_group=,
                     :shard_names, :shards_for_group, :shards, :sharded, 
                     :config, :initialize_shards, :shard_name, to: :proxy_config, prefix: false
    end
end

BOOM! Redis load problem solved! Rather than hitting Redis before each MySQL request, all we simply had to do was reference our local ActiveRecord connection object.

Lesson Learned

One of the big things we learned from this whole experience was how important it is to KNOW YOUR GEMS! It is crazy easy to include a gem in your Gemfile, but when you do, make sure you have a general understanding of how it works.

I am not saying you need to go and read the source code for every one of your gems. That would take you forever to do. But maybe, the next time you add a new gem, consider setting it up manually for the first time in a console. This will allow you to see how it is configured and setup. If we had done this, we would have had a better understanding of how our Octopus sharding gem was configured and we could have saved ourselves this entire Redis headache!

Another great way to learn more about how your gems are working is through logging! Set the logger level to debug for your gem and after interacting with your app, checkout the logs. They will give you some good insights into how that gem is working and interacting with the rest of your code and external services. For more information on using logs, checkout my Live, Log, and Prosper post.

Happy New Year and Happy Coding!

If you are interested in other ways to prevent database hits using Ruby checkout my Cache Is King speech from RubyConf which is what inspired this post.