RethinkDB: Planning, Deployment, Operation

What is RethinkDB?

RethinkDB is the first open-source, scalable JSON database built from the ground up for the realtime web. It inverts the traditional database architecture by exposing an exciting new access model — instead of polling for changes, the developer can tell RethinkDB to continuously push updated query results to applications in realtime. RethinkDB’s realtime push architecture dramatically reduces the time and effort necessary to build scalable realtime apps.

In addition to being designed from the ground up for realtime apps, RethinkDB offers a flexible query language, intuitive operations and monitoring APIs, and is easy to setup and learn.

When is RethinkDB a good choice?

RethinkDB is a great choice when your applications could benefit from realtime feeds to your data.

The query-response database access model works well on the web because it maps directly to HTTP’s request-response. However, modern applications require sending data directly to the client in realtime. Use cases where companies benefited from RethinkDB’s realtime push architecture include:

• Collaborative web and mobile apps
• Streaming analytics apps
• Multiplayer games
• Realtime marketplaces
• Connected devices

For example, when a user changes the position of a button in a collaborative design app, the server has to notify other users that are simultaneously working on the same project. Web browsers support these use cases via WebSockets and long-lived HTTP connections, but adapting database systems to realtime needs still presents a huge engineering challenge.

RethinkDB is the first open-source, scalable database designed specifically to push data to applications in realtime. It dramatically reduces the time and effort necessary to build scalable realtime apps.

When is RethinkDB not a good choice?

• RethinkDB is not a good choice if you need full ACID support or strong schema enforcement — in this case you are better off using a relational database such as MySQL or PostgreSQL.
• If you are doing deep, computationally-intensive analytics you are better off using a system like Hadoop or a column-oriented store like Vertica.
• In some cases RethinkDB trades off write availability in favor of data consistency. If high write availability is critical and you don’t mind dealing with conflicts you may be better off with a Dynamo-style system like Riak.

System planning before deployment

The RethinkDB documentation should be your first source of information about its requirements. This page serves mostly to document some of its more obscure requirements.

RethinkDB Server will run on any modern OS. Note that the Fedora package isn’t officially supported. Also, official support for Windows is fairly recent (April 2016).

Storage Requirements

When it comes to storage for RethinkDB, there are many things that are nice to have (e.g. SSDs, high-speed input/output [IOPS], replication, reliability, scalability, pay-for-what-you-use), but there are few requirements other than:

1. have enough storage to store all your data (and its replicas), and
2. make sure your storage solution (hardware and interconnects) can handle your expected read & write rates.

For RethinkDB’s failover mechanisms to work, every RethinkDB table must have at least three replicas (i.e. a primary replica and two others). For example, if you want to store 10 GB of unique data, then you need at least 30 GB of storage. (Indexes and internal metadata are stored in RAM.)

As for the read & write rates, what do you expect those to be for your situation? It’s not enough for the storage system alone to handle those rates: the interconnects between the nodes must also be able to handle them.

Storage Notes Specific to RethinkDB

• The RethinkDB storage engine has a number of SSD optimizations, so you can benefit from using SSDs
• RethinkDB tables can have at most 64 shards. What does that imply? Suppose you only have one table, with 64 shards. How big could that table be? It depends on how much data can be stored in each node. If the maximum amount of data that a node can store is d, then the biggest-possible shard is d, and the biggest-possible table size is 64 times that. (All shard replicas would have to be stored on other nodes beyond the initial 64.) If there are two tables, the second table could also have 64 shards, stored on 64 other maxed-out nodes, so the total amount of unique data in the database would be (64 shards/table)×(2 tables)×d. In general, if you have T tables, the maximum amount of unique data that can be stored in the database (i.e. the amount of data before replication) is 64×T×d.
• When you set up storage for your RethinkDB data, you may have to select a filesystem. (Sometimes, the filesystem is already decided by the choice of storage.) We recommend using a filesystem that supports direct I/O (Input/Output). Many compressed or encrypted file systems don’t support direct I/O. The ext4 filesystem supports direct I/O (but be careful: if you enable the data=journal mode, then direct I/O support will be disabled; the default is data=ordered). If your chosen filesystem supports direct I/O and you’re using Linux, then you don’t need to do anything to request or enable direct I/O. RethinkDB does that.
• RethinkDB stores its data in a specific directory. You can tell RethinkDB which directory using the RethinkDB config file, as explained below. In this documentation, we assume the directory is /data. If you set up a separate device (partition, RAID array, or logical volume) to store the RethinkDB data, then mount that device on /data.

Memory (RAM) Requirements

RethinkDB requires data structures in RAM on each server proportional to the size of the data on that server’s disk, usually around 1% of the size of the total data set ( only the data stored on the particular server).

The storage engine is used in conjunction with a custom, B-Tree-aware caching engine which allows file sizes many orders of magnitude greater than the amount of available memory. RethinkDB can operate on a terabyte of data with about ten gigabytes of free RAM. But for reasonable performance, you should probably aim at something closer to 5–10% of the data size. The 1% is the bare minimum and doesn’t include any caching. If you want to run near the minimum, you’ll also need to manually lower RethinkDB’s cache size through the --cache-size parameter to free up enough RAM for the metadata overhead...

RethinkDB automatically configures the cache size limit to be about half the available memory, but it can be no lower than 100 MB. You can manually change the cache size limit (e.g. to free up RAM for queries, metadata, or other things).

If a RethinkDB process (on a server) runs out of RAM, the operating system will start swapping RAM out to disk, slowing everything down. According to @danielmewes:

Going into swap is usually pretty bad for RethinkDB, and RethinkDB servers that have gone into swap often become so slow that other nodes in the cluster consider them unavailable and terminate the connection to them. I recommend adjusting RethinkDB’s cache size conservatively to avoid this scenario. RethinkDB will still make use of additional RAM through the operating system’s block cache (though less efficiently than when it can keep data in its own cache).

Filesystem Requirements

RethinkDB “supports most commonly used file systems” (source) but it has issues with BTRFS (B-tree file system).

It’s best to use a filesystem that supports direct I/O, because that will improve RethinkDB performance (if you tell RethinkDB to use direct I/O). Many compressed or encrypted filesystems don’t support direct I/O.

Deployment RethinkDB

For this tutorial, we will create new RethinkDB cluster with 3 nodes: one proxy node and two data nodes. We also use TLS to secure connections between servers, between servers and clients, and to the web UI.

File structure:

Configure host to use ssl connection:

127.0.0.1 rethinkdb.local

Generate a key and matching certificate

The easiest way to do this is with the openssl command line tool. (Under Linux and OS X, this is already installed; for Windows, you may be able to find precompiled binaries from the list on the OpenSSL wiki.)

First, generate a 2048-bit key and save it to key.pem:

openssl genrsa -out key.pem 2048

Then, generate a certificate, cert.pem, from that key:

openssl req -new -x509 -key key.pem -out cert.pem -days 3650

OpenSSL will ask you to enter information for the certificate. While some of these questions can be left at their default, the “Common Name” must match the domain name of your server. For local testing purposes you can use rethinkdb.local, but not in production.

Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:California
Locality Name (eg, city) []:Mountain View
Organization Name (eg, company) [Internet Widgits Pty Ltd]:RethinkDB
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:rethinkdb.local
Email Address []:

Configuration file for rethinkdb-proxy:

Configuration file for rethinkdb01:

Configuration file for rethinkdb02:

Docker compose:

Start all services:

docker-compose up -d

Access admin dashboard:

Backing Up and Restoring Data

RethinkDB’s Replication as a form of Backup

RethinkDB already has internal replication: every document is stored on R different nodes, where R is the replication factor. Those replicas can be thought of as “live backups” because if one node goes down, the cluster will continue to work and no data will be lost.

At this point, there should be someone saying, “But replication isn’t backup!“

It’s true. Replication alone isn’t enough, because something bad might happen inside the database, and that could affect the replicas. For example, what if someone logged in as a RethinkDB admin and did a “drop table”? We currently plan for each node to be protected by a next-generation firewall (or something similar) to prevent such things from getting very far.

Live Replication of RethinkDB Data Files

That’s just one possible way of setting up the file system so as to provide extra reliability.

Another way to get similar reliability would be to mount the RethinkDB data directory on an Amazon EBS volume. Each Amazon EBS volume is, “automatically replicated within its Availability Zone to protect you from component failure, offering high availability and durability.”

rethinkdb dump (to a File)

RethinkDB can create an archive of all data in the cluster (or all data in specified tables), as a compressed file. According to the RethinkDB blog post when that functionality became available:

Since the backup process is using client drivers, it automatically takes advantage of the MVCC [multiversion concurrency control] functionality built into RethinkDB. It will use some cluster resources, but will not lock out any of the clients, so you can safely run it on a live cluster.

To back up all the data in a BigchainDB cluster, the RethinkDB admin user must run a command like the following on one of the nodes:

rethinkdb dump -e bigchain.bigchain -e bigchain.votes

Backup by Copying RethinkDB Data Files

It’s possible to back up a RethinkDB database by creating a point-in-time copy of the RethinkDB data files (on all nodes, at roughly the same time). It’s not a very practical approach to backup: the resulting set of files will be much larger (collectively) than what one would get using rethinkdb dump, and there are no guarantees on how consistent that data will be, especially for recently-written data.

References:

Frequently asked questions

https://docs.bigchaindb.com/projects/server/en/v1.1.0/appendices/rethinkdb-backup.html