Setting up a Redis Cluster for scalability and high availability

Amazon ElastiCache (Redis OSS) cluster mode setup, configuration, security, and provisioning

In this tutorial, you will learn how to create and configure a Redis Cluster with ElastiCache (Redis OSS) version 7.0 with TLS-encryption enabled. With cluster mode enabled, your Redis Cluster gains enhanced scalability and high availability. You can start small and easily scale your Redis data as your application grows, and by setting up replicas in different availability zones you can also increase your read capacity. By following this tutorial you will also learn some best practices for Redis Cluster deployments.

There's a very rich ecosystem of Redis clients with support for cluster mode features. That makes working with sharded data extremely easy and reliable as clients can keep track of the location of hashslots in a local cache, thus improving the performance of the system as a whole.

The ElastiCache node created in this tutorial is free tier eligible.

Requirements

At some point in this tutorial, you will have to connect to your Redis Cluster in order to experience first hand some of its features. That means you will need access to an EC2 instance with a Redis client installed. If you already have an EC2 instance with a Redis client installed, you can skip to step 1. Otherwise, connect to your Amazon EC2 instance and follow the instructions below to install a Redis client.

If you don't have a C compiler available, install one by running:

$ sudo yum install gcc

Download and compile a Redis client:

$ curl -O http://download.redis.io/redis-stable.tar.gz
$ tar xvzf redis-stable.tar.gz
$ cd redis-stable
$ make distclean  # Ubuntu systems only
$ make

Now you are all set to start the tutorial.

About this Tutorial
Time	10 - 20 minutes
Cost	Free Tier Eligible
Use Case	Scaling, High Availability, Real-time application
Products	AWS ElastiCache (Redis OSS), AWS Free Tier
Audience	Developers
Level	Beginner
Last Updated	June 05, 2019

Step 1: Enter the ElastiCache Dashboard

Open the ElastiCache Dashboard, then:

1.1 — On the top right corner, select the region where you want to launch the Redis Cluster.

1.2 — Click on “Get Started Now”.

Step 2: Create your Redis Cluster

2.1 — Select “Redis” as your Cluster engine.

2.2 — Check “Cluster Mode enabled”.

Step 3: Configure Redis Settings

3.1 — Choose a name for your Redis Cluster, e.g. “elc-tutorial”.

3.2 — Change the Node type to cache.t2.micro. That node type is fine for this tutorial, but for a production cluster the size of the node should depend on your workload and you should start with the m5 or r5 instance families.

3.3 — In Number of Shards, select 3. It means the data will be partitioned in three different master nodes.

3.4 — In Replicas per Shard, select 2. It means each master node will have two replicas. In case of a failure, an automatic failover will be triggered and one of the replicas will take over the role of the master node.

3.5 — Select a Subnet group. If you need more information about Subnet groups, please refer to the documentation.

Step 4: Configure Advanced Redis settings

4.1 — Check the box for “Multi-AZ with Auto-Failover”.

Each master node will be created in a different availability zone, and each of its replicas will also be allocated to a different availability zone. This is a best practice for improved reliability.

Leave the default values for the other fields.

Step 5: Configure the Security settings

For this example we won't use encryption, but keep in mind you can configure both encryption for data at-rest and for data in-transit.

5.1 — Select a Security group for your Redis Cluster.

This is important: make sure the Security group you select allows incoming TCP connections on port 6379. If that's not the case, you won't be able to connect to your Redis nodes.

Step 6: Import data to cluster

For this example, we won't load any seed RDB file so we can skip this configuration step altogether. Just keep in mind that this option is available.

Step 7: Configure backups

Daily backups are important for most use cases, and a good recommendation is to enable backups with a retention period that will give you enough time to act in case anything bad happens. For this tutorial, we won't use any backups.

7.1 — Uncheck “Enable automatic backups”.

Step 8: Maintenance settings

8.1 — Specify a maintenance window that suits your needs.

Here you can think about the time and day when your application has a low workload. For our current needs, we can just state “No preference”.

Step 9: Review and create

After a quick review of all the fields in the form, you can hit “Create”.

9.1 — Click on “Create”.

A Redis Cluster will get initialized and once it becomes “available” you will be able to continue with Step 10.

If you prefer to use the Amazon CLI, the command below will create the Redis Cluster in one go:

If you prefer to use the Amazon CLI, the command below will create the Redis Cluster in one go:

$ aws elasticache create-replication-group \
--replication-group-id elc-tutorial \
--replication-group-description "Tutorial example" \
--num-node-groups 3 \
--cache-node-type cache.t2.micro \
--cache-parameter-group default.redis5.0.cluster.on \
--engine redis \
--engine-version 5.0.3 \
--cache-subnet-group-name sn-value \
--security-group-ids sg-value \
--node-group-configuration \
"ReplicaCount=2,PrimaryAvailabilityZone='us-east-1a',ReplicaAvailabilityZones='us-east-1b','us-east-1c',Slots=0-5460" \
"ReplicaCount=2,PrimaryAvailabilityZone='us-east-1b',ReplicaAvailabilityZones='us-east-1c','us-east-1a',Slots=5461-10921" \
"ReplicaCount=2,PrimaryAvailabilityZone='us-east-1c',ReplicaAvailabilityZones='us-east-1a','us-east-1b',Slots=10922-16383"

For the argument to --security-group-ids, you have to replace sg-value with the ID of the Security Group you want to use.

As for the argument to --cache-subnet-group-name, replace sn-value with the name of a Subnet Group. If you need to create a Subnet Group, use the following command:

$ aws elasticache create-cache-subnet-group \
--cache-subnet-group-name elc-tutorial-subnet \
--cache-subnet-group-description "Tutorial Subnet Group" \
--subnet-ids sn-ids

The value of sn-ids must be a list of subnet IDs.

If you prefer to use the Amazon API, the request below will create a Redis Cluster:

https://elasticache.us-east-1.amazonaws.com/
?Action=CreateReplicationGroup 
&CacheParameterGroup=default.redis5.0.cluster.on
&Engine=redis
&EngineVersion=5.0.3
&ReplicationGroupDescription=Tutorial%20example
&ReplicationGroupId=elc-tutorial
&NumNodeGroups=3
&PrimaryClusterId=elc-primary
&ReplicasPerNodeGroup=2
&Version=<version>
&SignatureVersion=4
&SignatureMethod=HmacSHA256
&Timestamp=<timestamp>
&X-Amz-Credential=<credential>

In the previous request you need to provide values for Version, Timestamp, and X-Amz-Credential.

Step 10: Connect to Redis

Find the configuration endpoint of your Redis Cluster.

10.1 — Click on the arrow to display the Redis Cluster details.

10.2 — Copy the Configuration Endpoint.

In the examples, each time an endpoint is mentioned you should use the hostname of your Configuration Endpoint.

10.4 — Use the redis-cli utility to connect to your Redis node.

$ ./src/redis-cli -c -h endpoint

10.5 — Test the connection with a PING.

endpoint:6379> PING
	PONG

Step 11: Trigger a failover

Now you have a working and healthy Redis Cluster. One of the features of the cluster mode is the fact that if a node dies, the cluster can heal itself. In order to test that claim, you can trigger a manual failover and the following will happen: a read-replica will be selected to take over the the role of the master, and once the failover is executed you will be able to connect to the new master. In the meantime, a new replica will be added automatically so that the cluster will still have one master and two replicas.

Using the endpoint from step 10, check the role of the node.

11.1 — Check the role of the endpoint.

	endpoint:6379> ROLE
	1) "master"
 	...

You want to connect to the master. If you didn't connect to a master, try with a different endpoint. You have three chances to get it right.

Once you have located the master node:

11.2 — Select any of the nodes, click on “Action” and select “Failover primary”.

Read the message to understand what's going on, and then click “Continue”. One of the replicas will become the new master, and the number of replicas per master will be restored. You can run the CLUSTER NODES command to verify what is happening.

Step 12: Delete your cluster

To finish this experiment, you will learn how to delete your Redis Cluster when it's not needed anymore.

In order to delete your Redis Cluster, go to the ElastiCache Dashboard and follow these instructions:

12.1 — Select “Redis” on the left pane.

This will show you a list of all your Redis clusters.

12.2 — Select the Redis Cluster you created for this tutorial.

12.3 — Click on “Delete”.

12.4 — You will be asked if you want to create a final backup. That's usually a good idea, but it's not necessary for this tutorial. Select “No” and click on “Delete”.

The status of your cluster will change to “deleting”.

Congratulations

You have created a Redis Cluster with cluster mode enabled. The nodes were spread across availability zones and configured with automatic failovers. You have also learnt about some best practices for Redis Cluster deployments.

Was this tutorial helpful?

Feedback

Recommended next steps

Learn more

If you want to know more about Redis Cluster, read the Redis Cluster 101 blog post. For more getting started content, developer guide, and use-case technical blogs, visit our resources page.

Choose the right node size

Determining the size of your Redis nodes may seem challenging, but we have created some guidelines to help you choose the right node size based on your needs.

Caching strategies

Finally, you can read more about caching strategies and best practices to improve performance and reliability.