Repair Cassandra with Reaper

Repairs are a critical anti-entropy operation in Cassandra. In the past, there have been many custom solutions to manage them outside of your main Cassandra installation. K8ssandra Operator provides the Reaper web interface that eliminates the need for a custom solution. Just as K8ssandra Operator makes Cassandra setup easy, Reaper makes configuration of repairs even easier.

Tools

• Web Browser
• kubectl

Prerequisites

• Kubernetes cluster with the following elements deployed:

  • If you haven’t already installed a K8ssandraCluster using K8ssandra Operator, see the local install topic.
  • A K8ssandraCluster with Reaper enabled, see the Deploying Reaper in k8ssandra-operator topic.

• Access to the Reaper web interface requires either:

  • setting up a custom Ingress resource
  • or modifying Reaper’s Kubernetes service as a LoadBalancer (in cloud environments), which will expose it over a public IP
  • or using port forwarding, which is another way to provide external access to resources that have been deployed by K8ssandra Operator in your Kubernetes environment:
    • Developers, see Set up port forwarding.
    • Site reliability engineers, see Configure port forwarding.

Access the Reaper web interface

Check that the pods are running. Example:

kubectl get pods

Output:

NAME                                                         READY   STATUS    RESTARTS   AGE
cass-operator-controller-manager-55f6b84454-zpcfd            1/1     Running   1          10d
k8ssandra-dc1-default-stargate-deployment-7847d945b4-vxth8   1/1     Running   0          10d
k8ssandra-dc1-default-sts-0                                  3/3     Running   0          10d
k8ssandra-dc1-default-sts-1                                  3/3     Running   0          10d
k8ssandra-dc1-default-sts-2                                  3/3     Running   0          10d
k8ssandra-dc1-reaper-58cd6b795b-dw9dw                        1/1     Running   0          10d
k8ssandra-operator-6d4dd9fb8f-5kzl6                          1/1     Running   0          10d

The Reaper interface is secured by default. The default username is <k8c name>-reaper-ui with a randomized password (here it would be k8ssandra-reaper-ui). Read the corresponding secret content to get the password and authenticate to the UI:

kubectl get secret k8ssandra-reaper-ui -o jsonpath='{.data.password}' | base64 --decode

List the services to identify the Reaper service:

kubectl get services

Output:

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                  AGE
k8ssandra-dc1-all-pods-service     ClusterIP   None            <none>        9042/TCP                                                 10d
k8ssandra-dc1-service              ClusterIP   None           <none>         9042/TCP,9142/TCP,8080/TCP,9103/TCP,9000/TCP,9160/TCP    10d
...
k8ssandra-dc1-reaper-service       ClusterIP   10.96.39.160    <none>        8080/TCP,8081/TCP                                        10d
...

Assuming you choose to port forward the Reaper service, use the following command (adjust according to your cluster name and settings):

% kubectl port-forward svc/k8ssandra-dc1-reaper-service 8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

Then, the Reaper UI should be accessible at the following URL: http://127.0.0.1:8080/webui

What can I do in Reaper?

To access Reaper, navigate to http://localhost:8080/webui/.

Check the cluster’s health

In the Reaper UI, notice how the nodes are displayed inside the datacenter for the cluster.

The color of the nodes indicates the overall load the nodes are experiencing at the current moment.

See Check a cluster’s health.

Schedule a cluster repair

On the UI’s left sidebar, notice the Schedule option.

Click Schedules

Click Add schedule and fill out the details when you are done click the final add schedule to apply the new repair job. A Cassandra best practice is to have one repair complete per week to prevent zombie data from coming back after a deletion.

Enter values for the keyspace, tables, owner, and other fields. Then click Add Schedule. The details for adding a schedule are similar to the details for the Repair form, except the “Clause” field is replaced with two fields:

• “Start time”
• “Interval in days”

After creating a scheduled repair, the page is updated with a list of Active and Paused repair schedules.

For additional information, see Schedule a cluster repair on the Reaper site.

Autoschedule repairs

When you enable the autoschedule feature, Reaper dynamically schedules repairs for all non-system keyspaces in a cluster. A cluster’s keyspaces are monitored and any modifications (additions or removals) are detected. When a new keyspace is created, a new repair schedule is created automatically for that keyspace. Conversely, when a keyspace is removed, the corresponding repair schedule is deleted.

To enable autoschedule in Reaper, set the property .spec.reaper.autoScheduling.enabled to true.

Run a cluster repair

On the repair schedule you just configured, click Run now.

Notice the repair job kicking off.

Recommended reading

• Running a cluster repair
• Reaper details
• Blog articles

Next steps

• Explore other K8ssandra Operator tasks.
• See the Reference topics for information about K8ssandra Operator Custom Resource Definitions (CRDs) and the single K8ssandra Operator Helm chart.
• See the Reaper Custom Resource Definition (CRD) reference.