ScyllaDB University Live | Free Virtual Training Event
Learn more
ScyllaDB Documentation Logo Documentation
  • Server
    • ScyllaDB Open Source
    • ScyllaDB Enterprise
    • ScyllaDB Alternator
  • Cloud
  • Tools
    • ScyllaDB Manager
    • ScyllaDB Monitoring Stack
    • ScyllaDB Operator
  • Drivers
    • CQL Drivers
    • DynamoDB Drivers
Download
Menu
ScyllaDB Docs Scylla Manager Repair

Repair¶

  • Features

  • Parallel repairs

  • Repair intensity

  • Changing repair speed

  • Repair order

Repair is important to make sure that data across the nodes is consistent. To learn more about repairs please consult this Scylla University lesson.

Scylla Manager automates the repair process and allows you to configure how and when repair occurs. Scylla Manager repair task revolves around scheduling many Scylla repair jobs with selected --intensity in --parallel. Repair task is responsible for fully repairing all tables selected with --keyspace parameter, while a single repair job repairs chosen (by Scylla Manager) token ranges of a given table owned by a specific replica set. All nodes from this replica set take part in the repair job and any node can take part only in a single repair job at any given time.

When you create a cluster a repair task is automatically scheduled. This task is set to occur each week by default, but you can change it to another time, change its parameters or add additional repair tasks if needed.

Features¶

  • Glob patterns to select keyspaces or tables to repair

  • Parallel repairs

  • Control over repair intensity and parallelism even for ongoing repairs

  • Repair order improving performance and stability

  • Resilience to schema changes

  • Retries

  • Pause and resume

Parallel repairs¶

Each node can take part in at most one Scylla repair job at any given moment, but Scylla Manager can repair distinct replica sets in a token ring in parallel. This is beneficial for big clusters. For example, a 9 node cluster and a keyspace with replication factor 3, can be repaired up to 3 times faster in parallel. The following diagram presents a benchmark results comparing different parallel flag values. In a benchmark we ran 9 Scylla 2020.1 nodes on AWS i3.2xlarge machines under 50% load, for details check this blog post

../_images/parallel.png

By default Scylla Manager runs repairs with full parallelism, you can change that using sctool repair –parallel flag.

Repair intensity¶

Intensity specifies how many token ranges can be repaired in a Scylla node at every given time. The default intensity is one, you can change that using sctool repair –intensity flag.

Scylla Manager 2.2 adds support for intensity value zero. In that case the number of token ranges is calculated based on node memory and adjusted to the Scylla maximal number of ranges that can be repaired in parallel (see max_repair_ranges_in_parallel in Scylla logs). If you want to repair faster, try using intensity zero.

Note that the less the cluster is loaded the more it makes sense to increase intensity. If you increase intensity on a loaded cluster it may not give speed benefits since cluster have no resources to process more repairs. In our experiments in a 50% loaded cluster increasing intensity from 1 to 2 gives about 10-20% boost and increasing it further will have little impact.

Changing repair speed¶

Repair speed is controlled by two parameters: --parallel and --intensity Those parameters can be set when you:

  • Schedule a repair with sctool repair

  • Update a repair specification with sctool repair update

  • Update a running repair task with sctool repair control

More on the topic of repair speed can be found in Repair faster and Repair slower articles.

Repair order¶

Scylla Manager repairs keyspace by keyspace and table by table in order to achieve greater repair stability and performance.

Keyspaces and tables are ordered according to the following rules:

  • repair internal (with system prefix) tables before user tables

  • repair base tables before Materialized Views and Secondary Indexes

  • repair smaller keyspaces and tables first

Note

Ensuring that base tables are repaired before views is possible only when Scylla Manager has CQL credentials to repaired cluster.

Was this page helpful?

PREVIOUS
Compatibility Matrix
NEXT
Repair faster
Scylla Manager
  • 3.2
    • master
    • 3.2
    • 3.1
    • 3.0
    • 2.6
    • 2.5
    • 2.4
    • 2.3
    • 2.2
  • Docker
  • Install ScyllaDB Manager
  • Install ScyllaDB Manager Agent
  • ScyllaDB Manager Upgrade
  • Add a Cluster
  • Backup
    • Setup Amazon S3
    • Setup S3 compatible storage
    • Setup Google Cloud Storage
    • Setup Azure Blob Storage
    • Examples
    • Specification
  • Restore
    • Restore tables
    • Restore schema
    • Examples
    • Compatibility Matrix
  • Repair
    • Repair faster
    • Repair slower
    • Examples
  • Health Check
  • CLI sctool
    • Environment variables
    • Completion
    • Download files
    • Backup
    • Restore
    • Cluster
    • Info
    • Progress
    • Repair
    • Start
    • Status
    • Stop
    • Suspend & Resume
    • Tasks
    • Version
  • Configuration Files
    • Scylla Manager Config
    • Scylla Manager Agent Config
  • Swagger File
  • ScyllaDB Monitoring
  • Troubleshooting
  • Slack
  • Create an issue
  • Edit this page

On this page

  • Repair
    • Features
    • Parallel repairs
    • Repair intensity
    • Changing repair speed
    • Repair order
Logo
Docs University Contact Us About Us
Mail List Icon Slack Icon Forum Icon
© 2023, ScyllaDB. All rights reserved.
Last updated on 26 September 2023.
Powered by Sphinx 4.3.2 & ScyllaDB Theme 1.5.1