ScyllaDB Docs Scylla Manager Backup Retention Lock

For AI agents: a documentation index is available at https://manager.docs.scylladb.com/branch-3.12/llms.txt. A Markdown version of this page is at https://manager.docs.scylladb.com/branch-3.12/backup/retention-lock.md.

Retention Lock¶

ScyllaDB Manager can protect snapshot files from accidental or malicious deletion by applying object retention locks or event based holds on snapshot files stored in backup bucket. When retention lock is enabled, snapshot files in the backup location cannot be deleted until the retention period expires.

Retention lock is currently supported for Google Cloud Storage only.

How It Works ¶

Object retention lock modes (`unlocked`, `locked`)¶

When object retention lock is enabled on a backup task, ScyllaDB Manager applies object-level retention to all snapshot files from given backup task execution. This includes schema files, SSTable files, and manifest files. The retention lock is applied during dedicated stage (RETENTION_LOCK) that runs after backup is finalized and all snapshot files are already in backup location.

The retention period for each snapshot is calculated from the snapshot creation timestamp contained in the snapshot tag, and lasts for the specified retention days.

This means that a snapshot with --retention-days 30 will have its files protected for exactly 30 days from when the snapshot was taken, regardless of how long the backup task takes to complete.

This approach results in making a per-file request for both newly uploaded and deduplicated files. When making highly deduplicated backups to a colder storage tier (low storage costs, high request costs), consider using event-based-hold mode which avoids making additional requests for deduplicated files.

Event based hold mode (`event-based-hold`)¶

This mode utilizes the following cloud provider features for protecting snapshot files:

Default bucket event based holds - uploaded objects have event based hold set. This hold needs to be removed before protected object can be deleted.
Bucket retention lock - removing event based hold from an object starts specified retention period during which the object can’t be deleted.

ScyllaDB Manager utilizes those bucket features in the following way:

Newly uploaded snapshot files automatically have event based hold applied.
Deduplicated snapshot files keep their hold.
Files referenced by previous snapshot which are not a part of the current one have their event based holds released. This starts their retention period.

The retention period is configured via bucket configuration, not the --retention-days or --retention flags. ScyllaDB Manager won’t attempt to purge stale snapshots according to backup task retention policy, if they are still protected by either event based holds or already started retention period.

The main benefit of this approach is that it makes only a single per-file request and does not repeat those requests for deduplicated files. When making highly deduplicated backups to a colder storage tier (low storage costs, high request costs), it’s possible that the request costs can dominate the overall backup costs. event-based-hold mode aims to reduce costs in such scenarios.

Note that since the holds for previous snapshot are released only during the backup task execution, corresponding files won’t be removed from the backup storage for at least bucket_retention_period + backup_task_interval.

Note that because all objects uploaded to the bucket are subject to the default retention policy, files coming from aborted backups, temporary manifests and permission check files can’t be removed until their holds are released and the retention period expires.

Modes ¶

Retention lock supports the following modes controlled by the sctool backup –retention-lock-mode flag:

disabled (default): No retention lock is applied to snapshot files.
unlocked: Retention lock is applied but can be shortened or removed with special permissions (see Prerequisites).
locked: Retention lock is applied and cannot be overridden. Once set, the lock cannot be removed or shortened, even by the bucket owner.
event-based-hold: While referenced by the newest snapshot, files are protected by default event based holds. After that, they are protected by default retention period.

Override Lock ¶

The sctool backup –override-retention-lock flag allows overriding previously applied retention locks in unlocked mode. This flag is recommended in the following cases:

Changing mode from unlocked to locked: When a previous backup was created with unlocked mode, upgrading to locked mode requires overriding the existing unlocked locks on shared files (see Shared files) that are referenced by both the old and new snapshot.
Decreasing retention period in unlocked mode: Shortening --retention-days requires overriding the previously set, longer retention period on shared files.

Prerequisites ¶

Bucket configuration for unlocked and locked modes

The GCS bucket used as the backup location must have Object Retention enabled. Refer to the Enable and use object retention configurations documentation for instructions on creating a bucket with Object Retention enabled.

Bucket configuration for event-based-hold mode

The GCS bucket used as the backup location must have a default retention policy configured - it defines the protection period of snapshot files (see How It Works). Refer to the Use and lock retention policies for instructions on setting a default retention policy on a bucket.

It is also recommended to enable the default event based hold option on the bucket, so that the initial request setting the hold can be avoided. Refer to the Use object holds for details.

Permissions

The GCS service account used by ScyllaDB Manager Agent must have the following permissions on the backup bucket:

storage.objects.update — required for updating object metadata.
storage.objects.setRetention — required for applying retention locks to snapshot files in unlocked and locked modes.
storage.objects.overrideUnlockedRetention — required when using the --override-retention-lock flag to modify or remove existing locks in unlocked mode.

These permissions are included in the following predefined IAM role:

Storage Object Admin (roles/storage.objectAdmin)

These permissions are in addition to the standard permissions required for backup operations as described in Setup Google Cloud Storage.

Usage ¶

Creating retention lock configuration ¶

You can create a new backup task with retention lock enabled:

sctool backup -c <cluster ID> -L gcs:<bucket> --retention-lock-mode locked --retention-days 30

You can also update an existing backup task to enable retention lock:

sctool backup update -c <cluster ID> <backup task ID> --retention-lock-mode unlocked --retention-days 14

Note that when unlocked or locked retention lock mode is enabled:

--retention-days should be set to a positive value.
Count-based --retention should not be set.

The event-based-hold mode can be combined with any retention policy, but the most intuitive configuration is to set --retention-days to the same value as the bucket’s retention period:

sctool backup -c <cluster ID> -L gcs:<bucket> --retention-lock-mode event-based-hold --retention-days <bucket retention period days>

Changing retention lock configuration ¶

The following guidelines are recommended when updating the retention lock mode or retention period on an existing backup task or when configuring retention lock on multiple backup tasks executed on the same DCs and tables. These constraints are not enforced by server-side validation, but deviating from them may lead to errors during the purge stage (see Shared files):

Avoid changing the mode from locked to unlocked.
When changing from unlocked to locked, use the --override-retention-lock flag (see Override Lock).
In locked mode, --retention-days should only be increased, not decreased.
Decreasing --retention-days in unlocked mode should be accompanied by the --override-retention-lock flag (see Override Lock).

Warning

Changing the retention lock mode between unlocked/locked and event-based-hold (in either direction) is not supported. These approaches to protecting snapshot files are not compatible with each other and require different bucket configurations. To switch between them, create a new backup task pointing to a different backup location configured for the desired mode.

Shared files ¶

ScyllaDB Manager deduplicates SSTables across backups — multiple snapshots (possibly from different backup tasks) can reference the same underlying files. When retention lock is applied to a snapshot, it is also applied to all files referenced by that snapshot, including files shared with other snapshots.

Warning

Avoid configuring multiple backup tasks with different retention lock settings that target the same subset of DCs or tables.

Conflicting retention lock configurations (e.g., different modes or retention periods) across backup tasks that share overlapping files can cause errors during the purge stage.

At the end of each backup task execution, ScyllaDB Manager purges stale snapshots across all registered backup tasks. If any backup task has a retention lock misconfiguration — for example, a lower --retention-days value that conflicts with an existing backup created with a higher --retention-days and --retention-lock-mode=locked — the purge stage will fail for all backup tasks, not just the misconfigured one. Because the purge stage runs after the snapshot upload is complete, the backup data itself is preserved. However, stale snapshots will accumulate, increasing storage consumption and costs.

To resolve such errors, update the --retention-days and --retention-lock-mode flags (along with --override-retention-lock if necessary) on the problematic backup task to use values that are consistent with those of other tasks sharing the same files.

Was this page helpful?