ScyllaDB Manager Upgrade¶

Stop all Scylla Manager tasks (or wait for them to finish)¶

On the Manager Server check current status of the manager tasks:

sctool task list -c <cluster>

Or on versions 3.0 and higher using:

sctool tasks -c <cluster>

None of the listed tasks should have status in RUNNING.

Stop the ScyllaDB Manager Server¶

On the Manager Server instruct Systemd to stop the server process:

sudo systemctl stop scylla-manager

Ensure that it is stopped with:

sudo systemctl status scylla-manager

It should have a status of “Active: inactive (dead)”.

Stop the ScyllaDB Manager Agent on all nodes¶

On each scylla node in the cluster run:

sudo systemctl stop scylla-manager-agent

Ensure that it is stopped with:

sudo systemctl status scylla-manager-agent

It should have a status of “Active: inactive (dead)”.

Upgrade the ScyllaDB Manager Server and Client¶

1. On the Manager server, update the Manager repo file. Go to ScyllaDB Manager in the ScyllaDB Download Center, and select your platform and the version to which you want to upgrade to display the relevant command.

   The following examples show how to update the repo for Manager 3.1:

   Example for CentosExample for Ubuntu

   sudo curl -o /etc/yum.repos.d/scylla-manager.repo -L http://downloads.scylladb.com/rpm/centos/scylladb-manager-3.1.repo
   

   Note

   You don’t need to update the repo file if you upgrade to a patch release, for example, from Manager 3.1.1 to 3.1.2.

   You can display the contents of the Manager repo file to confirm that the displayed version is the version to which you want to upgrade.

   • On CentOS/Red Hat, run: cat /etc/yum.repos.d/scylla-manager.repo

   • On Debian/Ubuntu, run: cat /etc/apt/sources.list.d/scylla-manager.list

2. On the Manager server, instruct the package manager to update the server and the client:

   CentOS/ Red HatDebian/ Ubuntu

   sudo yum update scylla-manager-server scylla-manager-client -y
   

Upgrade the ScyllaDB Manager Agent on all nodes¶

On each scylla node instruct package manager to update the agent:

CentOS, Red Hat:

sudo yum update scylla-manager-agent -y

Debian, Ubuntu:

sudo apt-get update
sudo apt-get install scylla-manager-agent -y

Run scyllamgr_agent_setup script on all nodes¶

This step requires sudo rights:

$ sudo scyllamgr_agent_setup
Do you want to create scylla-helper.slice if it does not exist?
Yes - limit Scylla Manager Agent and other helper programs memory. No - skip this step.
[YES/no] YES
Do you want the Scylla Manager Agent service to automatically start when the node boots?
Yes - automatically start Scylla Manager Agent when the node boots. No - skip this step.
[YES/no] YES

First step relates to limiting resources that are available to the agent and second instructs systemd to run agent on node restart.

Reconcile configuration files¶

Upgrades can create changes to the structure and values of the default yaml configuration file. If the previous version’s configuration file was modified with custom values, this could result in a conflict. The upgrade procedure can’t resolve this without help from an administrator. If you followed instructions from the upgrade packages sections of this document, and you elected to save both the new and old configuration files, the new version of the configuration file is saved in the same directory as the old one with an added extension suffix for both server and agent. These files are stored in the /etc/scylla-manager directory.

On a CentOS configuration, a conflict looks like:

# On the Scylla Manager node
/etc/scylla-manager/scylla-manager.yaml # old file containing custom values
/etc/scylla-manager/scylla-manager.yaml.rpmnew # new default file from new version
# On all Scylla nodes
/etc/scylla-manager-agent/scylla-manager-agent.yaml # old file containing custom values
/etc/scylla-manager-agent/scylla-manager-agent.yaml.rpmnew # new default file from new version

On an Ubuntu configuration, a conflict looks like:

# On the Scylla Manager node
/etc/scylla-manager/scylla-manager.yaml # old file containing custom values
/etc/scylla-manager/scylla-manager.yaml.dpkg-dist # new default file from new version
# On all Scylla nodes
/etc/scylla-manager-agent/scylla-manager-agent.yaml # old file containing custom values
/etc/scylla-manager-agent/scylla-manager-agent.yaml.dpkg-dist # new default file from new version

It is required to manually inspect both files and reconcile old values with the new configuration. Remember to carry over any custom values like database credentials, backup, repair, and any other configuration. This can be done by manually updating values in the new config file and then renaming files:

For CentOS:

# On the Scylla Manager node
cd /etc/scylla-manager/
mv scylla-manager.yaml scylla-manager.yaml.old  #renames the old config file as old
mv scylla-manager.yaml.rpmnew scylla-manager.yaml
# On all Scylla nodes
cd /etc/scylla-manager-agent/
mv scylla-manager-agent.yaml scylla-manager-agent.yaml.old
mv scylla-manager-agent.yaml.rpmnew scylla-manager-agent.yaml

For Ubuntu:

# On the Scylla Manager node
cd /etc/scylla-manager/
mv scylla-manager.yaml scylla-manager.yaml.old
mv scylla-manager.yaml.dpkg-dist scylla-manager.yaml
# On all Scylla nodes
cd /etc/scylla-manager-agent/
mv scylla-manager-agent.yaml scylla-manager-agent.yaml.old
mv scylla-manager-agent.yaml.dpkg-dist scylla-manager-agent.yaml

Start the ScyllaDB Manager Agent on all nodes¶

On each scylla node instruct Systemd to start the agent process:

sudo systemctl start scylla-manager-agent

Ensure that it is running with:

sudo systemctl status scylla-manager-agent

It should have a status of “Active: active (running)”.

Start the ScyllaDB Manager Server¶

On the Manager Server instruct Systemd to start the server process:

sudo systemctl daemon-reload
sudo systemctl start scylla-manager

Ensure that it is started with:

sudo systemctl status scylla-manager

It should have a status of “Active: active (running)”.

Validate status of the cluster¶

On the Manager Server check the version of the client and the server:

sctool version
Client version: 3.x.y-0.20200123.7cf18f6b
Server version: 3.x.y-0.20200123.7cf18f6b

Check that cluster is up:

sctool status -c <cluster>

All running nodes should be up.

Stop all ScyllaDB Manager tasks (or wait for them to finish)¶

On the Manager Server check current status of the manager tasks:

sctool tasks -c <cluster>

None of the listed tasks should have status in RUNNING.

Stop the ScyllaDB Manager Server¶

On the Manager Server instruct Systemd to stop the server process:

sudo systemctl stop scylla-manager

Ensure that it is stopped with:

sudo systemctl status scylla-manager

It should have a status of “Active: inactive (dead)”.

Stop the ScyllaDB Manager Agent on all nodes¶

On each scylla node in the cluster run:

sudo systemctl stop scylla-manager-agent

Ensure that it is stopped with:

sudo systemctl status scylla-manager-agent

It should have a status of “Active: inactive (dead)”.

Downgrade the ScyllaDB Manager Server and Client¶

On the Manager Server instruct package manager to downgrade server and the client:

CentOS, Red Hat:

sudo yum downgrade scylla-manager-server-2.x* scylla-manager-client-2.x* -y

Debian, Ubuntu:

sudo apt-get install scylla-manager-server=2.x scylla-manager-client=2.x -y

Downgrade the ScyllaDB Manager Agent on all nodes¶

On each scylla node instruct package manager to downgrade the agent:

CentOS, Red Hat:

sudo yum downgrade scylla-manager-agent-2.x* -y

Debian, Ubuntu:

sudo apt-get install scylla-manager-agent=2.x -y

Revert to the old configuration¶

If you followed instructions from the Upgrade Steps section and you had configuration conflict when upgrading, then listing the configuration directory should give you both new and old configuration:

/etc/scylla-manager/scylla-manager.yaml # New version that you want to disable
/etc/scylla-manager/scylla-manager.yaml.old # Previous version that you want to rollback

To restore the old configuration:

cd /etc/scylla-manager/
mv scylla-manager.yaml scylla-manager.yaml.new
mv scylla-manager.yaml.old scylla-manager.yaml

The procedure is the same for the Scylla Manager Agent (on all nodes):

cd /etc/scylla-manager-agent/
mv scylla-manager-agent.yaml scylla-manager-agent.yaml.new
mv scylla-manager-agent.yaml.old scylla-manager-agent.yaml

Start the ScyllaDB Manager Agent on all nodes¶

On all nodes instruct Systemd to start the agent process:

sudo systemctl start scylla-manager-agent

Ensure that it is running with:

sudo systemctl status scylla-manager-agent

It should have a status of “Active: active (running)”.

Start the ScyllaDB Manager Server¶

On the Manager Server instruct Systemd to start the server process:

sudo systemctl stop scylla-manager

Ensure that it is stopped with:

sudo systemctl status scylla-manager

It should have a status of “Active: active (running)”.

Validate status of the cluster¶

On the Manager Server check the version of the client and the server:

sctool version
Client version: 2.x
Server version: 2.x

Check that cluster is up:

sctool status -c <cluster>

All running nodes should be up.