Scylla Documentation Logo Documentation
  • Server
    • Scylla Open Source
    • Scylla Enterprise
    • Scylla Alternator
  • Cloud
    • Scylla Cloud
    • Scylla Cloud Docs
  • Tools
    • Scylla Manager
    • Scylla Monitoring Stack
    • Scylla Operator
  • Drivers
    • CQL Drivers
    • DynamoDB Drivers
Download
Menu

Caution

You're viewing documentation for a previous version of Scylla Manager. Switch to the latest stable version.

Scylla Manager Health Check

Health Check¶

Scylla Manager automatically adds three health check tasks when the cluster is added to the Scylla Manager and to existing clusters during the upgrade procedure. You can see the tasks created by the healthcheck when you run the sctool task list command.

For example:

sctool task list -c prod-cluster

returns:

sctool task list -c prod-cluster
╭─────────────────────────────────────────────────────────────┬───────────┬────────────────────────────────┬────────╮
│ Task                                                        │ Arguments │ Next run                       │ Status │
├─────────────────────────────────────────────────────────────┼───────────┼────────────────────────────────┼────────┤
│ healthcheck/8988932e-de2f-4c42-a2f8-ae3b97fd7126            │           │ 02 Apr 20 12:28:10 CEST (+15s) │ NEW    │
| healthcheck_alternator/79170f1f-8bda-481e-8538-c3ff9894d235 │           │ 02 Apr 20 12:28:10 CEST (+15s) │ NEW    │
│ healthcheck_rest/9b7e694d-a1e3-42f1-8ca6-d3dfd9f0d94f       │           │ 02 Apr 20 12:28:40 CEST (+1h)  │ NEW    │
│ repair/0fd8a43b-eacf-4df8-9376-2a31b0dee6cc                 │           │ 03 Apr 20 00:00:00 CEST (+7d)  │ NEW    │
╰─────────────────────────────────────────────────────────────┴───────────┴────────────────────────────────┴────────╯

We can see three healthcheck related tasks:

  • Healthcheck - which checks the Scylla CQL, repeating every 15 seconds.

  • Healthcheck Alternator - which checks the Scylla Alternator API, repeating every 15 seconds.

  • Healthcheck REST - which checks the Scylla REST API, repeating every minute.

Scylla Health Check¶

The Scylla health check task ensures that CQL native port is accessible on all the nodes. For each node, in parallel, Scylla Manager opens a connection to a CQL port and asks for server options. If there is no response or the response takes longer than configured value (as recorded in the Scylla Manager Scylla Manager Config the node is considered to be DOWN otherwise the node is considered to be UP. The results are available using the sctool status command.

For example:

sctool status -c prod-cluster

returns:

sctool status -c prod-cluster
Datacenter: eu-west
╭────┬────────────┬───────────┬───────────┬───────────────┬──────────┬──────┬──────────┬────────┬──────────┬──────────────────────────────────────╮
│    │ Alternator │ CQL       │ REST      │ Address       │ Uptime   │ CPUs │ Memory   │ Scylla │ Agent    │ Host ID                              │
├────┼────────────┼───────────┼───────────┼───────────────┼──────────┼──────┼──────────┼────────┼──────────┼──────────────────────────────────────┤
│ UN │ UP (4ms)   │ UP (3ms)  │ UP (2ms)  │ 34.203.122.52 │ 237h2m1s │ 4    │ 15.43GiB │ 4.1.0  │ 2.2.0    │ 8bfd18f1-ac3b-4694-bcba-30bc272554df │
│ UN │ UP (15ms)  │ UP (11ms) │ UP (12ms) │ 10.0.138.46   │ 237h2m1s │ 4    │ 15.43GiB │ 4.1.0  │ 2.2.0    │ 238acd01-813c-4c55-bd65-5219bb19bc20 │
│ UN │ UP (17ms)  │ UP (5ms)  │ UP (7ms)  │ 10.0.196.204  │ 237h2m1s │ 4    │ 15.43GiB │ 4.1.0  │ 2.2.0    │ bde4581a-b25e-49fc-8cd9-1651d7683f80 │
│ UN │ UP (10ms)  │ UP (4ms)  │ UP (5ms)  │ 10.0.66.115   │ 237h2m1s │ 4    │ 15.43GiB │ 4.1.0  │ 2.2.0    │ 918a52aa-cc42-43a4-a499-f7b1ccb53b18 │
╰────┴────────────┴───────────┴───────────┴───────────────┴──────────┴──────┴──────────┴────────┴──────────┴──────────────────────────────────────╯

Scylla Manager reads CQL IP address and port from the node configuration, and can automatically detect TLS/SSL connection. There are two types of CQL health check Credentials agnostic health check and CQL query health check.

Node information¶

New in version 2.2: Scylla Manager

Node status check also provides additional columns that show properties of the available nodes. Those are:

  • CPUs - Total OS CPU count

  • Memory - Total OS memory available

  • Uptime - How long the system has been running without restarts

  • Scylla - Version of Scylla server running on the node

  • Agent - Version Scylla Manger Agent running on the node

  • Host - UUID of the node

  • Address - IP address of the node

Scylla Monitoring¶

If you have enabled the Scylla Monitoring stack, Scylla Manager dashboard includes the same cluster status report. In addition, the Prometheus Alert Manager has an alert to report when a Scylla node health check fails.

Credentials agnostic health check¶

Scylla Manager does not require database credentials to work. CQL health check is based on sending CQL OPTIONS frame and does not start a CQL session. This is simple and effective but does not test CQL all the way down. For that you may consider upgrading to :ref: <cql-query-health-check>`.

CQL query health check¶

New in version 2.2: Scylla Manager

You may specify CQL username and password flags when adding cluster to Scylla Manager using sctool cluster add command. It’s also possible to add or change that using sctool cluster update command. Once Scylla Manager has CQL credential to the cluster, when performing a health check, it would try to connect to each node and execute SELECT now() FROM system.local query.

Scylla Alternator Health Check¶

New in version 2.2: Scylla Manager

If Alternator is enabled it will check the Scylla Alternator API connectivity for all nodes in parallel. In Scylla 4.0 it uses simplified ping checking if the socket is open and if it’s responding. In Scylla 4.1+ it queries the system table.

Please check the Scylla Manager Scylla Manager Config to adjust timeouts for your cluster.

Scylla REST API Health Check¶

Checks Scylla REST API connectivity by performing single HTTP request-response cycle between Scylla Manager Server and all Scylla nodes in parallel.

Please check the Scylla Manager Scylla Manager Config to adjust timeouts for your cluster.

PREVIOUS
Examples
NEXT
CLI sctool
  • 2.4
    • master
    • 3.0
    • 2.6
    • 2.5
    • 2.4
    • 2.3
    • 2.2
  • Install Scylla Manager
  • Install Scylla Manager Agent
  • Add a Cluster
  • Backup
    • Setup AWS S3
    • Setup S3 compatible storage
    • Setup Google Cloud Storage
    • Setup Azure Blob Storage
    • Examples
  • Restore
    • Download files command
  • Repair
    • Repair faster
    • Repair slower
    • Examples
  • Health Check
  • CLI sctool
    • Global flags and variables
    • Cluster
    • Backup
    • Repair
    • Status
    • Suspend & Resume
    • Task
    • Version
  • Configuration Files
    • Scylla Manager Config
    • Scylla Manager Agent Config
  • Swagger File
  • Scylla Monitoring
  • Troubleshooting
  • Older Versions
  • Slack
  • Create an issue
  • Edit this page

On this page

  • Health Check
    • Scylla Health Check
    • Node information
    • Scylla Monitoring
    • Credentials agnostic health check
    • CQL query health check
    • Scylla Alternator Health Check
    • Scylla REST API Health Check
Logo
Docs Contact Us About Us
Mail List Icon Slack Icon
© 2022, ScyllaDB. All rights reserved.
Last updated on 06 December 2022.
Powered by Sphinx 4.3.2 & ScyllaDB Theme 1.3.3