# Client-side geographic failover ```json metadata { "schema_version": 2, "title": "Client-side geographic failover", "description": "Improve reliability using the failover features of redis-py.", "categories": ["docs","develop","stack","oss","rs","rc","oss","kubernetes","clients"], "topics": ["failover","failback","resilience","health checks","retries"], "relatedPages": ["/develop/clients/failover"], "scope": ["client-specific","implementation"], "tableOfContents": {"sections":[{"children":[{"id":"endpoint-configuration","title":"Endpoint configuration"},{"id":"client-configuration","title":"Client configuration"},{"id":"circuit-breaker-configuration","title":"Circuit breaker configuration"},{"id":"retry-configuration","title":"Retry configuration"},{"id":"failback-configuration","title":"Failback configuration"},{"id":"general-failover-configuration","title":"General failover configuration"},{"id":"failover-callbacks","title":"Failover callbacks"}],"id":"failover-configuration","title":"Failover configuration"},{"children":[{"id":"pinghealthcheck-default","title":"PingHealthCheck (default)"},{"id":"lag-aware-health-check","title":"LagAwareHealthCheck (Redis Software only)"},{"id":"custom-health-check-strategy","title":"Custom health check strategy"}],"id":"health-check-configuration","title":"Health check configuration"},{"children":[{"id":"manual-failback","title":"Manual failback"}],"id":"managing-databases-at-runtime","title":"Managing databases at runtime"},{"id":"pubsub-and-re-subscription","title":"Pub/Sub and re-subscription"},{"id":"behavior-when-all-endpoints-are-unhealthy","title":"Behavior when all endpoints are unhealthy"},{"children":[{"id":"excessive-or-constant-health-check-failures","title":"Excessive or constant health check failures"},{"id":"slow-failback-after-recovery","title":"Slow failback after recovery"}],"id":"troubleshooting","title":"Troubleshooting"}]} , "codeExamples": [] } ``` redis-py supports [Client-side geographic failover](https://en.wikipedia.org/wiki/Failover) to improve the availability of connections to Redis databases. This page explains how to configure redis-py for failover. For an overview of the concepts, see the main [Client-side geographic failover](https://redis.io/docs/latest/develop/clients/failover) page. ## Failover configuration The example below shows a simple case with a list of two servers, `redis-east` and `redis-west`, where `redis-east` is the preferred target. If `redis-east` fails, redis-py should fail over to `redis-west`. Supply the weighted endpoints using a list of `DatabaseConfig` objects (see [Selecting a failover target](https://redis.io/docs/latest/develop/clients/failover#selecting-a-failover-target) for a full description of how the weighted list is used). Use the `weight` option to order the endpoints, with the highest weight being tried first. Then, use the list to create a `MultiDbConfig` object, which you can pass to the `MultiDBClient` constructor to create the client. `MultiDBClient` implements the usual Redis commands using an internal `RedisClient` instance, but will also handle the connection management and failover transparently. ```py from redis.multidb.client import MultiDBClient from redis.multidb.config import MultiDbConfig, DatabaseConfig db_configs = [ DatabaseConfig( client_kwargs={"host": "redis-east.example.com", "port": "14000"}, weight=1.0 ), DatabaseConfig( client_kwargs={"host": "redis-west.example.com", "port": "14000"}, weight=0.5 ), ] cfg = MultiDbConfig(databases_config=db_configs) client = MultiDBClient(cfg) ``` ### Endpoint configuration The `DatabaseConfig` class provides several options to configure each endpoint, as described in the table below. Supply the configurations for the whole set of endpoints by passing a list of `DatabaseConfig` objects to the `MultiDbConfig` constructor in the `databases_config` parameter. | Option | Description | | --- | --- | | `client_kwargs` | Keyword parameters to pass to the internal client constructor for this endpoint. Use it to specify the host, port, username, password, and other connection parameters (see [Connect to the server](https://redis.io/docs/latest/develop/clients/redis-py/connect) for more information). This is especially useful if you are using a custom client class (see [Client configuration](#client-configuration) below for more information). | | `from_url` | Redis URL to connect to this endpoint, as an alternative to passing the host and port in `client_kwargs`. | | `from_pool` | A `ConnectionPool` to supply the endpoint connection (see [Connect with a connection pool](https://redis.io/docs/latest/develop/clients/redis-py/connect#connect-with-a-connection-pool) for more information) | | `weight` | Priority of the endpoint, with higher values being tried first. Default is `1.0`. | | `grace_period` | Duration in seconds to keep an unhealthy endpoint disabled before attempting a failback. Default is `60` seconds. | | `health_check_url` | URL for health checks that use the database's REST API (see [`LagAwareHealthCheck`](#lag-aware-health-check) for more information). | ### Client configuration `MultiDbConfig` provides the `client_class` option to specify the class of the internal client to use for each endpoint. The default is the basic `redis.Redis` client, but you could, for example, replace this with `redis.asyncio.client.Redis` for an asynchronous basic client, or with `redis.cluster.RedisCluster`/`redis.asyncio.cluster.RedisCluster` for a cluster client. Use the `client_kwargs` option of `DatabaseConfig` to supply any extra parameters required by the client class (see [Endpoint configuration](#endpoint-configuration) above for more information). ```py cfg = MultiDbConfig( ... client_class=redis.asyncio.client.Redis, ... ) ``` ### Circuit breaker configuration `MultiDbConfig` gives you several options to configure the circuit breaker (see [Detecting connection problems](https://redis.io/docs/latest/develop/clients/failover#detecting-connection-problems) for more information on how the circuit breaker works): | Option | Description | | --- | --- | | `failures_detection_window` | Duration in seconds to keep failures and successes in the sliding window. Default is `2` seconds. | | `min_num_failures` | Minimum number of failures that must occur to trigger a failover. Default is `1000`. | | `failure_rate_threshold` | Fraction of failed commands required to trigger a failover. Default is `0.1` (10%). | ### Retry configuration `MultiDbConfig` provides the `command_retry` option to configure retries for failed commands. This follows the usual approach to configuring retries used with a standard `RedisClient` connection (see [Retries](https://redis.io/docs/latest/develop/clients/redis-py/produsage#retries) for more information). ```py cfg = MultiDbConfig( ... # Retry failed commands up to three times using exponential backoff # with jitter between attempts. command_retry=Retry( retries=3, backoff=ExponentialWithJitterBackoff(base=1, cap=10), ), ... ) ``` ### Failback configuration The `MultiDbConfig` constructor provides the following option to configure the failback behavior: | Option | Description | | --- | --- | | `auto_fallback_interval` | Time interval between automatic failback attempts. Set this to `-1` to disable automatic failback. Default value is `30` seconds (which means failback is enabled by default). | ### General failover configuration There are also a few other options you can pass to the `MultiDbConfig` constructor to control the failover behavior: | Option | Description | | --- | --- | | `failover_attempts` | Number of attempts to fail over to a new endpoint before giving up. Default is `10`. | | `failover_delay` | Time interval between successive failover attempts. Default is `12` seconds. | | `event_dispatcher` | `EventDispatcher` object to use for emitting events. Supply this to register custom event listeners (see [Failover callbacks](#failover-callbacks) below for more information). | ### Failover callbacks You may want to take some custom action when a failover occurs. For example, you could log a warning, increment a metric, or externally persist the cluster connection state. You can implement a custom event listener using a class that implements `EventListenerInterface`. This includes a `listen()` method that is called when the event occurs. You should specifically listen for the `ActiveDatabaseChanged` event, which is emitted when a failover happens. The example below shows how to implement a simple listener class and register an instance of it using an `EventDispatcher` object. You can then pass the `EventDispatcher` to the `MultiDbConfig` constructor to enable the custom behavior. ```py import logging from redis.multidb.client import MultiDBClient from redis.multidb.config import MultiDbConfig from redis.event import EventListenerInterface, EventDispatcher from redis.multidb.event import ActiveDatabaseChanged ... class LogFailoverEventListener(EventListenerInterface): def __init__(self, logger: logging.Logger): self.logger = logger def listen(self, event: ActiveDatabaseChanged): self.logger.warning( f"Failover happened. Active database switched from {event.old_database} to {event.new_database}" ) event_dispatcher = EventDispatcher() listener = LogFailoverEventListener(logging.getLogger(__name__)) # Register custom listener event_dispatcher.register_listeners( { ActiveDatabaseChanged: [listener], } ) config = MultiDbConfig( event_dispatcher=event_dispatcher, ... ) client = MultiDBClient(config) ``` ## Health check configuration Each health check consists of one or more separate "probes", each of which is a simple test (such as a [`PING`](https://redis.io/docs/latest/commands/ping) command) to determine if the database is available. The results of the separate probes are combined using a configurable policy to determine if the database is healthy. Note that an initial health check is performed on each database when you create the client or when you add a new database. This initial health check has a separate configuration from the periodic health checks, so you can use a stricter policy here to ensure you don't add a database that is already unhealthy. `MultiDbConfig` provides the following options to configure the health check behavior: | Option | Description | | --- | --- | | `health_check_interval` | Time interval between successive health checks (each of which may consist of multiple probes). Default is `5` seconds. | | `health_check_probes` | Number of separate probes performed during each health check. Default is `3`. | | `health_check_probes_delay` | Delay between probes during a health check. Default is `0.5` seconds. | | `health_check_timeout` | Timeout for the full health check operation (including all probes). | | `health_check_policy` | `HealthCheckPolicies` enum value to specify the policy for determining database health from the separate probes of a health check. The options are `HealthCheckPolicies.HEALTHY_ALL` (all probes must succeed), `HealthCheckPolicies.HEALTHY_ANY` (at least one probe must succeed), and `HealthCheckPolicies.HEALTHY_MAJORITY` (more than half the probes must succeed). The default policy is `HealthCheckPolicies.HEALTHY_MAJORITY`. | | `health_check` | Custom list of `HealthCheck` objects to specify how to perform each probe during a health check. This defaults to just the simple [`PingHealthCheck`](#pinghealthcheck-default). | | `initial_health_check_policy` | `InitialHealthCheck` enum value to specify the policy to use during the initial health check. The options are `InitialHealthCheck.ALL_AVAILABLE` (all probes must succeed), `InitialHealthCheck.ANY_AVAILABLE` (at least one probe must succeed), and `InitialHealthCheck.MAJORITY_AVAILABLE` (more than half the probes must succeed). The default policy is `InitialHealthCheck.ALL_AVAILABLE`. | There are several strategies available for health checks that you can configure using the `MultiClusterClientConfig` builder. The sections below explain these strategies in more detail. ### `PingHealthCheck` (default) The default strategy, `PingHealthCheck`, periodically sends a Redis [`PING`](https://redis.io/docs/latest/commands/ping) command and checks that it gives the expected response. Any unexpected response or exception indicates an unhealthy server. Although `PingHealthCheck` is very simple, it is a good basic approach for most Redis deployments. ### `LagAwareHealthCheck` (Redis Software only) {#lag-aware-health-check} `LagAwareHealthCheck` is designed specifically for Redis Software [Active-Active](https://redis.io/docs/latest/operate/rs/databases/active-active) deployments. It determines the health of the server by using the [REST API](https://redis.io/docs/latest/operate/rs/references/rest-api) to check the synchronization lag between a specific database and the others in the Active-Active setup. If the lag is within a specified tolerance, the server is considered healthy. `LagAwareHealthCheck` uses the `health_check_url` value for the endpoint to connect to the database's REST API, so you must specify this in the `DatabaseConfig` for each endpoint: ```py db_configs = [ DatabaseConfig( client_kwargs={"host": "redis-east.example.com", "port": "14000"}, weight=1.0, health_check_url="https://health.redis-east.example.com" ), DatabaseConfig( client_kwargs={"host": "redis-west.example.com", "port": "14000"}, weight=0.5, health_check_url="https://health.redis-west.example.com" ), ] ``` You must also add a `LagAwareHealthCheck` instance to the `health_check` list in the `MultiDbConfig` constructor: ```py cfg = MultiDbConfig( databases_config=db_configs, health_check=[LagAwareHealthCheck( rest_api_port=9443, lag_aware_tolerance=100, # ms verify_tls=True, # auth_basic=("user", "pass"), # ca_file="/path/ca.pem", # client_cert_file="/path/cert.pem", # client_key_file="/path/key.pem", )], ... ) client = MultiDBClient(cfg) ``` The `LagAwareHealthCheck` constructor accepts the following options: | Option | Description | | --- | --- | | `rest_api_port` | Port number for Redis Software REST API (default is 9443). | | `lag_aware_tolerance` | Tolerable synchronization lag between databases in milliseconds (default is 100ms). | | `timeout` | REST API request timeout in seconds (default is 30 seconds). | | `auth_basic` | Tuple of (username, password) for basic authentication. | | `verify_tls` | Whether to verify TLS certificates (defaults to `True`). | | `ca_file` | Path to CA certificate file for TLS verification. | | `ca_path` | Path to CA certificates directory for TLS verification. | | `ca_data` | CA certificate data as string or bytes. | | `client_cert_file` | Path to client certificate file for mutual TLS. | | `client_key_file` | Path to client private key file for mutual TLS. | | `client_key_password` | Password for encrypted client private key | ### Custom health check strategy You can supply your own custom health check strategy by deriving a new class from the `AbstractHealthCheck` class. For example, you might use this to integrate with external monitoring tools or to implement checks that are specific to your application. Add an instance of your custom class to the `health_check` list in the `MultiDbConfig` constructor, as with [`LagAwareHealthCheck`](#lag-aware-health-check). Note that health checks are executed in an asyncio event loop, so you must implement the `check_health()` method as an async method. The example below shows a simple custom strategy that sends a Redis [`ECHO`](https://redis.io/docs/latest/commands/echo) command and checks for the expected response. ```py from redis.asyncio.multidb.healthcheck import AbstractHealthCheck, AsyncRedisClientT class EchoHealthCheck(AbstractHealthCheck): """ Health check based on ECHO command. """ async def check_health(self, database, hc_client: AsyncRedisClientT) -> bool: await connection.send_command("ECHO", "healthcheck") response = await connection.read_response() return response in (b"healthcheck", "healthcheck") cfg = MultiDbConfig( ... health_check=[EchoHealthCheck(retry=Retry(retries=3))], ... ) client = MultiDBClient(cfg) ``` ## Managing databases at runtime Although you will typically configure all databases during the initial connection, you can also modify the configuration at runtime. You can add and remove database endpoints, and update their weights: ```py from redis.multidb.client import MultiDBClient from redis.multidb.config import MultiDbConfig, DatabaseConfig cfg = MultiDbConfig( databases_config = [ DatabaseConfig( client_kwargs={"host": "redis-east.example.com", "port": "14000"}, weight=1.0 ), DatabaseConfig( client_kwargs={"host": "redis-west.example.com", "port": "14000"}, weight=0.5 ), ] ) client = MultiDBClient(cfg) # Add a database programmatically. The `skip_initial_health_check` # parameter defaults to `True`, which avoids adding the database if it # is unhealthy, but you can override this by setting it to `False` # explicitly. other = DatabaseConfig( client_kwargs={"host": "redis-south.example.com", "port": "14000"}, weight=0.5 ) client.add_database(other, skip_initial_health_check=True) # Get a list of databases, sorted by weight. databases = client.get_databases() # Update the weights of each database whose weight is less than 1. for db in databases: if db.weight < 1.0: client.update_database_weight(db, 0.9) # Remove a database from the failover set. client.remove_database(databases[0]) ``` ### Manual failback By default, the failback mechanism runs health checks on all servers in the weighted list and selects the highest-weighted server that is healthy. However, you can also use the `set_active_database()` method of `MultiDBClient` to select which database to use manually: ```py # Manually set `other` as the active database. client.set_active_database(other) ``` Note that `set_active_database()` is thread-safe. If you decide to implement manual failback, you will need a way for external systems to trigger this method in your application. For example, if your application exposes a REST API, you might consider creating a REST endpoint to call `set_active_database()`. ## Pub/Sub and re-subscription `MultiDBClient` supports [Pub/Sub](https://redis.io/docs/latest/develop/pubsub) messaging with automatic re-subscription to channels during failover. This means you don't have to detect failovers and re-subscribe manually: - **Subscriber failover**: When the active database becomes unhealthy, the subscriber automatically reconnects to the next available database and re-subscribes to the same channels. - **Publisher failover**: The publisher switches to the next available database and continues publishing to the same channels. For reliable Pub/Sub delivery during failover, use `MultiDBClient` instances for *both* publishers and subscribers. Create a Pub/Sub interface from the client in the usual way using the `pubsub()` method, then subscribe to one or more channels: ```py pubsub = client.pubsub() pubsub.subscribe("news", "alerts") # If a failover happens here, the subscriptions are automatically # re-established on the new active database. msg = pubsub.get_message(timeout=1.0) if msg: print(msg) ``` Re-subscription happens transparently and is independent of any custom event listeners you register (see [Failover callbacks](#failover-callbacks)). > [!NOTE] > Message loss can still occur if the failover events happen in > the reverse order, with the publisher failing over to the new database > before the subscriber. Messages published during this window may not reach > a subscriber that is still connected to the previous database. ## Behavior when all endpoints are unhealthy In the extreme case where no endpoint is healthy, a command will throw a `TemporaryUnavailableException`. This indicates that the client is periodically checking to see if any endpoint becomes healthy again. The number of times it will keep checking is configured by the `failover_attempts` option in `MultiDbConfig` and the delay between attempts is configured by the `failover_delay` option (see [General failover configuration](#general-failover-configuration)). With the default settings, `failover_attempts` * `failover_delay` gives a period of 120 seconds to find a healthy endpoint. You can still keep retrying commands after a `TemporaryUnavailableException` is thrown (for example, you could add this exception to the `supported_errors` list in your `Retry` configuration, as described in [Retries](https://redis.io/docs/latest/develop/clients/redis-py/produsage#retries)). However, if the client exhausts all the available failover attempts before any endpoint becomes healthy again, commands will throw a `NoValidDatabaseException`. The client won't recover automatically from this situation, so you should handle it by reconnecting with the `MultiDBClient` constructor after a suitable delay (see [Failover configuration](#failover-configuration) for a connection example). ## Troubleshooting This section lists some common problems and their solutions. ### Excessive or constant health check failures If all health checks fail, you should first rule out authentication problems with the Redis server and also make sure there are no persistent network connectivity problems. If you are using [`PingHealthCheck`](#pinghealthcheck-default) or a [custom health check strategy](#custom-health-check-strategy), check that the `socket_timeout` is not too low for your network conditions (see [Timeouts](https://redis.io/docs/latest/develop/clients/redis-py/produsage#timeouts) for more information). For [`LagAwareHealthCheck`](#lag-aware-health-check), check that the `health_check_url` is set correctly for each endpoint. Note that health checks might be taking longer to execute than you anticipated, so make sure your `timeout` setting is not too low. ### Slow failback after recovery If failback is too slow after a server recovers, you can try reducing the `health_check_interval` period and also reducing the `grace_period` before failback is attempted (see [Health check configuration](#health-check-configuration) for more information about these options).