# Upgrade Redis Enterprise with OpenShift CLI ```json metadata { "schema_version": 2, "title": "Upgrade Redis Enterprise with OpenShift CLI", "description": "This task describes how to upgrade a Redis Enterprise cluster via OpenShift CLI.", "categories": ["docs","operate","kubernetes"], "tableOfContents": {"sections":[{"id":"prerequisites","title":"Prerequisites"},{"children":[{"id":"download-the-bundle","title":"Download the bundle"},{"id":"apply-the-bundle","title":"Apply the bundle"},{"id":"reapply-webhook","title":"Reapply the admission controller webhook"},{"id":"verify-the-operator-is-running","title":"Verify the operator is running"},{"id":"reapply-the-scc","title":"Reapply the SCC"}],"id":"upgrade-the-operator","title":"Upgrade the operator"},{"children":[{"id":"edit-redisenterpriseimagespec","title":"Edit redisEnterpriseImageSpec"},{"id":"reapply-roles-and-role-bindings","title":"Reapply roles and role bindings"},{"id":"monitor-the-upgrade","title":"Monitor the upgrade"},{"id":"upgrade-databases","title":"Upgrade databases"}],"id":"upgrade-the-redis-enterprise-cluster","title":"Upgrade the Redis Enterprise Cluster"},{"children":[{"id":"invalid-module-version","title":"Invalid module version"},{"id":"invalid-database-version","title":"Invalid database version"}],"id":"troubleshooting","title":"Troubleshooting"}]} , "codeExamples": [] } ``` Redis implements rolling updates for software upgrades in Kubernetes deployments. The upgrade process includes updating three components: 1. [Upgrade the Redis Enterprise operator](#upgrade-the-operator) 1. [Upgrade the Redis Enterprise cluster (REC)](#upgrade-the-redis-enterprise-cluster) 1. [Upgrade Redis Enterprise databases (REDB)](#upgrade-databases) ## Prerequisites The following steps ensure you have the minimum versions of all components necessary to upgrade to 7.8.2-6 or later. **Without these minimum versions, the upgrade will freeze and require manual recovery.** See the [troubleshooting](#troubleshooting) section for details on recovering a failed upgrade. #### Kubernetes version Check [Supported Kubernetes distributions](https://redis.io/docs/latest/operate/kubernetes/reference) to make sure your Kubernetes distribution is supported. If not, upgrade your Kubernetes distribution before upgrading the Redis operator. #### Redis operator version Your Redis Enterprise clusters must be running version 7.4.2-2 or later before upgrading to 7.8.2-6 or later. See the [7.4 upgrade](https://redis.io/docs/latest/operate/kubernetes/7.4.6/upgrade/openshift-cli/) for detailed steps. #### Redis database version Your Redis databases must be running version 7.2 or later before upgrading your cluster version. See [upgrade databases](#upgrade-databases) for detailed steps. You can find your database version in the [REDB `spec.redisVersion` field](https://redis.io/docs/latest/operate/kubernetes/reference/redis_enterprise_database_api#redisversion). #### RHEL9-compatible modules Upgrading to Redis operator versions 7.8.2-6 and later involves migrating your Redis Enterprise nodes to RHEL9 from either Ubuntu 18 or RHEL8. If your databases use modules, you need to manually install modules compatible with RHEL9. To see which modules you have installed, run: ```sh curl -k -u : -X GET https://localhost:9443/v1/modules | jq -r 'map([.module_name, .semantic_version, (.platforms | keys)]) | .[] | .[0] as $name | .[1] as $version | .[2][] | $name + "-" + $version + "-" + .' | sort ``` To see which modules are currently in use, run: ```sh curl -k -u : -X GET https://localhost:9443/v1/bdbs | jq -r '.[].module_list | map(.module_name + "-" + .semantic_version) | .[]' ``` See [Upgrade modules](https://redis.io/docs/latest/operate/rs/installing-upgrading/modules/upgrade-module) for details on how to upgrade modules with the `rladmin` tool. #### Valid license Use `kubectl get rec` and verify the `LICENSE STATE` is valid on your REC before you start the upgrade process. ## Upgrade the operator ### Download the bundle Make sure you pull the correct version of the bundle. You can find the version tags by checking the [operator releases on GitHub](https://github.com/RedisLabs/redis-enterprise-k8s-docs/releases) or by [using the GitHub API](https://docs.github.com/en/rest/reference/repos#releases). For OpenShift environments, the name of the bundle is `openshift.bundle.yaml`, and so the `curl` command to run is: ```sh curl --silent -O https://raw.githubusercontent.com/RedisLabs/redis-enterprise-k8s-docs/$VERSION/openshift.bundle.yaml ``` If you need a different release, replace `VERSION` in the above with a specific release tag. ### Apply the bundle Apply the bundle to deploy the new operator binary. This will also apply any changes in the new release to custom resource definitions, roles, role binding, or operator service accounts. > [!NOTE] > If you are not pulling images from Docker Hub, update the operator image spec to point to your private repository. > If you have made changes to the role, role binding, RBAC, or custom resource definition (CRD) in the previous version, merge them with the updated declarations in the new version files. If you are using OpenShift, run this instead: ```sh oc apply -f openshift.bundle.yaml ``` After running this command, you should see a result similar to this: ```sh role.rbac.authorization.k8s.io/redis-enterprise-operator configured serviceaccount/redis-enterprise-operator configured rolebinding.rbac.authorization.k8s.io/redis-enterprise-operator configured customresourcedefinition.apiextensions.k8s.io/redisenterpriseclusters.app.redislabs.com configured customresourcedefinition.apiextensions.k8s.io/redisenterprisedatabases.app.redislabs.com configured deployment.apps/redis-enterprise-operator configured ``` ### Reapply the admission controller webhook {#reapply-webhook} If you have the admission controller enabled, you need to manually reapply the `ValidatingWebhookConfiguration`. > [!NOTE] > [Versions 6.4.2 and later](https://redis.io/docs/latest/operate/kubernetes/release-notes/previous-releases/) uses a new `ValidatingWebhookConfiguration` resource to replace `redb-admission`. To use newer releases, delete the old webhook resource and apply the new file. 1. Delete the existing `ValidatingWebhookConfiguration` on the Kubernetes cluster (named `redb-admission`). ```sh kubectl delete ValidatingWebhookConfiguration redb-admission ``` 1. Apply the resource from the new file. ```sh kubectl apply -f deploy/admission/webhook.yaml ``` 1. Verify the `admission-tls` secret exists. ```sh kubectl get secret admission-tls ``` The output should look similar to ```sh NAME TYPE DATA AGE admission-tls Opaque 2 2m43s ``` 1. Save the certificate to a local environment variable. ```sh CERT=`kubectl get secret admission-tls -o jsonpath='{.data.cert}'` ``` 1. Create a Kubernetes validating webhook, replacing `` with the namespace where the REC was installed. The `webhook.yaml` template can be found in [redis-enterprise-k8s-docs/admission](https://github.com/RedisLabs/redis-enterprise-k8s-docs/tree/master/admission) ```sh sed 's/OPERATOR_NAMESPACE//g' webhook.yaml | kubectl create -f - ``` 1. Create a patch file for the Kubernetes validating webhook. ```sh cat > modified-webhook.yaml < [!WARNING] > We recommend upgrading the REC as soon as possible after updating the operator. After the operator upgrade completes, the operator suspends the management of the REC and its associated REDBs, until the REC upgrade completes. > ### Reapply the SCC If you are using OpenShift, you will also need to manually reapply the [security context constraints](https://docs.openshift.com/container-platform/4.8/authentication/managing-security-context-constraints.html) file ([`scc.yaml`](https://redis.io/docs/latest/operate/kubernetes/deployment/openshift/openshift-cli#deploy-the-operator)) and bind it to your service account. ```sh oc apply -f openshift/scc.yaml ``` ```sh oc adm policy add-scc-to-user redis-enterprise-scc-v2 \ system:serviceaccount:: ``` If you are upgrading from operator version 6.4.2-6 or before, see the ["after upgrading"](#after-upgrading) section to delete the old SCC and role binding after all clusters are running 6.4.2-6 or later. ## Upgrade the Redis Enterprise Cluster > [!WARNING] > Verify your license is valid before upgrading. Invalid licenses will cause the upgrade to fail. > > Use `oc get rec` and verify the `LICENSE STATE` is valid on your REC before you start the upgrade process. The Redis Enterprise cluster (REC) can be updated automatically or manually. To trigger automatic upgrade of the REC after the operator upgrade completes, specify `autoUpgradeRedisEnterprise: true` in your REC spec. If you don't have automatic upgrade enabled, follow the below steps for the manual upgrade. Before beginning the upgrade of the Redis Enterprise cluster, check the K8s operator [release notes](https://redis.io/docs/latest/operate/kubernetes/release-notes) to find the Redis Enterprise image tag. After the operator upgrade is complete, you can upgrade Redis Enterprise cluster (REC). ### Edit `redisEnterpriseImageSpec` 1. Edit the REC custom resource YAML file. ```sh oc edit rec ``` 1. Replace the `versionTag:` declaration under `redisEnterpriseImageSpec` with the new version tag. ```YAML spec: redisEnterpriseImageSpec: imagePullPolicy: IfNotPresent repository: redislabs/redis versionTag: ``` 1. Save the changes to apply. ### Reapply roles and role bindings If your operator is monitoring multiple namespaces, you'll need to [reapply your role and role bindings](https://redis.io/docs/latest/operate/kubernetes/re-clusters/multi-namespace#create-role-and-role-binding-for-managed-namespaces) for each managed namespace. See [Manage databases in multiple namespaces](https://redis.io/docs/latest/operate/kubernetes/re-clusters/multi-namespace) for more details. ### Monitor the upgrade You can view the state of the REC with `oc get rec`. During the upgrade, the state should be `Upgrade`. When the upgrade is complete and the cluster is ready to use, the state will change to `Running`. If the state is `InvalidUpgrade`, there is an error (usually relating to configuration) in the upgrade. ```sh $ oc get rec NAME NODES VERSION STATE SPEC STATUS LICENSE STATE SHARDS LIMIT LICENSE EXPIRATION DATE AGE rec 3 6.2.10-107 Upgrade Valid Valid 4 2022-07-16T13:59:00Z 92m ``` To see the status of the current rolling upgrade, run: ```sh oc rollout status sts ``` ### Upgrade databases After the cluster is upgraded, you can upgrade your databases. Specify your new database version in the `spec.redisVersion` field for your REDB and REAADB custom resources. Supported database versions include `"7.2"` and `"7.4"` (note this value is a string). Note that if your cluster [`redisUpgradePolicy`](https://redis.io/docs/latest/operate/kubernetes/reference/redis_enterprise_cluster_api#redisupgradepolicy) or your database [`redisVersion`](https://redis.io/docs/latest/operate/kubernetes/reference/redis_enterprise_database_api#redisversion) are set to `major`, you won't be able to upgrade those databases to minor versions. See [Redis upgrade policy](https://redis.io/docs/latest/operate/rs/installing-upgrading/upgrading#redis-upgrade-policy) for more details. ## Troubleshooting If you start an upgrade without meeting the [prerequisites](#prerequisites), the operator will freeze the upgrade. Check the operator logs for the source of the error. The REDB reconsilliation doesn't work during an upgrade, so you need to apply a manual fix with the Redis Software API (examples below). The updates will also need to be added to the REDB custom resource. ### Invalid module version If the operator logs show an event related to an unsupported module, download the updated module locally, and install it using the `v2/modules` API endpoint. ```sh curl -sfk -u : -X POST -F 'module=@' https://localhost:9443/v2/modules ``` After updating the modules with the Redis Software API, update the REDB custom resource to reflect the change. ### Invalid database version If the operator logs show an event related to an incompatible database version, upgrade the database using the Redis Software API. ```sh curl -sfk -u : -X POST -H "Content-Type: application/json" -d '{"redis_version": }' https://localhost:9443/v1/bdbs//upgrade ``` After updating the database with the Redis Software API, update the REDB custom resource to reflect the change.