# Deploy a pipeline ```json metadata { "schema_version": 2, "title": "Deploy a pipeline", "description": "Learn how to deploy an RDI pipeline", "categories": ["docs","integrate","rs","rdi"], "group": "di", "tableOfContents": {"sections":[{"children":[{"id":"set-secrets-with-the-cli","title":"Set secrets with the CLI"},{"id":"manage-secrets-with-the-cli","title":"Manage secrets with the CLI"},{"id":"set-secrets-for-k8shelm-deployment-using-kubectl-command","title":"Set secrets for K8s/Helm deployment using Kubectl command"}],"id":"set-secrets","title":"Set secrets"},{"id":"deploy-a-pipeline","title":"Deploy a pipeline"},{"id":"display-the-pipeline-status","title":"Display the pipeline status"},{"id":"start-and-stop-a-pipeline","title":"Start and stop a pipeline"},{"id":"reset-a-pipeline","title":"Reset a pipeline"},{"id":"clear-a-pipeline","title":"Clear a pipeline"},{"id":"wait","title":"Wait for changes to complete"}]} , "codeExamples": [] } ``` The sections below explain how to deploy a pipeline after you have created the required [configuration](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines). ## Set secrets Before you deploy your pipeline, you must set the authentication secrets for the source and target databases. Every secret belongs to one database: a source, identified by its name in `config.yaml`, or the target. You name that database with the `--db` option of the [`redis-di set-secret`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-set-secret) command. The table below lists the available secret keys. The username and password are required, while the other keys are only relevant for TLS/mTLS connections. | Secret key | Description | | :-- | :-- | | `USERNAME` | Username for the database | | `PASSWORD` | Password for the database | | `CACERT` | (For TLS only) CA certificate | | `CERT` | (For mTLS only) Client certificate | | `KEY` | (For mTLS only) Private key | | `KEY_PASSWORD` | (For mTLS only) Private key password | You can reference a secret in `config.yaml` using an environment variable that is derived from the secret key and the database name. The variable name consists of the database name in uppercase (with each dash replaced by an underscore), followed by `_DB_`, followed by the key. For example, if you set `PASSWORD` with `--db mysql` the corresponding environment variable is `MYSQL_DB_PASSWORD`, which the source references as `${MYSQL_DB_PASSWORD}`. If you set `PASSWORD` with `--db target`, the environment variable is `TARGET_DB_PASSWORD`. The sample [config.yaml file](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config#example) shows these references in use, and [Multiple sources in one pipeline](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/multiple-sources) covers the source naming rules. > [!NOTE] > The scope-prefixed keys `SOURCE_DB_*` and `TARGET_DB_*` are also accepted, and can be > used without specifying `--db`. RDI accepts a `SOURCE_DB_*` key only for a pipeline with exactly > one source. Prefer the per-database keys with `--db`, which work for any pipeline. See > [Existing names are kept after an upgrade](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade) for more information. > [!NOTE] > When creating secrets for TLS or mTLS, ensure that all certificates and keys are in `PEM` format. The only exception to this is that for PostgreSQL, the source's private key secret must be in `DER` format. If you have a key in `PEM` format, you must convert it to `DER` before creating the secret using the command: ```bash openssl pkcs8 -topk8 -inform PEM -outform DER \ -in /path/to/myclient.pem \ -out /path/to/myclient.pk8 -nocrypt ``` This command assumes that the private key is not encrypted. See the [`openssl` documentation](https://docs.openssl.org/master/) to learn how to convert an encrypted private key. ### Set secrets with the CLI Use [`redis-di set-secret`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-set-secret) to set secrets for any installation type (VM, Kubernetes, or Redis Cloud). The command lines for a source named `mysql` are shown below. If your pipeline has multiple sources, you should run the command once for each source, using the appropriate database name. ```bash # For username and password redis-di set-secret USERNAME --db mysql yourUsername redis-di set-secret PASSWORD --db mysql yourPassword # With source TLS, in addition to the above redis-di set-secret CACERT --db mysql /path/to/myca.crt # With source mTLS, in addition to the above redis-di set-secret CERT --db mysql /path/to/myclient.crt redis-di set-secret KEY --db mysql /path/to/myclient.key # Use this only if the private key is password-protected redis-di set-secret KEY_PASSWORD --db mysql yourKeyPassword ``` The corresponding command lines for target secrets are: ```bash # For username and password redis-di set-secret USERNAME --db target yourUsername redis-di set-secret PASSWORD --db target yourPassword # With target TLS, in addition to the above redis-di set-secret CACERT --db target /path/to/myca.crt # With target mTLS, in addition to the above redis-di set-secret CERT --db target /path/to/myclient.crt redis-di set-secret KEY --db target /path/to/myclient.key # Use this only if the private key is password-protected redis-di set-secret KEY_PASSWORD --db target yourKeyPassword ``` By default, `set-secret` waits for the pipeline to apply the change before returning. When you set several secrets at once, set all but the last one with `--wait=false` to avoid a timeout while the pipeline is only partially updated. See [Wait for changes to complete](#wait) below for details. ### Manage secrets with the CLI Along with `set-secret`, the CLI has commands to list, inspect, and delete secrets. Because the API never returns secret values, these commands show only the secret keys and whether they are set, not the stored values. ```bash # List all the secrets of a pipeline, with the database each one belongs to redis-di list-secrets # List only the secrets of one database redis-di list-secrets --db mysql # Show a single secret and whether it is set redis-di describe-secret PASSWORD --db mysql # Delete a secret (prompts for confirmation unless you add --force) redis-di delete-secret CACERT --db mysql ``` See the reference pages for [`list-secrets`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-list-secrets), [`get-secret`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-get-secret), [`describe-secret`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-describe-secret), and [`delete-secret`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-delete-secret) for the full list of options. ### Set secrets for K8s/Helm deployment using Kubectl command > [!NOTE] > It is strongly recommended to manage secrets with the `redis-di` CLI rather than with > `kubectl` directly. The CLI applies the correct labels automatically, validates the secret keys, and > works the same way across all installation types. For a Kubernetes/Helm deployment, you can also use [`kubectl create secret generic`](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_create/kubectl_create_secret_generic/) to set secrets instead of the CLI. The general pattern of the commands is: ```bash kubectl create secret generic \ --namespace=rdi \ --from-literal== ``` Where `` is `-db` for the secrets of a source, or `target-db` for target secrets. The examples below use a source named `mysql`, so its secret is `mysql-db`. If you use TLS or mTLS for either the source or target databases, you also need to create the `-db-ssl` and/or `target-db-ssl` K8s secrets that contain the certificates used to establish secure connections. The general pattern of the commands is: ```bash kubectl create secret generic -ssl \ --namespace=rdi \ --from-file== ``` The specific command lines for source secrets are as follows: ```bash # Without source TLS # Create or update mysql-db secret kubectl create secret generic mysql-db --namespace=rdi \ --from-literal=MYSQL_DB_USERNAME=yourUsername \ --from-literal=MYSQL_DB_PASSWORD=yourPassword \ --save-config --dry-run=client -o yaml | kubectl apply -f - # With source TLS # Create or update mysql-db secret kubectl create secret generic mysql-db --namespace=rdi \ --from-literal=MYSQL_DB_USERNAME=yourUsername \ --from-literal=MYSQL_DB_PASSWORD=yourPassword \ --from-literal=MYSQL_DB_CACERT=/etc/certificates/mysql_db/ca.crt \ --save-config --dry-run=client -o yaml | kubectl apply -f - # Create or update mysql-db-ssl secret kubectl create secret generic mysql-db-ssl --namespace=rdi \ --from-file=ca.crt=/path/to/myca.crt \ --save-config --dry-run=client -o yaml | kubectl apply -f - # With source mTLS # Create or update mysql-db secret. Include the MYSQL_DB_KEY_PASSWORD line # only if the private key is password-protected. kubectl create secret generic mysql-db --namespace=rdi \ --from-literal=MYSQL_DB_USERNAME=yourUsername \ --from-literal=MYSQL_DB_PASSWORD=yourPassword \ --from-literal=MYSQL_DB_CACERT=/etc/certificates/mysql_db/ca.crt \ --from-literal=MYSQL_DB_CERT=/etc/certificates/mysql_db/client.crt \ --from-literal=MYSQL_DB_KEY=/etc/certificates/mysql_db/client.key \ --from-literal=MYSQL_DB_KEY_PASSWORD=yourKeyPassword \ --save-config --dry-run=client -o yaml | kubectl apply -f - # Create or update mysql-db-ssl secret kubectl create secret generic mysql-db-ssl --namespace=rdi \ --from-file=ca.crt=/path/to/myca.crt \ --from-file=client.crt=/path/to/myclient.crt \ --from-file=client.key=/path/to/myclient.key \ --save-config --dry-run=client -o yaml | kubectl apply -f - ``` The corresponding command lines for target secrets are: ```bash # Without target TLS # Create or update target-db secret kubectl create secret generic target-db --namespace=rdi \ --from-literal=TARGET_DB_USERNAME=yourUsername \ --from-literal=TARGET_DB_PASSWORD=yourPassword \ --save-config --dry-run=client -o yaml | kubectl apply -f - # With target TLS # Create or update target-db secret kubectl create secret generic target-db --namespace=rdi \ --from-literal=TARGET_DB_USERNAME=yourUsername \ --from-literal=TARGET_DB_PASSWORD=yourPassword \ --from-literal=TARGET_DB_CACERT=/etc/certificates/target_db/ca.crt \ --save-config --dry-run=client -o yaml | kubectl apply -f - # Create or update target-db-ssl secret kubectl create secret generic target-db-ssl --namespace=rdi \ --from-file=ca.crt=/path/to/myca.crt \ --save-config --dry-run=client -o yaml | kubectl apply -f - # With target mTLS # Create or update target-db secret. Include the TARGET_DB_KEY_PASSWORD line # only if the private key is password-protected. kubectl create secret generic target-db --namespace=rdi \ --from-literal=TARGET_DB_USERNAME=yourUsername \ --from-literal=TARGET_DB_PASSWORD=yourPassword \ --from-literal=TARGET_DB_CACERT=/etc/certificates/target_db/ca.crt \ --from-literal=TARGET_DB_CERT=/etc/certificates/target_db/client.crt \ --from-literal=TARGET_DB_KEY=/etc/certificates/target_db/client.key \ --from-literal=TARGET_DB_KEY_PASSWORD=yourKeyPassword \ --save-config --dry-run=client -o yaml | kubectl apply -f - # Create or update target-db-ssl secret kubectl create secret generic target-db-ssl --namespace=rdi \ --from-file=ca.crt=/path/to/myca.crt \ --from-file=client.crt=/path/to/myclient.crt \ --from-file=client.key=/path/to/myclient.key \ --save-config --dry-run=client -o yaml | kubectl apply -f - ``` Note that the certificate paths contained in the `CACERT`, `CERT`, and `KEY` secrets are internal to RDI, so you *must* use the values shown in the example above. Each source has its own certificate directory, named after the source (for example, `/etc/certificates/mysql_db/` for a source named `mysql`). You should only change the certificate paths when you create the `-db-ssl` and `target-db-ssl` secrets. You must also label any secrets that you create directly with `kubectl` so that the RDI operator discovers them as pipeline secrets. Give each secret the following labels, where the `app.kubernetes.io/instance` label corresponds to the pipeline name (the name is just `default` for the default pipeline): | Label | Value | | :-- | :-- | | `app.kubernetes.io/name` | `pipeline` | | `app.kubernetes.io/instance` | `default` | | `product` | `rdi` | Apply the labels to each secret with [`kubectl label`](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_label/): ```bash kubectl label secret mysql-db --namespace=rdi --overwrite \ app.kubernetes.io/name=pipeline \ app.kubernetes.io/instance=default \ product=rdi kubectl label secret target-db --namespace=rdi --overwrite \ app.kubernetes.io/name=pipeline \ app.kubernetes.io/instance=default \ product=rdi # With source TLS or mTLS kubectl label secret mysql-db-ssl --namespace=rdi --overwrite \ app.kubernetes.io/name=pipeline \ app.kubernetes.io/instance=default \ product=rdi # With target TLS or mTLS kubectl label secret target-db-ssl --namespace=rdi --overwrite \ app.kubernetes.io/name=pipeline \ app.kubernetes.io/instance=default \ product=rdi ``` ## Deploy a pipeline When you have created your configuration, including the [jobs](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples), you are ready to deploy. Use the [`redis-di deploy`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-deploy) command to deploy a pipeline: ```bash redis-di deploy --dir ``` RDI first validates the configuration and then deploys it if it is correct. You can control the validation and what happens after deployment with the following options: - `--dry-run`: Validate the configuration without deploying it. Off by default. - `--validate-tables`: Validate the configuration against the source and target databases, for example that the tables it references exist. On by default; pass `--validate-tables=false` to skip this check, which is useful when the databases are not reachable at deploy time. - `--validate-cdc`: Additionally validate that the source database is correctly configured for [change data capture (CDC)](https://redis.io/docs/latest/integrate/redis-data-integration/architecture#overview). Off by default; enable it with `--validate-cdc`. - `--start`: Start the pipeline as soon as it is deployed. On by default; pass `--start=false` to deploy the pipeline without starting it, then start it later with [`redis-di start`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-start). See the [`redis-di deploy`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-deploy) reference page for the full list of options. You can also use [Redis Insight](https://redis.io/docs/latest/develop/tools/insight/rdi-connector) to configure and deploy pipelines for both VM and K8s installations. ## Display the pipeline status Once a pipeline is deployed, use the [`redis-di describe`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-describe) command (also available as `redis-di status`) to display its status. This combines the pipeline configuration with its runtime status, showing its overall state, its sources and targets, its jobs and components, and its per-stream statistics and performance metrics. ```bash redis-di describe ``` To watch the status update live, pair the command with `watch`: ```bash watch -n 1 redis-di describe ``` For a shorter overview, [`redis-di list`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-list) prints a one-line summary of the pipeline, and [`redis-di get`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-get) does the same for a single pipeline. See the [`redis-di describe`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-describe) reference page for details. ## Start and stop a pipeline Use [`redis-di stop`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-stop) to pause a running pipeline and [`redis-di start`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-start) to resume it. Stopping a pipeline halts data processing without deleting the pipeline or its configuration. A Debezium source restarts according to its [snapshot mode](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config#choose-a-snapshot-mode) and the availability of its saved source position. ```bash redis-di stop redis-di start ``` To act on a single source instead of the whole pipeline, add `--source`: ```bash redis-di stop --source mysql redis-di start --source mysql ``` Note that a source can only run if its parent pipeline is running. See [Multiple sources in one pipeline](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/multiple-sources) for more information. ## Reset a pipeline Use [`redis-di reset`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-reset) to clear the pipeline's saved source positions. With the default `initial` snapshot mode, a Debezium collector reloads selected source data and then resumes change data capture (CDC). Reset does not change the configured [snapshot mode](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config#choose-a-snapshot-mode). Modes that skip existing rows do not reload them after a reset, and `initial_only` mode does not capture subsequent changes. Reset does not flush the target Redis database. A new snapshot can overwrite target records, but [missed deletes can leave stale records](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config#missed-deletes). ```bash redis-di reset ``` Add `--source` to reset a single source. The other sources keep their data, but the whole pipeline stops while the reset runs and starts again afterwards: ```bash redis-di reset --source mysql ``` ## Clear a pipeline {#clear-a-pipeline} To stop a pipeline and discard its configuration, deploy an empty configuration with the `--empty` option of [`redis-di deploy`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-deploy): ```bash redis-di deploy --empty ``` RDI removes the pipeline's data plane components and deletes its data from the RDI database, including the change data streams, offsets, schema history, dead-letter queue entries, statistics, deduplication state, and record counters of every source. The pipeline itself remains, with an empty configuration, so you can deploy a new configuration to it at any time. The pipeline secrets are not affected, so remove them yourself with [`redis-di delete-secret`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-delete-secret) if you no longer need them. The records the pipeline wrote to the target database are not deleted either. > [!NOTE] > Clearing a pipeline discards the source aliases of any source that predates RDI's > support for multiple sources, so if you deploy such a source again under the same name you > must set its secrets again, and also change its secret references and job `server_name` values. See > [Redeploying a configuration after clearing a pipeline](https://redis.io/docs/latest/integrate/redis-data-integration/installation/upgrade#redeploying-a-configuration-after-clearing-a-pipeline) > for a before and after example. ## Wait for changes to complete {#wait} The commands that change a pipeline's state, namely `deploy`, `start`, `stop`, `reset`, `set-secret`, and `delete-secret`, do not return as soon as the API accepts the request. By default, they wait for the pipeline to finish transitioning to the expected state, polling its status until it succeeds, reaches an error, or the `--timeout` (2 minutes by default) elapses. This is usually what you want: the command reflects the real outcome, so a script can rely on the change having taken effect and can fail fast if it did not. In some cases, though, a pipeline needs *several* changes before it can transition to a healthy state, and waiting after each individual change would time out. The clearest example is rotating both the username and the password of a database: if you set only the username with the default `--wait=true`, the pipeline tries to reconnect with the new username and the old password, fails, and the command times out after two minutes with the pipeline in a broken state. To avoid this, set all the related secrets, or at least all of them except the last, with `--wait=false`, so the pipeline applies them together and only the final command waits for it to become healthy: ```bash redis-di set-secret USERNAME --db mysql newUsername --wait=false redis-di set-secret PASSWORD --db mysql newPassword ``` The same applies to any set of changes that are only valid together.