{ "schema_version": 2, "id": "operate/rc/rdi/setup", "title": "Prepare source database", "url": "https://redis.io/docs/latest/operate/rc/rdi/setup/", "summary": "Prepare your source database, network setup, and database credentials for Data integration.", "aliases": [ "/operate/rc/databases/rdi/setup/", "/operate/rc/databases/rdi/setup" ], "tags": [ "docs", "operate", "rc" ], "last_updated": "2026-10-19T20:49:52+03:00", "page_type": "content", "content_hash": "034b7c5c95dfab11a30202595a4aa2271a17e3cb932191a9dd73d6a099056f53", "sections": [ { "id": "prepare-source-database", "title": "Prepare source database", "role": "overview", "text": "Create the target Redis Cloud Pro database before starting the pipeline. See the [quickstart prerequisites](https://redis.io/docs/latest/operate/rc/rdi/quick-start#prerequisites).\n\nPrepare every source database before adding it to a pipeline. Each source needs its own change data capture (CDC) configuration, connectivity, and credentials. See [Prerequisites](https://redis.io/docs/latest/operate/rc/rdi#prerequisites) for a list of supported source databases and database versions.\n\nSee [Prepare source databases](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs) to find steps for your database type:\n- [MongoDB Atlas](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb)\n- [Snowflake](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake)\n- [Supabase](#supabase)\n- Hosted on an AWS EC2 instance:\n - [MySQL and mariaDB](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb)\n - [Oracle](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle)\n - [SQL Server](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server)\n - [PostgreSQL](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql)\n- Hosted on AWS RDS or AWS Aurora:\n - [AWS Aurora PostgreSQL and AWS RDS PostgreSQL](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql)\n - [AWS Aurora MySQL and AWS RDS MySQL](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql)\n - [AWS RDS SQL Server](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver)\n\nSee the [RDI architecture overview](https://redis.io/docs/latest/integrate/redis-data-integration/architecture#overview) for more information about CDC." }, { "id": "postgresql-publications", "title": "PostgreSQL publications", "role": "content", "text": "For PostgreSQL sources, including databases hosted on EC2, Aurora/RDS, and Supabase,\n[check replica identity](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql#check-replica-identity-and-choose-a-publication-mode)\nbefore deploying. Selecting tables in the console does not narrow an existing\n`FOR ALL TABLES` publication. That publication includes current and future tables, and\nupdates or deletes on any published table without replica identity can fail in your\napplication.\n\nFor a publication managed by your database administrator, create an explicit table list\nas shown in the [PostgreSQL preparation guide](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql#choose-a-publication-mode).\nIn **Configure source**, under **Collector properties**, open **Edit advanced properties**\nand enter these source properties:\n\n| Property | Value |\n|:--|:--|\n| `publication.name` | The name of the publication created by your administrator |\n| `publication.autocreate.mode` | `disabled` |\n\nKeep the publication's table list aligned with the source's selected tables. With\n`disabled`, RDI does not update the publication when you change the selection.\n\nTo let RDI create and manage a publication for selected tables, use `filtered` instead.\nThis requires [publication creation and table ownership permissions](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql#set-privileges-for-debezium-to-create-postgresql-publications-with-pgoutput).\nUse a separate publication for each source with different table selections.\n\nIf a source already uses `FOR ALL TABLES`, follow\n[Replace an existing all-tables publication](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql#replace-an-existing-all-tables-publication).\nChanging the collector property alone does not convert that publication." }, { "id": "supabase", "title": "Supabase", "role": "content", "text": "You can use a hosted [Supabase](https://supabase.com/) PostgreSQL database as\nthe source for an RDI pipeline on Redis Cloud. The integration was validated with\nRDI 1.19.0 and hosted Supabase PostgreSQL 17.6.\n\nFirst, follow the steps in [Prepare Supabase for RDI](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase)\nto create the database role, grant table access, account for Row Level\nSecurity, and create a publication. Then complete the Redis Cloud-specific steps below.\n\n> [!WARNING]\n> Supabase AWS PrivateLink connectivity isn't supported. Supabase shares a\n> Resource Configuration through AWS Resource Access Manager and requires a\n> Resource-type VPC endpoint. This differs from the AWS PrivateLink\n> endpoint-service connectivity supported by RDI on Redis Cloud, so you can skip\n> [Set up AWS Private Link connectivity](#set-up-connectivity) and use the Supabase\n> public direct database endpoint instead.\n\n#### Configure public connectivity\n\nSupabase logical replication requires the direct database endpoint. Don't use\na Supavisor transaction or session pooler endpoint.\n\nThe direct endpoint uses IPv6 by default, but RDI on Redis Cloud requires an\nIPv4 endpoint, so you must enable the Supabase\n[dedicated IPv4 add-on](https://supabase.com/docs/guides/platform/ipv4-address)\n(you need a paid Supabase plan to do this).\n\nWhen you create the RDI pipeline:\n\n1. Select **PostgreSQL** as the source type.\n1. Select **Public endpoint**.\n1. Copy every **Redis Cloud outbound IP address** displayed by the setup flow.\n1. In Supabase, open **Database settings** > **Network restrictions**.\n1. Add every Redis Cloud outbound address as a `/32` CIDR.\n\nIf you recreate the RDI workspace, its outbound addresses can change. Add the\nnew addresses to Supabase before starting the replacement pipeline, and remove\nthe old addresses after the new connection succeeds.\n\n#### Configure secrets\n\nFollow the steps in [Share source database credentials](#share-source-database-credentials)\nto create and share:\n\n- A credentials secret containing the dedicated Supabase `username` and\n `password`.\n- A plaintext CA certificate secret containing the certificate downloaded\n from Supabase **Database settings** > **SSL configuration**.\n\nEncrypt both secrets with the customer-managed AWS KMS key configured for the\nRDI workspace. Use the AWS region that contains your Redis Cloud subscription.\n\nIn the pipeline's **Secrets** section:\n\n1. Enter the credentials secret ARN.\n1. Select **TLS** under **Transit security**.\n1. Enter the CA certificate secret ARN.\n1. Select **Validate**.\n\nRDI on Redis Cloud uses TLS and validates the Supabase CA certificate.\n\n#### Configure the source\n\nIn the RDI pipeline setup flow, open the source configuration and enter the\nfollowing values:\n\n| Field | Value |\n|:--|:--|\n| Source IP address / Hostname | `db..supabase.co` |\n| Port | `5432` |\n| Database | `postgres` |\n\nUnder **Collector properties**, set:\n\n| Property | Value |\n|:--|:--|\n| `plugin.name` | `pgoutput` |\n| `publication.name` | `rdi_publication` |\n| `publication.autocreate.mode` | `disabled` |\n| `slot.name` | A unique value, such as `rdi_supabase` |\n\nThe publication name must match the publication you created in Supabase. Use a\nunique replication slot name for each active pipeline connected to the\nproject.\n\nSelect **Test source**. After the test succeeds, select the schemas and tables\nto capture and deploy the pipeline.\n\n#### Monitor the pipeline\n\nAfter deployment:\n\n1. Confirm the initial snapshot reaches zero pending and rejected records.\n1. Insert, update, and delete test records in Supabase.\n1. Confirm the corresponding counters increase in the pipeline metrics.\n\nSupabase logical replication slots retain write-ahead log (WAL) while the\npipeline is stopped. Follow the steps in\n[Monitor replication slots](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase#7-monitor-replication-slots)\nto monitor retained WAL and prepare for Supabase PostgreSQL upgrades." }, { "id": "get-cluster-account-id", "title": "Get cluster account ID", "role": "content", "text": "Before you can set up your source connectivity and secrets, you need the AWS Account ID for your Redis Cloud cluster so that you can give it access to your connectivity and secrets. \n\n1. On the [Redis Cloud console](https://cloud.redis.io/), open your target database's **Data Integration** tab.\n1. Select **Add pipeline**, or resume an existing draft. To add a source to a running pipeline, select **Add source** on its **Dashboard**.\n\n ![The Add pipeline control is available while the workspace is being created.](https://redis.io/docs/latest/images/rc/rdi/rdi-workspace-add-pipeline.png)\n\n1. For a new pipeline, complete **Settings**, including the target database, and select **Continue**.\n1. In **Add sources**, select the source type and enter a unique **Source name**. This name identifies the source in the pipeline configuration and transformation jobs. See [Add sources](https://redis.io/docs/latest/operate/rc/rdi/define#pipeline-setup) for naming rules.\n1. Select **Continue** to open **Configure source**.\n1. Under **Source connectivity**, copy the **Role ARN** and extract its AWS account ID.\n\n ![The source connectivity Role ARN and availability zones.](https://redis.io/docs/latest/images/rc/rdi/rdi-setup-connectivity-arn.png)\n\n The account ID is the number after `arn:aws:iam::`. For example, `arn:aws:iam::123456789012:role/redis-data-pipeline` contains account ID `123456789012`.\n\n1. For a source using **Public Endpoint**, also copy the Redis Cloud outbound IP address to add to the source database's allowlist.\n1. Select **Save & exit** to return to setup after preparing connectivity and secrets.\n\nRepeat the preparation for each source. Keep track of which endpoint service and secrets belong to each source; configuring one source does not configure the others." }, { "id": "set-up-connectivity", "title": "Set up AWS Private Link connectivity", "role": "content", "text": "> [!NOTE]\n> If your source database is accessible via a public endpoint and you want to use public connectivity for your data pipeline, proceed to [Share source database credentials](#share-source-database-credentials). \n\nIf your source database is not accessible via a public endpoint, you need to set up an endpoint service through AWS PrivateLink to be able to connect to it. For how traffic flows over PrivateLink, how to connect to an on-premises database over AWS Direct Connect, and how to keep the connection available during failover, see the [AWS PrivateLink reference](https://redis.io/docs/latest/operate/rc/rdi/networking/aws-privatelink).\n\nChoose the diagram and instructions for your source database:\n\n- Database hosted on an AWS EC2 instance:\n\n ![The network setup for a database hosted on an AWS EC2 instance.](https://redis.io/docs/latest/images/rc/rdi/rdi-setup-diagram-ec2.png)\n\n- Database hosted on AWS RDS or AWS Aurora:\n\n ![RDI connects to the RDS or Aurora writer through PrivateLink and an internal network load balancer. Lambda updates the target IP address after an RDS event.](https://redis.io/docs/latest/images/rc/rdi/rdi-setup-diagram-aurora-lambda.svg)\n\n Solid arrows show database connections. Dashed arrows show how Lambda updates the target group when the database address changes. Lambda does not carry database traffic.\n\nSelect the steps for your database setup.\n\n**EC2 instance:**\n\nTo set up PrivateLink for a database hosted on an EC2 instance:\n\n1. [Create a network load balancer](#create-network-load-balancer-ec2) that will route incoming TCP connections to your database.\n1. [Create an endpoint service](#create-endpoint-service-ec2) through AWS PrivateLink." }, { "id": "create-network-load-balancer-ec2", "title": "Create network load balancer", "role": "content", "text": "In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Compute** > **EC2**. [Create a network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html#configure-load-balancer) with the following settings:\n\n1. In **Basic configuration**: \n - **Scheme**: Select **Internal**.\n - **Load balancer IP address type**: Select **IPv4**.\n1. In **Network mapping**, select the VPC and availability zone associated with your source database.\n1. Create a security group for the network load balancer in the source database VPC:\n - **Inbound rules**: Leave empty. You will disable inbound rule checks for PrivateLink traffic in a later step.\n - **Outbound rules**: Allow TCP to the database security group on the database port (`5432` for PostgreSQL unless you configured another port).\n1. In the load balancer settings, select this group under **Security groups**.\n1. In **Listeners and routing**: \n 1. Select **Create target group** to [create a target group](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-target-group.html) with the following settings:\n 1. In **Specify group details**:\n - **Target type**: Select **Instances**.\n - **Protocol : Port**: Select **TCP**, and then enter the database port (`5432` for PostgreSQL unless you configured another port).\n - The **IP address type** and **VPC** should be selected already and match the VPC you selected earlier.\n 1. In **Health checks**, select **TCP** and use the traffic port.\n 1. In **Register targets**, select the EC2 instance that runs your source database, enter the port, and select **Include as pending below**. Then, select **Create target group** to create your target group. Return to **Listeners and routing** in the Network Load Balancer setup.\n 1. Set the following **Listener** properties:\n - **Protocol**: Select **TCP**.\n - **Port**: Enter your source database's port (`5432` for PostgreSQL unless you configured another port), matching the target group.\n - **Default action**: Select the target group you created in the previous step.\n1. Review the network load balancer settings, and then select **Create load balancer** to continue.\n1. After the network load balancer is active, select **Security**. \n\n Select **Edit**, deselect **Enforce inbound rules on PrivateLink traffic**, and select **Save changes**. This allows accepted PrivateLink connections through the load balancer without an inbound CIDR rule. Configure allowed principals and connection acceptance in the endpoint service steps that follow.\n\n1. Open the security group attached to your source database.\n1. Select **Edit inbound rules**, then **Add rule** to add a rule with the following settings:\n - **Type**: Select **Custom TCP**.\n - **Port range**: Enter the database port (`5432` for PostgreSQL unless you configured another port).\n - **Source**: Select the network load balancer security group ID. This rule allows both database connections and the TCP health checks configured on the same port.\n Select **Save rules** to save your changes.\n1. Return to the network load balancer and select **Actions** > **Edit Load Balancer Attributes**.\n - Under **Load balancer targets selection policy** select **Enable cross-zone load balancing**.\n Click the **Save Changes** button." }, { "id": "create-endpoint-service-ec2", "title": "Create endpoint service", "role": "content", "text": "In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Networking & Content Delivery** > **VPC**. There, select **PrivateLink and Lattice** > **Endpoint services**. [Create an endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) with the following settings:\n\n1. In **Available load balancers**, select the [network load balancer](#create-network-load-balancer-ec2) you created.\n1. In **Additional settings**, choose the following settings:\n - **Require acceptance for endpoint**: Select **Acceptance required**.\n - **Supported IP address types**: Select **IPv4**.\n1. Select **Create** to create the endpoint service.\n\nAfter you create the endpoint service, you need to add Redis Cloud as an Allowed Principal on your [endpoint service VPC permissions](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions). \n\n1. In the Redis Cloud Console, copy the Amazon Resource Name (ARN) provided in the **Setup connectivity** section.\n1. Return to the endpoint service list on the [Amazon VPC console](https://console.aws.amazon.com/vpc/). Select the endpoint service you just created.\n1. Navigate to **Allow principals** tab.\n1. Add the Redis Cloud ARN you copied and choose **Allow principals**.\n1. Save the service name for later. \n\nFor more details on AWS PrivateLink, see [Share your services through AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html).\n\n-\n\n**AWS RDS or Aurora:**\n\n-\n\nConnect the network load balancer (NLB) directly to your RDS or Aurora database. The database address can change after failover or maintenance. We recommend using the Lambda function to keep the NLB target group up to date.\n\n1. [Create a network load balancer](#create-network-load-balancer-rds) that forwards TCP connections to your database.\n1. [Create an endpoint service](#create-endpoint-service-rds) through AWS PrivateLink.\n1. [Set up the Lambda function](#setup-lambda-function) to update the target group when the database address changes.\n\n> [!NOTE]\n> RDS Proxy is not recommended for RDI and does not support PostgreSQL logical replication. If you need this alternative for a compatible database, see the [RDS Proxy setup guide](https://redis.io/docs/latest/operate/rc/rdi/rds-proxy)." }, { "id": "create-network-load-balancer-rds", "title": "Create network load balancer", "role": "content", "text": "In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Compute** > **EC2**. [Create a network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html#configure-load-balancer) with the following settings:\n\n1. In **Basic configuration**:\n - **Scheme**: Select **Internal**.\n - **Load balancer IP address type**: Select **IPv4**.\n1. In **Network mapping**, select the VPC and availability zone associated with your source database.\n1. Create a security group for the network load balancer in the source database VPC:\n - **Inbound rules**: Leave empty. You will disable inbound rule checks for PrivateLink traffic in a later step.\n - **Outbound rules**: Allow TCP to the database security group on the database port (`5432` for PostgreSQL unless you configured another port).\n1. In the load balancer settings, select this group under **Security groups**.\n1. In **Listeners and routing**:\n 1. Select **Create target group** to [create a target group](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-target-group.html) with the following settings:\n 1. In **Specify group details**:\n - **Target type**: Select **IP Addresses**.\n - **Protocol : Port**: Select **TCP**, and then enter the database port (`5432` for PostgreSQL unless you configured another port).\n - The **IP address type** and **VPC** should be selected already and match the VPC you selected earlier.\n 1. In **Health checks**, select **TCP** and use the traffic port.\n 1. In **Register targets**, enter the currently resolved private IPv4 address of your database, enter the port, and select **Include as pending below**. Then, select **Create target group** to create your target group. Return to **Listeners and routing** in the Network Load Balancer setup.\n\n To get the current private IPv4 address of your database, run the following command on an EC2 instance in the same VPC as the database:\n [code example]\n Replace `` with the writer endpoint of your RDS or Aurora database. The address can change after failover or maintenance. [Configure the existing failover Lambda](#setup-lambda-function) to update the target group when the writer address changes.\n 1. Set the following **Listener** properties:\n - **Protocol**: Select **TCP**.\n - **Port**: Enter your source database's port (`5432` for PostgreSQL unless you configured another port), matching the target group.\n - **Default action**: Select the target group you created in the previous step.\n1. Review the network load balancer settings, and then select **Create load balancer** to continue.\n1. After the network load balancer is active, select **Security**.\n\n Select **Edit**, deselect **Enforce inbound rules on PrivateLink traffic**, and select **Save changes**. This allows accepted PrivateLink connections through the load balancer without an inbound CIDR rule. Configure allowed principals and connection acceptance in the endpoint service steps that follow.\n\n1. Open the security group attached to your source database.\n\n1. Select **Edit inbound rules**, then **Add rule** to add a rule with the following settings:\n - **Type**: Select **Custom TCP**.\n - **Port range**: Enter the database port (`5432` for PostgreSQL unless you configured another port).\n - **Source**: Select the network load balancer security group ID. This rule allows both database connections and the TCP health checks configured on the same port.\n Select **Save rules** to save your changes.\n1. Return to the network load balancer and select **Actions** > **Edit Load Balancer Attributes**.\n - Under **Load balancer targets selection policy** select **Enable cross-zone load balancing**.\n Click the **Save Changes** button." }, { "id": "create-endpoint-service-rds", "title": "Create endpoint service", "role": "content", "text": "In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Networking & Content Delivery** > **VPC**. There, select **PrivateLink and Lattice** > **Endpoint services**. [Create an endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) with the following settings:\n\n1. In **Available load balancers**, select the [network load balancer](#create-network-load-balancer-rds) you created.\n1. In **Additional settings**, choose the following settings:\n - **Require acceptance for endpoint**: Select **Acceptance required**.\n - **Supported IP address types**: Select **IPv4**.\n1. Select **Create** to create the endpoint service.\n\nAfter you create the endpoint service, you need to add Redis Cloud as an Allowed Principal on your [endpoint service VPC permissions](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions). \n\n1. In the Redis Cloud Console, copy the Amazon Resource Name (ARN) provided in the **Setup connectivity** section.\n1. Return to the endpoint service list on the [Amazon VPC console](https://console.aws.amazon.com/vpc/). Select the endpoint service you just created.\n1. Navigate to **Allow principals** tab.\n1. Add the Redis Cloud ARN you copied and choose **Allow principals**.\n1. Save the service name for later. \n\nFor more details on AWS PrivateLink, see [Share your services through AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html)." }, { "id": "setup-lambda-function", "title": "Set up Lambda to update database addresses", "role": "content", "text": "> [!NOTE]\n> The Lambda function is optional but recommended for production. Without it, you must update the target group yourself when the database address changes.\n\nRDS sends event notifications through Amazon Simple Notification Service (SNS). SNS invokes the Lambda function. Lambda looks up the current writer address using the Domain Name System (DNS) and updates the NLB target group. RDI can reconnect after the new database target is ready.\n\n#### Option 1: Use the Redis Terraform module\n\nRedis provides a ready-to-use Terraform module that automates the Lambda function deployment. This is the recommended approach.\n\n##### Prerequisites\n\n- [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) >= 1.5.7\n- [AWS CLI](https://aws.amazon.com/cli/) configured with credentials\n- The ARNs from the Network Load Balancer and Endpoint Service you created in the previous steps\n\n##### Required variables\n\nBefore deploying the Lambda module, gather the following information:\n\n| Variable | Description | Where to find it |\n|----------|-------------|------------------|\n| `identifier` | A unique name for the Lambda resources | Choose a descriptive name (e.g., `rdi-failover-handler`) |\n| `db_endpoint` | Your RDS cluster or instance endpoint | AWS Console → RDS → Your database → Connectivity |\n| `db_port` | Your database port | AWS Console → RDS → Your database → Connectivity (default: `5432` for PostgreSQL, `3306` for MySQL, `1433` for SQL Server) |\n| `elb_tg_arn` | The NLB Target Group ARN | AWS Console → EC2 → Target Groups → Your target group |\n| `rds_arn` | The RDS cluster or instance ARN | AWS Console → RDS → Your database → Configuration |\n| `rds_cluster_identifier` | The RDS cluster identifier | AWS Console → RDS → Your cluster name |\n\n##### Deploy the Lambda module\n\n1. Clone the Redis cloud automation repository:\n\n [code example]\n\n1. Create a `terraform.tfvars` file with your configuration:\n\n [code example]\n\n1. Initialize and apply Terraform:\n\n [code example]\n\n##### How the Lambda function works\n\nThe deployed Lambda function:\n\n1. **Monitors RDS events**: Subscribes to RDS failover events via SNS\n1. **Detects failover**: When a failover occurs, RDS triggers an SNS notification\n1. **Resolves new IP**: The Lambda function queries DNS to get the new primary's IP address\n1. **Updates NLB target**: Automatically updates the NLB Target Group with the new IP\n\nThis process typically completes within 30-60 seconds for Aurora, or 60-120 seconds for standard RDS.\n\n##### Verify the deployment\n\nAfter deployment, verify the Lambda function is configured correctly:\n\n1. Check the Lambda function in AWS Console → Lambda → Functions\n1. Verify the environment variables are set correctly:\n - `Cluster_EndPoint`: Your RDS endpoint\n - `RDS_Port`: Your database port\n - `NLB_TG_ARN`: Your NLB Target Group ARN\n1. Check the SNS subscription in AWS Console → SNS → Subscriptions\n\n#### Option 2: Full infrastructure deployment\n\nFor new deployments, Redis provides a complete Terraform example that deploys the entire infrastructure including the RDS database, NLB, PrivateLink, and Lambda function.\n\nSee the [AWS RDS PrivateLink Failover Example](https://github.com/redis/rdi-cloud-automation/tree/main/examples/aws-rds-privatelink-failover) for:\n\n- Multi-engine support (PostgreSQL, MySQL, SQL Server)\n- Automatic CDC user creation\n- Complete VPC and networking setup\n- Lambda-based failover handling\n\n#### Option 3: Manual Lambda setup\n\nFor custom implementations, refer to the AWS documentation:\n[Access Amazon RDS across VPCs using AWS PrivateLink and Network Load Balancer](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/)\n\n-\n\n**MongoDB Atlas:**\n\n-\n\nTo set up Private Link for a MongoDB Atlas source database:\n\nMongoDB Atlas manages its own endpoint service. The flow is a two-way handshake — you get an endpoint service ID from Atlas, give it to Redis Cloud, and then take the VPC Endpoint ID that Redis Cloud returns back to Atlas to complete the connection.\n\n> [!NOTE]\n> Create the Atlas private endpoint in the same AWS region as your Redis Cloud target database." }, { "id": "create-a-private-endpoint-in-mongodb-atlas", "title": "Create a private endpoint in MongoDB Atlas", "role": "content", "text": "1. In the [MongoDB Atlas UI](https://cloud.mongodb.com/), go to **Security** > **Network Access**.\n1. Select the **Private Endpoint** tab, then select **Dedicated Cluster**.\n1. Select **Add Private Endpoint**.\n - Select **AWS** as the cloud provider.\n - Select the AWS region that matches your Redis Cloud target database.\n - Select **Next**.\n1. Atlas displays an **Endpoint Service ID** (for example, `vpce-svc-xxxxxxxxxxxxxxxxx`). Copy this value." }, { "id": "register-the-endpoint-service-with-redis-cloud", "title": "Register the endpoint service with Redis Cloud", "role": "content", "text": "1. In the Redis Cloud console, in the pipeline creation flow, go to the **Source connectivity** step.\n1. In the **Private Link service name** field, paste the Endpoint Service ID you copied from Atlas.\n1. Redis Cloud creates a VPC endpoint and displays a **VPC Endpoint ID** (for example, `vpce-xxxxxxxxxxxxxxxxx`). Copy this value." }, { "id": "complete-the-connection-in-mongodb-atlas", "title": "Complete the connection in MongoDB Atlas", "role": "content", "text": "1. Return to the **Add Private Endpoint** page in the Atlas UI. In the **Your VPC Endpoint ID** field, enter the VPC Endpoint ID you copied from the Redis Cloud console. Select **Create**.\n1. Wait for the endpoint status to show as **Available**. This can take a few minutes.\n1. In the Atlas UI, go to your cluster and select **Connect** > **Private Endpoint**.\n1. Choose the private endpoint you just registered (the `vpce-` ID you entered above).\n1. Choose a connection method, then select **Shell**.\n1. Copy the connection string shown.\n\n > [!NOTE]\n > Copy the connection string from the **Private Endpoint** connection method only. The standard connection string does not route traffic through the private endpoint." }, { "id": "finish-pipeline-setup", "title": "Finish pipeline setup", "role": "content", "text": "1. Return to the Redis Cloud pipeline creation flow and paste the connection string into the **Source configuration** section." }, { "id": "share-source-database-credentials", "title": "Share source database credentials", "role": "content", "text": "Share the credentials and certificates for each source through AWS Secrets Manager. Enter the matching secret ARNs when you configure that source in the console.\n\nTo do this, you need to:\n1. [Create an encryption key](#create-encryption-key) using AWS Key Management Service with the right permissions.\n1. [Create secrets](#create-database-credentials-secrets) containing the source database credentials encrypted using that key." }, { "id": "create-encryption-key", "title": "Create encryption key", "role": "content", "text": "In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Key Management Service**. [Create an encryption key](https://docs.aws.amazon.com/kms/latest/developerguide/create-symmetric-cmk.html) with the following settings:\n\n1. In **Step 1 - Configure key**:\n - **Key type**: Select **Symmetric**.\n - **Key usage**: Select **Encrypt and decrypt**.\n - Under **Advanced options**, set the following:\n - **Key material origin**: Select **KMS - recommended**.\n - **Regionality**: Select **Single-Region key**.\n1. In **Step 2 - Add labels**, add an alias and description for the key.\n1. In **Step 3 - Define key administrative permissions**, under **Key deletion**, select **Allow key administrators to delete this key**.\n1. In **Step 4 - Define key usage permissions**, under **Other AWS accounts**, select **Add another AWS account**. Enter the AWS account ID for the Redis Cloud cluster that you saved earlier.\n\nReview the key policy and key settings, and then select **Finish** to create the key." }, { "id": "create-database-credentials-secrets", "title": "Create database credentials secrets", "role": "content", "text": "To let Redis Cloud access your source database, you need to create AWS secrets for the source database's credentials and certificates. \n\nThe required secrets depend on your source database's security configuration. The following table shows the required secrets for each configuration:\n\n| Security configuration | Required secrets |\n| :-- | :-- |\n| Username and password only | |\n| Username and key pair only *(Snowflake source databases only)* | |\n| TLS connection | |\n| mTLS connection | |\n| mTLS connection with client key passphrase | |\n\n> [!NOTE]\n> When creating secrets for TLS or mTLS, ensure that all certificates and keys are in `PEM` format. The only exception to this is that for PostgreSQL, the source's private key secret must be in `DER` format. If you have a key in `PEM` format, you must convert it to `DER` before creating the secret using the command:\n\n[code example]\n\nThis command assumes that the private key is not encrypted. See the [`openssl` documentation](https://docs.openssl.org/master/) to learn how to convert an encrypted private key.\n\n\nSelect a tab to learn how to create the required secret.\n\n**Credentials secret:**\n\nIn the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings:\n\n- **Key/value pairs**: Enter the following key/value pairs.\n\n - `username`: Database username for the RDI pipeline user\n - `password`: Database password for the RDI pipeline user\n\n > [!NOTE]\n > Snowflake source databases that use user/key pair authentication should only enter the `username` for the database.\n\n* **Encryption key**: Select the [encryption key](#create-encryption-key) you created earlier.\n\n* **Resource permissions**: Add the following permissions to your secret to allow the Redis data pipeline to access your secret. Replace `` with the AWS account ID for the Redis Cloud cluster that you saved earlier.\n\n[code example]\n\nAfter you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. Save the secret ARN to use when you [define your source database](https://redis.io/docs/latest/operate/rc/rdi/define).\n\n-\n\n**CA Certificate secret:**\n\n-\n\nIn the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings:\n\n- **Key/value pairs**: Select **Plaintext** and enter the server certificate.\n\n* **Encryption key**: Select the [encryption key](#create-encryption-key) you created earlier.\n\n* **Resource permissions**: Add the following permissions to your secret to allow the Redis data pipeline to access your secret. Replace `` with the AWS account ID for the Redis Cloud cluster that you saved earlier.\n\n[code example]\n\nAfter you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. Save the secret ARN to use when you [define your source database](https://redis.io/docs/latest/operate/rc/rdi/define).\n\n-\n\n**Client certificate secret:**\n\n-\n\nIn the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings:\n\n- **Key/value pairs**: Select **Plaintext** and enter the client certificate.\n\n* **Encryption key**: Select the [encryption key](#create-encryption-key) you created earlier.\n\n* **Resource permissions**: Add the following permissions to your secret to allow the Redis data pipeline to access your secret. Replace `` with the AWS account ID for the Redis Cloud cluster that you saved earlier.\n\n[code example]\n\nAfter you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. Save the secret ARN to use when you [define your source database](https://redis.io/docs/latest/operate/rc/rdi/define).\n\n-\n\n**Client key secret:**\n\n-\n\nUse the [AWS CLI create-secret command](https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/create-secret.html) or the [AWS CreateSecret API endpoint](https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_CreateSecret.html) to create a binary secret containing the client key.\n\nFor example, using the AWS CLI, run the following command:\n\n[code example]\n\nWhere:\n- `` - Name of the secret\n- `` - Path to the client key file\n- `` - ARN of the [encryption key](#create-encryption-key) you created earlier\n\nAfter you create the secret, you need to add permissions to allow the data pipeline to access it. \n\nIn the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. Select the private key secret you just created and then select **Edit permissions**. \n\nAdd the following permissions to your secret. Replace `` with the AWS account ID for the Redis Cloud cluster that you saved earlier.\n\n[code example]\n\nAfter you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. Save the secret ARN to use when you [define your source database](https://redis.io/docs/latest/operate/rc/rdi/define).\n\n-\n\n**Client key passphrase secret:**\n\n-\n\nIn the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings:\n\n- **Key/value pairs**: Select **Plaintext** and enter the client key passphrase.\n\n* **Encryption key**: Select the [encryption key](#create-encryption-key) you created earlier.\n\n* **Resource permissions**: Add the following permissions to your secret to allow the Redis data pipeline to access your secret. Replace `` with the AWS account ID for the Redis Cloud cluster that you saved earlier.\n\n[code example]\n\nAfter you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. Save the secret ARN to use when you [define your source database](https://redis.io/docs/latest/operate/rc/rdi/define).\n\n-\n\n**Private key secret:**\n\n-\n\nThis secret is required for Snowflake source databases that use key-pair authentication.\n\nIn the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings:\n\n- **Key/value pairs**: Select **Plaintext** and enter the private key in plain text PEM format.\n\n* **Encryption key**: Select the [encryption key](#create-encryption-key) you created earlier.\n\n* **Resource permissions**: Add the following permissions to your secret to allow the Redis data pipeline to access your secret. Replace `` with the AWS account ID for the Redis Cloud cluster that you saved earlier.\n\n[code example]\n\nAfter you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. Save the secret ARN to use when you [define your source database](https://redis.io/docs/latest/operate/rc/rdi/define)." }, { "id": "next-steps", "title": "Next steps", "role": "content", "text": "After you have prepared connectivity and credentials for each source, resume your pipeline draft from the workspace and complete **Configure source**. Continue with [Create data pipeline](https://redis.io/docs/latest/operate/rc/rdi/define)." } ], "examples": [ { "id": "create-network-load-balancer-rds-ex0", "language": "sh", "code": "$ nslookup ", "section_id": "create-network-load-balancer-rds" }, { "id": "setup-lambda-function-ex0", "language": "bash", "code": "git clone https://github.com/redis/rdi-cloud-automation.git\n cd rdi-cloud-automation/modules/aws-rds-lambda", "section_id": "setup-lambda-function" }, { "id": "setup-lambda-function-ex1", "language": "hcl", "code": "identifier = \"rdi-failover-handler\"\n db_endpoint = \"your-cluster.cluster-xxxxxxxxx.us-east-1.rds.amazonaws.com\"\n db_port = 5432\n elb_tg_arn = \"arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/your-tg/xxxxxxxxx\"\n rds_arn = \"arn:aws:rds:us-east-1:123456789012:cluster:your-cluster\"\n rds_cluster_identifier = \"your-cluster\"", "section_id": "setup-lambda-function" }, { "id": "setup-lambda-function-ex2", "language": "bash", "code": "terraform init\n terraform apply", "section_id": "setup-lambda-function" }, { "id": "create-database-credentials-secrets-ex0", "language": "bash", "code": "openssl pkcs8 -topk8 -inform PEM -outform DER \\\n -in /path/to/myclient.pem \\\n -out /path/to/myclient.pk8 -nocrypt", "section_id": "create-database-credentials-secrets" }, { "id": "create-database-credentials-secrets-ex1", "language": "json", "code": "{\n \"Version\" : \"2012-10-17\",\n \"Statement\" : [ {\n \"Sid\" : \"RedisDataIntegrationRoleAccess\",\n \"Effect\" : \"Allow\",\n \"Principal\" : \"*\",\n \"Action\" : [ \"secretsmanager:GetSecretValue\", \"secretsmanager:DescribeSecret\" ],\n \"Resource\" : \"*\",\n \"Condition\" : {\n \"StringLike\" : {\n \"aws:PrincipalArn\" : \"arn:aws:iam:::role/redis-data-pipeline-secrets-role\"\n }\n }\n } ]\n}", "section_id": "create-database-credentials-secrets" }, { "id": "create-database-credentials-secrets-ex2", "language": "json", "code": "{\n \"Version\" : \"2012-10-17\",\n \"Statement\" : [ {\n \"Sid\" : \"RedisDataIntegrationRoleAccess\",\n \"Effect\" : \"Allow\",\n \"Principal\" : \"*\",\n \"Action\" : [ \"secretsmanager:GetSecretValue\", \"secretsmanager:DescribeSecret\" ],\n \"Resource\" : \"*\",\n \"Condition\" : {\n \"StringLike\" : {\n \"aws:PrincipalArn\" : \"arn:aws:iam:::role/redis-data-pipeline-secrets-role\"\n }\n }\n } ]\n}", "section_id": "create-database-credentials-secrets" }, { "id": "create-database-credentials-secrets-ex3", "language": "json", "code": "{\n \"Version\" : \"2012-10-17\",\n \"Statement\" : [ {\n \"Sid\" : \"RedisDataIntegrationRoleAccess\",\n \"Effect\" : \"Allow\",\n \"Principal\" : \"*\",\n \"Action\" : [ \"secretsmanager:GetSecretValue\", \"secretsmanager:DescribeSecret\" ],\n \"Resource\" : \"*\",\n \"Condition\" : {\n \"StringLike\" : {\n \"aws:PrincipalArn\" : \"arn:aws:iam:::role/redis-data-pipeline-secrets-role\"\n }\n }\n } ]\n}", "section_id": "create-database-credentials-secrets" }, { "id": "create-database-credentials-secrets-ex4", "language": "sh", "code": "aws secretsmanager create-secret \\\n --name \\\n --secret-binary fileb:// \\\n --kms-key-id ", "section_id": "create-database-credentials-secrets" }, { "id": "create-database-credentials-secrets-ex5", "language": "json", "code": "{\n \"Version\" : \"2012-10-17\",\n \"Statement\" : [ {\n \"Sid\" : \"RedisDataIntegrationRoleAccess\",\n \"Effect\" : \"Allow\",\n \"Principal\" : \"*\",\n \"Action\" : [ \"secretsmanager:GetSecretValue\", \"secretsmanager:DescribeSecret\" ],\n \"Resource\" : \"*\",\n \"Condition\" : {\n \"StringLike\" : {\n \"aws:PrincipalArn\" : \"arn:aws:iam:::role/redis-data-pipeline-secrets-role\"\n }\n }\n } ]\n}", "section_id": "create-database-credentials-secrets" }, { "id": "create-database-credentials-secrets-ex6", "language": "json", "code": "{\n \"Version\" : \"2012-10-17\",\n \"Statement\" : [ {\n \"Sid\" : \"RedisDataIntegrationRoleAccess\",\n \"Effect\" : \"Allow\",\n \"Principal\" : \"*\",\n \"Action\" : [ \"secretsmanager:GetSecretValue\", \"secretsmanager:DescribeSecret\" ],\n \"Resource\" : \"*\",\n \"Condition\" : {\n \"StringLike\" : {\n \"aws:PrincipalArn\" : \"arn:aws:iam:::role/redis-data-pipeline-secrets-role\"\n }\n }\n } ]\n}", "section_id": "create-database-credentials-secrets" }, { "id": "create-database-credentials-secrets-ex7", "language": "json", "code": "{\n \"Version\" : \"2012-10-17\",\n \"Statement\" : [ {\n \"Sid\" : \"RedisDataIntegrationRoleAccess\",\n \"Effect\" : \"Allow\",\n \"Principal\" : \"*\",\n \"Action\" : [ \"secretsmanager:GetSecretValue\", \"secretsmanager:DescribeSecret\" ],\n \"Resource\" : \"*\",\n \"Condition\" : {\n \"StringLike\" : {\n \"aws:PrincipalArn\" : \"arn:aws:iam:::role/redis-data-pipeline-secrets-role\"\n }\n }\n } ]\n}", "section_id": "create-database-credentials-secrets" } ] }