{ "schema_version": 2, "id": "integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server", "title": "Prepare SQL Server for RDI", "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server/", "summary": "Prepare SQL Server databases to work with RDI", "aliases": [ "/integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/sql-server/", "/integrate/redis-data-integration/ingest/data-transformation/prepare-dbs/sql-server/", "/integrate/redis-data-integration/ingest/installation/prepare-dbs/sql-server/" ], "tags": [ "docs", "integrate", "rs", "rdi" ], "last_updated": "2026-10-09T12:24:46+03:00", "page_type": "content", "content_hash": "64d6b86d9ee10c243a091ed24033f97b3f80fc5c68f943a603999ebc267497c2", "sections": [ { "id": "overview", "title": "Overview", "role": "overview", "text": "To prepare your SQL Server database for Debezium, you must first create a dedicated Debezium user,\nrun a script to enable CDC globally, and then separately enable CDC for each table you want to\ncapture. You need administrator privileges to do this.\n\nOnce you enable CDC, it captures all of the INSERT, UPDATE, and DELETE operations\non your chosen tables. The Debezium connector can then emit these events to RDI.\nRDI only reads from the source database; it captures changes through the CDC tables\nand never modifies the source data.\n\nThe following checklist summarizes the steps to prepare a SQL Server\ndatabase for RDI, with links to the sections that explain the steps in\nfull detail. You may find it helpful to track your progress with the\nchecklist as you complete each step.\n\n[code example]" }, { "id": "1-create-a-debezium-user", "title": "1. Create a Debezium user", "role": "content", "text": "It is strongly recommended to create a dedicated Debezium user for the connection between RDI\nand the source database. When using an existing user, ensure that the required \npermissions are granted and that the user is added to the CDC role.\n\n[code example]\n\n1. \n Create the Debezium user with the Transact-SQL below:\n\n [code example]\n\n Replace `MyUser`, `My_Password` and `MyDB` with your chosen values.\n\n1. \n Grant the user the necessary permissions:\n\n [code example]" }, { "id": "2-enable-cdc-on-the-database", "title": "2. Enable CDC on the database", "role": "content", "text": "There are two system stored procedures to enable CDC (you need\nadministrator privileges to run these). Use `sys.sp_cdc_enable_db`\nto enable CDC for the whole database and then `sys.sp_cdc_enable_table` to enable CDC for individual tables. \n\nBefore running the procedures, ensure that:\n\n- You are a member of the `sysadmin` fixed server role for the SQL Server.\n- You are a `db_owner` of the database.\n- The SQL Server Agent is running.\n\nThen, assuming your database is called `MyDB`, run the script below to enable CDC:\n\n[code example]\n\n> [!NOTE]\n> For SQL Server on AWS RDS, you must use a different stored procedure:\n> [code example]\n\nWhen you enable CDC for the database, it creates a schema called `cdc` and also\na CDC user, metadata tables, and other system objects." }, { "id": "3-enable-cdc-for-the-tables-you-want-to-capture", "title": "3. Enable CDC for the tables you want to capture", "role": "content", "text": "[code example]\n\n1. \n You must also enable CDC on the tables you want Debezium to capture using the\n following commands (again, you need administrator privileges for this):\n\n [code example]\n\n Repeat this for every table you want to capture.\n\n > [!NOTE]\n > The value for `@role_name` can’t be a fixed database role, such as `db_datareader`. \n > Specifying a new name will create a corresponding database role that has full access to the\n > captured change data.\n\n1. \n Add the Debezium user to the CDC role:\n\n [code example]" }, { "id": "4-check-that-you-have-access-to-the-cdc-table", "title": "4. Check that you have access to the CDC table", "role": "content", "text": "You can use another stored procedure `sys.sp_cdc_help_change_data_capture`\nto query the CDC information for the database and check you have enabled\nit correctly. To do this, connect as the Debezium user you created previously (`MyUser`).\n\n[code example]\n\n1. \n Run the `sys.sp_cdc_help_change_data_capture` stored procedure to query\n the CDC configuration. For example, if your database was called `MyDB` then you would\n run the following:\n\n [code example]\n\n1. \n The query returns configuration information for each table in the database that\n has CDC enabled and that contains change data that you are authorized to\n access. If the result is empty then you should check that you have privileges\n to access both the capture instance and the CDC tables." }, { "id": "troubleshooting", "title": "Troubleshooting", "role": "errors", "text": "If no CDC is happening then it might mean that SQL Server Agent is down. You can check for this using the SQL query shown below:\n\n[code example]\n\nIf the query returns a result of 0, you need to need to start SQL Server Agent using the following commands:\n\n[code example]" }, { "id": "5-using-sql-server-with-custom-data-fields", "title": "5. Using SQL Server with custom data fields", "role": "content", "text": "When using custom data types, you must ensure that the Debezium user has the necessary permissions to access \nthose types. You can grant these permissions using the following SQL commands:\n\n[code example]\n\nIf you don't grant these permissions you may encounter errors like:\n`\"The column \"Account\" is referenced as PRIMARY KEY, but a matching column is not defined in table \"DB.dbo.MyTable\"!\"`" }, { "id": "sql-server-capture-job-agent-configuration-parameters", "title": "SQL Server capture job agent configuration parameters", "role": "content", "text": "In SQL Server, the parameters that control the behavior of the capture job agent\nare defined in the SQL Server table `msdb.dbo.cdc_jobs`. If you experience performance\nproblems while running the capture job agent then you can adjust the capture jobs\nsettings to reduce CPU load. To do this, run the `sys.sp_cdc_change_job` stored procedure\nwith your new parameter values.\n\n> [!NOTE]\n> A full guide to configuring the SQL Server capture job agent parameters\n> is outside the scope of the Redis documentation.\n\nThe following parameters are the most important ones for modifying the capture agent behavior\nof the Debezium SQL Server connector:\n\n* `pollinginterval`: This specifies the number of seconds that the capture agent\n waits between log scan cycles. A higher value reduces the load on the database\n host, but increases latency. A value of 0 specifies no wait between scans.\n The default value is 5.\n* `maxtrans`: This specifies the maximum number of transactions to process during\n each log scan cycle. After the capture job processes the specified number of\n transactions, it pauses for the length of time that `pollinginterval` specifies\n before the next scan begins. A lower value reduces the load on the database host,\n but increases latency. The default value is 500.\n* `maxscans`: This specifies a limit on the number of scan cycles that the capture\n job can attempt when capturing the full contents of the database transaction log.\n If the continuous parameter is set to 1, the job pauses for the length of time\n that the `pollinginterval` specifies before it resumes scanning. A lower values\n reduces the load on the database host, but increases latency. The default value is 10.\n\nSee the SQL Server documentation for more information about capture agent parameters." }, { "id": "sql-server-on-azure", "title": "SQL Server on Azure", "role": "content", "text": "RDI can capture changes from Microsoft SQL Server hosted on Azure. The preparation\nsteps are similar to the on-premises instructions above, but Azure adds extra\nrequirements for authentication, networking, and (for Azure SQL Database) the\ndatabase service tier. Use the checklist below to track the additional steps.\n\n[code example]" }, { "id": "supported-azure-sql-products", "title": "Supported Azure SQL products", "role": "compatibility", "text": "| Product | Supported | Notes |\n| --- | --- | --- |\n| Azure SQL Database (single database or elastic pool) | Yes | Supported on any service tier in the vCore-based purchasing model. In the DTU-based purchasing model, CDC requires the S3 tier or higher — it is not supported on Basic, S0, S1, or S2. |\n| Azure SQL Managed Instance | Yes | Behaves like on-premises SQL Server. The SQL Server Agent is available and the on-premises CDC procedures apply unchanged. |\n| SQL Server on an Azure VM | Yes | Treat as on-premises — follow the [main SQL Server instructions](#1-create-a-debezium-user). The Azure-specific guidance below does not apply. |\n| Azure Synapse Analytics, Microsoft Fabric SQL database | No | These products do not support the SQL Server CDC features that Debezium relies on. |" }, { "id": "configure-network-access", "title": "Configure network access", "role": "configuration", "text": "The RDI connector must be able to reach the Azure SQL endpoint on the configured port\n(TCP 1433 by default; set via the `port` field in your RDI source configuration).\n\n- **Public endpoint**: add a server-level or database-level firewall rule that allows\n the public outbound IP address of the host running the RDI connector. See Microsoft's\n [Azure SQL firewall configuration](https://learn.microsoft.com/en-us/azure/azure-sql/database/firewall-configure)\n documentation for details.\n- **Private endpoint or VNet integration** (recommended for production): expose the\n Azure SQL server through a\n [private endpoint](https://learn.microsoft.com/en-us/azure/azure-sql/database/private-endpoint-overview)\n on the same VNet as the RDI connector — for example, when RDI runs on Azure\n Kubernetes Service.\n\nAzure SQL rejects unencrypted connections, so the RDI connection must always use TLS.\nThis is enforced by the [connector source settings](#configure-the-rdi-source-for-azure-sql)\ndescribed below." }, { "id": "enable-cdc-on-the-database-azure", "title": "Enable CDC on the database", "role": "content", "text": "The procedure depends on which Azure SQL product you are using.\n\n#### Azure SQL Database\n\nYou must be a member of the `db_owner` role on the database — Azure SQL Database has\nno `sysadmin` server role.\n\n> [!WARNING]\n> The identity used to enable CDC must match the type of identity that\n> created the database. If the database was created by a Microsoft Entra user, CDC must\n> be enabled (and later disabled) by a Microsoft Entra user; SQL logins cannot manage\n> CDC on it. The same restriction applies in reverse for databases created by SQL\n> logins.\n\nConnect to the user database and run:\n\n[code example]\n\nThis creates the `cdc` schema, the `cdc` database user, the CDC metadata tables, and\nother system objects in your database. Do not modify or drop these objects manually.\nThen enable CDC on each table you want to capture, using the same\n`sys.sp_cdc_enable_table` procedure described in the\n[on-premises instructions](#3-enable-cdc-for-the-tables-you-want-to-capture).\n\nCDC service-tier requirements differ between purchasing models:\n\n- **vCore-based purchasing model**: CDC is supported on any service tier, including\n General Purpose.\n- **DTU-based purchasing model**: CDC requires the S3 tier or higher. It is not\n supported on Basic, S0, S1, or S2.\n\nIf `sys.sp_cdc_enable_db` returns an error such as `Change data capture is not supported for this edition of SQL Server`,\nscale the database up before retrying.\n\n> [!NOTE]\n> Capture and cleanup run automatically on Azure SQL Database — there is no\n> SQL Server Agent. The internal scheduler runs the capture process every 20 seconds and\n> the cleanup process every hour, with a default change-data retention period of three\n> days. The capture cadence — the `pollinginterval` parameter described in the\n> [SQL Server capture job agent configuration parameters](#sql-server-capture-job-agent-configuration-parameters)\n> section — is fixed on Azure SQL Database and cannot be tuned. The `maxtrans` and\n> `maxscans` parameters from that section can still be adjusted via `sp_cdc_change_job`.\n\nEnabling CDC increases transaction log usage on Azure SQL Database because it disables\nthe aggressive log truncation behavior of Accelerated Database Recovery. You may need\nto scale the database to a higher service tier to provide enough transaction log\nthroughput for your workload combined with CDC. After a local or geo-replication\nfailover, CDC continues to operate automatically on the new primary; no manual\nreconfiguration is required.\n\nFor more information about CDC on Azure SQL Database, see Microsoft's\n[Change Data Capture with Azure SQL Database](https://learn.microsoft.com/en-us/azure/azure-sql/database/change-data-capture-overview?view=azuresql)\nguide.\n\n#### Reducing end-to-end latency on Azure SQL Database\n\nBecause the capture cadence on Azure SQL Database is fixed at ~20 seconds and\ncannot be tuned, the CDC step alone can add up to that much latency to your\nend-to-end change-propagation time. If your workload needs lower latency, you\ncan supplement the automatic Azure scheduler with an external worker that\nperiodically calls the `sys.sp_cdc_scan` stored procedure. The automatic\nscheduler continues to run; each manual call adds an extra CDC log scan in\nbetween, lowering the effective capture cadence to roughly the worker's\npolling interval.\n\nEach call runs one CDC log scan, bounded by the `maxtrans` and `maxscans`\nparameters covered in\n[SQL Server capture job agent configuration parameters](#sql-server-capture-job-agent-configuration-parameters).\nOn Azure SQL Database, `pollinginterval` and `continuous` do not apply, but\n`maxtrans` and `maxscans` remain tunable via `sp_cdc_change_job`. For low and\nmoderate change volumes the defaults are usually fine — each call drains the\npending transactions. For high-volume workloads, raise `maxtrans` and\n`maxscans` if a single call cannot keep up with the change rate.\n\nThis is a customer-operated workaround for an Azure platform limitation, not a\nRedis-supplied component. It does not apply to Azure SQL Managed Instance,\nSQL Server on Azure VM, or on-premises SQL Server — those use SQL Server Agent\nand the tunable `pollinginterval` parameter described in\n[SQL Server capture job agent configuration parameters](#sql-server-capture-job-agent-configuration-parameters).\n\n> [!WARNING]\n> Run **only one** instance of the scan worker per source database.\n> `sys.sp_cdc_scan` holds an exclusive log-reader lock for the duration of each\n> call; concurrent callers fail rather than running in parallel, so additional\n> replicas add no throughput and only generate error noise.\n\n##### Requirements\n\n- A database identity with permission to execute `sys.sp_cdc_scan`. This\n requires `db_owner` and is **more privilege than the Debezium user needs**, so\n create a separate login dedicated to the scan worker rather than reusing the\n RDI source credentials.\n- A single-replica runtime (a Kubernetes Deployment with `replicas: 1`, a\n systemd unit, a serverless cron with `maxConcurrency: 1`, or equivalent).\n- Network access from the worker to the Azure SQL endpoint on TCP 1433.\n\n##### Scan loop\n\nThe worker repeatedly opens a connection (or holds a long-lived one), runs\n`EXEC sys.sp_cdc_scan;` with a bounded command timeout, sleeps for the\nconfigured interval, and handles two expected error classes:\n\n- **Scan already in progress** — `sys.sp_cdc_scan` cannot run while another\n CDC log scan is active, either the Azure-internal scheduler's scan or a\n previous call from this worker that has not yet returned. The procedure\n returns a SQL error in this state. The error message has been observed to\n contain `sp_replcmds` (the underlying log-reader procedure), but the exact\n wording is not contractual — match by whatever signature your client\n surfaces, then log the occurrence and continue. Do not back off.\n- **Connection or transport errors** — close and reopen the connection with\n exponential backoff before the next attempt.\n\nThe example below shows the loop in pseudocode:\n\n[code example]\n\n##### Choosing the interval\n\nThe scan interval directly trades end-to-end latency against source-database\nload — each call reads the transaction log. Pick the largest interval that\nmeets your latency target:\n\n| Interval | Approximate CDC-step latency | Typical use |\n| --- | --- | --- |\n| No worker | Up to ~20s | Azure SQL Database default; the automatic scheduler runs every ~20s. |\n| 5s | Around 5s | Workload tolerates ~5s end-to-end. |\n| 2s | Around 2s under low to moderate load; can be higher under heavy write volume | Latency-sensitive workloads. Confirm the achieved latency under your own workload before relying on it. |\n\nIntervals below 1s are not recommended — each call has a fixed cost on the\nsource database and the marginal latency improvement is small.\n\n> [!WARNING]\n> CDC scans consume regular database resources. Every call reads\n> the transaction log, competing with the workload for CPU, memory, and log I/O.\n> An aggressive interval can degrade the source database, especially on lower\n> service tiers or under high write volume. Microsoft provides no SLA on CDC\n> freshness on Azure SQL Database; treat measured end-to-end latency under your\n> own workload as the source of truth, not the configured interval. If scans\n> start falling behind, raise the service tier, raise `maxtrans` and `maxscans`,\n> or relax the interval.\n\n##### Example Kubernetes deployment\n\nA minimal single-replica deployment skeleton — adapt the image, namespace, and\nsecret reference to your environment:\n\n[code example]\n\nThe secret referenced by `envFrom` must provide the credentials of the\n`db_owner` identity created for the scan worker — not the RDI source\ncredentials.\n\n##### Verifying the workaround\n\nAfter the worker has been running for a few minutes, confirm that the\neffective scan cadence has dropped to the worker's interval by querying the\n`sys.dm_cdc_log_scan_sessions` dynamic management view. This DMV records both\nthe automatic scheduler's scans and the worker's manual scans, so the gap\nbetween successive `start_time` values should now match the worker's interval:\n\n[code example]\n\nTo check the commit time of the latest change captured for a specific table,\nmap the highest captured LSN back to a time using `sys.fn_cdc_map_lsn_to_time`:\n\n[code example]\n\nThe difference between that value and the current time is an upper bound on\nhow stale the captured stream is for that table.\n\nYou can also confirm that end-to-end change propagation through RDI now meets\nyour latency target by measuring ` → `\non a representative table.\n\n#### Azure SQL Managed Instance\n\nFollow the on-premises instructions for\n[enabling CDC on the database](#2-enable-cdc-on-the-database) and\n[enabling CDC on the tables you want to capture](#3-enable-cdc-for-the-tables-you-want-to-capture).\nThe procedures are identical on Managed Instance." }, { "id": "create-a-database-user-for-debezium", "title": "Create a database user for Debezium", "role": "content", "text": "RDI only reads from the source database, so the Debezium user only needs read access.\nDo not grant `db_datawriter` or any other write permissions.\n\nYou can authenticate to Azure SQL using either SQL authentication or Microsoft Entra ID.\nMicrosoft Entra authentication with a service principal is the validated path for RDI;\nuse it for production deployments. SQL authentication is supported and is the simplest\noption for development or proof-of-concept setups.\n\n#### Option A: SQL authentication\n\nFollow the on-premises instructions for\n[creating the Debezium user](#1-create-a-debezium-user), with one Azure-specific\nchange: on Azure SQL Database, omit the `master`-database step and create a contained\nuser in the user database. Connect to your user database as the server admin and run:\n\n[code example]\n\nAfter enabling CDC on the tables you want to capture, add the user to the CDC role:\n\n[code example]\n\n> [!NOTE]\n> Use `VIEW DATABASE STATE` rather than `VIEW SERVER STATE`. The server-scoped\n> permission does not exist on Azure SQL Database.\n\n#### Option B: Microsoft Entra service principal\n\n1. **Register an application in Microsoft Entra ID.**\n In the Azure portal, go to **Microsoft Entra ID > App registrations > New registration**.\n Note the **Application (client) ID** — you'll use it as the RDI `user` value. Create\n a client secret under **Certificates & secrets** and note its value — you'll use it\n as the RDI `password` value.\n\n1. **Set a Microsoft Entra admin on the Azure SQL logical server.**\n In the Azure portal, open the logical SQL server and set a Microsoft Entra admin (a\n user or group you can sign in as). You will connect as this admin to create the\n contained user in the next step. (The permission to create contained users mapped\n to Microsoft Entra principals can also be delegated to other database principals;\n see Microsoft's [Microsoft Entra authentication for Azure SQL](https://learn.microsoft.com/en-us/azure/azure-sql/database/authentication-aad-overview)\n documentation.)\n\n1. **Create a contained database user for the service principal.**\n Connect to the user database as the Microsoft Entra admin (for example, using\n `sqlcmd -G` or Azure Data Studio) and run:\n\n [code example]\n\n After enabling CDC on the tables you want to capture, add the principal to the CDC role:\n\n [code example]\n\n > [!NOTE]\n > `` is the **display name** of the app registration — the\n > value shown in the **Name** column on the **App registrations** page — not its client\n > ID. The client ID is used by the RDI connector (see the next section), but the\n > database user must be created from the display name. If the display name is not\n > unique in your Microsoft Entra tenant (display names are not guaranteed unique),\n > disambiguate by adding the `WITH OBJECT_ID = ''` clause to the\n > `CREATE USER` statement." }, { "id": "configure-the-rdi-source-for-azure-sql", "title": "Configure the RDI source for Azure SQL", "role": "configuration", "text": "Use a `cdc` source with `type: sqlserver`. The example below shows the validated\nconfiguration for Azure SQL Database with Microsoft Entra service-principal\nauthentication:\n\n[code example]\n\nThe properties under `advanced.source` are passed straight through to the underlying\nDebezium SQL Server connector and JDBC driver. The Azure-specific values are:\n\n| Property | Purpose | Value for Azure SQL Database |\n| --- | --- | --- |\n| `driver.authentication` | Selects the JDBC Microsoft Entra authentication mode. | `ActiveDirectoryServicePrincipal` (validated). See [other Microsoft Entra authentication modes](#other-microsoft-entra-authentication-modes) for alternatives. |\n| `database.encrypt` | Enforces TLS on the JDBC connection. | `\"true\"`. Azure SQL rejects unencrypted connections. |\n| `database.trustServerCertificate` | If `true`, the driver skips certificate validation. | `\"false\"`. Azure SQL presents a valid certificate; never disable validation in production. |\n| `database.hostNameInCertificate` | Tells the JDBC driver which hostname pattern to expect in the server's TLS certificate. Set explicitly when the certificate's subject does not match the connection hostname directly. | `\"*.database.windows.net\"` (used in the RDI-validated configuration to match Azure SQL's wildcard certificate). |\n| `database.applicationIntent` | When set to `ReadOnly`, routes the connection to a read-only replica on tiers that support [read scale-out](https://learn.microsoft.com/en-us/azure/azure-sql/database/read-scale-out). | `ReadOnly`. Recommended because RDI only reads. On tiers where Azure SQL read scale-out is available (Business Critical and Hyperscale), this routes the RDI read connection to a read-only replica. On General Purpose, which has no read scale-out, the setting has no effect. |\n| `snapshot.mode` | The Debezium snapshot strategy. | `initial`. Captures a snapshot of the existing rows, then streams subsequent changes from the CDC tables. |\n\nFor SQL authentication, omit the `driver.authentication` line and set\n`${SQLSERVER_DB_USERNAME}` and `${SQLSERVER_DB_PASSWORD}` to the SQL user's credentials.\nKeep the other Azure-specific properties.\n\n#### Secret mapping\n\nFor Microsoft Entra service-principal authentication, the source's credentials must\nprovide:\n\n| Config reference | Value |\n| --- | --- |\n| `${SQLSERVER_DB_USERNAME}` | The service principal's **Application (client) ID** (a GUID). |\n| `${SQLSERVER_DB_PASSWORD}` | The service principal's **client secret**. |\n\nSet them with the source name, which is `sqlserver` for the source in this example:\n\n[code example]\n\n> [!WARNING]\n> The username value is the client ID (a GUID), but the contained\n> database user created in the previous section uses the service principal's **display\n> name**. These are two different identifiers for the same principal — mixing them up is\n> the most common cause of `Login failed for user ''` errors\n> at connection time.\n\n#### Other Microsoft Entra authentication modes\n\nThe Microsoft JDBC driver supports several other Microsoft Entra modes. The following\nare technically usable but are not currently validated by RDI — check with Redis\nsupport before using them in production:\n\n- **`ActiveDirectoryServicePrincipalCertificate`** — service principal authenticated by\n a certificate instead of a secret. Useful when organizational policy forbids\n long-lived shared secrets.\n- **`ActiveDirectoryManagedIdentity`** — for RDI installations running on an Azure\n resource (such as an Azure VM or Azure Kubernetes Service node) that has a system-\n or user-assigned managed identity.\n\nThe deprecated `ActiveDirectoryPassword` mode and the interactive\n`ActiveDirectoryInteractive` mode are not suitable for a server-side connector and are\nnot supported.\n\nSee Microsoft's\n[Connect using Microsoft Entra authentication](https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-using-azure-active-directory-authentication?view=sql-server-ver17)\nfor the full list of modes and their connection-string syntax." }, { "id": "verify-the-connection", "title": "Verify the connection", "role": "content", "text": "Connect to the database as the Debezium user (the SQL user or the Microsoft Entra\nservice principal) and run `sys.sp_cdc_help_change_data_capture` to confirm that the\nuser can see the captured tables. The query is the same as for\n[on-premises SQL Server](#4-check-that-you-have-access-to-the-cdc-table).\n\nYou can also confirm the database-level and table-level CDC state directly from the\ncatalog views:\n\n[code example]" }, { "id": "troubleshooting-1", "title": "Troubleshooting", "role": "errors", "text": "- **`Login failed for user ''`** — the contained database\n user was not created for this service principal, or it was created with the wrong\n identifier. Verify that the `CREATE USER ... FROM EXTERNAL PROVIDER` statement used\n the service principal's display name, and that `${SQLSERVER_DB_USERNAME}` contains its\n client ID. Query `sys.database_principals` on the source database to see which\n principals exist.\n- **`SSL Server certificate validation failed` or hostname mismatch** —\n `database.hostNameInCertificate` is missing or has the wrong value. For Azure SQL\n Database, set it to `\"*.database.windows.net\"` to match the wildcard certificate.\n- **`Change data capture is not supported for this edition of SQL Server`** — the\n Azure SQL Database is on an unsupported service tier. In the DTU purchasing model,\n scale up to S3 or higher. In the vCore purchasing model, CDC is supported on all\n tiers, so check that you are connecting to a standard Azure SQL Database (CDC is\n not supported on Azure SQL Edge or other variants).\n- **Connection timeouts** — the RDI connector's source IP is not allowed by the Azure\n SQL firewall, or the private endpoint is not reachable from the connector's network.\n Verify firewall rules in the Azure portal and that DNS resolves to the expected\n (public or private) endpoint." }, { "id": "handling-changes-to-the-schema", "title": "Handling changes to the schema", "role": "content", "text": "RDI can't adapt automatically when you change the schema of a CDC table in SQL Server. The existing capture instance continues to use its captured column structure. For example, adding a column can leave RDI streaming changes without that column instead of reporting an error. Create a new capture instance to capture the updated schema. See Debezium's [SQL Server schema evolution](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#sqlserver-schema-evolution) documentation for more information.\n\nIf you have administrator privileges, use this online procedure to update the capture instance while RDI is running. During the interval between changing the source schema and creating the new capture instance, events still use the old captured column structure. A short interval or low write traffic does not guarantee correct capture. If you cannot tolerate events that omit the new column, [update the schema offline](#update-the-schema-offline).\n\n[code example]\n\n1. \n Make your changes to the source table schema.\n\n1. \n Create a new capture table for the updated source table by running the `sys.sp_cdc_enable_table` stored\n procedure with a new, unique value for the parameter `@capture_instance`. For example, if the old value\n was `dbo_MyTable`, you could replace it with `dbo_MyTable_v2` (you can see the existing values by running\n stored procedure `sys.sp_cdc_help_change_data_capture`):\n\n [code example]\n\n1. \n When Debezium starts streaming from the new capture table and you no longer need the old captured changes, drop the old capture table by running the `sys.sp_cdc_disable_table` stored procedure with the parameter `@capture_instance` set to the old capture instance name, `dbo_MyTable`:\n\n Disabling the old instance deletes its change table and captured history. Rewinding an offset cannot recover changes that existed only in that capture table, even if SQL Server still retains them in the transaction log. If you need those changes for recovery or replay, retain the old instance until they are no longer needed. Retaining the instance does not prevent CDC cleanup from removing expired changes. Check the [CDC retention and capture instance limits](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server). SQL Server permits only two capture instances per source table, so retaining the old instance also affects the next schema migration.\n\n [code example]\n\n> [!NOTE]\n> Verify the values of the new or changed columns in the captured events after the migration. Continued streaming or a matching row count does not prove that the updated columns were captured." }, { "id": "update-the-schema-offline", "title": "Update the schema offline", "role": "content", "text": "Use this procedure to adapt Debezium's [offline schema update procedure](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#offline-schema-updates) to RDI. Keep the pipeline running and stop only the SQL Server source collector, so the processor and other source collectors remain available. Replace `` with your pipeline name and `` with the SQL Server source name from `config.yaml`.\n\n1. Suspend application writes to the tables captured by the SQL Server source collector.\n\n1. Wait for the final committed changes to reach the target Redis database. Use [`redis-di describe`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-describe) to check that the source's tables have zero pending records, no errors, and no new rejected records. Also verify the expected final changes in the target. Zero pending records alone does not prove that SQL Server's capture job has captured the final writes.\n\n [code example]\n\n1. Stop the SQL Server source collector with [`redis-di stop`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-stop). Wait for the command to complete successfully before changing the schema.\n\n [code example]\n\n1. [Apply the schema changes](#make-your-changes-to-the-source-table-schema) while application writes remain suspended.\n\n1. [Create a new capture instance](#create-a-new-capture-table-for-the-updated-source-table) with a unique name. Keep the old instance until RDI has switched to the new one and you no longer need its captured history.\n\n1. If your pipeline selects specific columns or references changed column names in transformations, update its configuration and [deploy it](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy) while application writes remain suspended.\n\n1. Restart the SQL Server source collector with [`redis-di start`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-start).\n\n [code example]\n\n1. Check `redis-di describe ` for source connectivity and errors before resuming application writes. Starting a source collector does not start a stopped pipeline.\n\n1. Resume application writes.\n\n1. Verify that changes made after the migration include the new or changed column values in the target as intended by your transformation jobs.\n\n1. [Drop the old capture instance](#drop-the-old-capture-table) only after RDI streams from the new instance and you no longer need the old captured history." } ], "examples": [ { "id": "overview-ex0", "language": "checklist {id=\"sqlserverlist\"}", "code": "- [ ] [Create a Debezium user](#1-create-a-debezium-user)\n- [ ] [Enable CDC on the database](#2-enable-cdc-on-the-database)\n- [ ] [Enable CDC for the tables you want to capture](#3-enable-cdc-for-the-tables-you-want-to-capture)\n- [ ] [Check that you have access to the CDC table](#4-check-that-you-have-access-to-the-cdc-table)", "section_id": "overview" }, { "id": "1-create-a-debezium-user-ex0", "language": "checklist {id=\"sqlserver-create-debezium-user\" nointeractive=\"true\" }", "code": "- [ ] [Create the Debezium user](#create-the-debezium-user)\n- [ ] [Grant the user the necessary permissions](#grant-the-user-the-necessary-permissions)", "section_id": "1-create-a-debezium-user" }, { "id": "1-create-a-debezium-user-ex1", "language": "sql", "code": "USE master\n GO\n CREATE LOGIN MyUser WITH PASSWORD = 'My_Password'\n GO\n USE MyDB\n GO\n CREATE USER MyUser FOR LOGIN MyUser\n GO", "section_id": "1-create-a-debezium-user" }, { "id": "1-create-a-debezium-user-ex2", "language": "sql", "code": "USE master\n GO\n GRANT VIEW SERVER STATE TO MyUser\n GO\n USE MyDB\n GO\n EXEC sp_addrolemember N'db_datareader', N'MyUser'\n GO", "section_id": "1-create-a-debezium-user" }, { "id": "2-enable-cdc-on-the-database-ex0", "language": "sql", "code": "USE MyDB\nGO\nEXEC sys.sp_cdc_enable_db\nGO", "section_id": "2-enable-cdc-on-the-database" }, { "id": "2-enable-cdc-on-the-database-ex1", "language": "sql", "code": "> EXEC msdb.dbo.rds_cdc_enable_db 'Chinook'\n> GO\n>", "section_id": "2-enable-cdc-on-the-database" }, { "id": "3-enable-cdc-for-the-tables-you-want-to-capture-ex0", "language": "checklist {id=\"sqlserver-enable-cdc-tables\" nointeractive=\"true\" }", "code": "- [ ] [Enable CDC on the tables you want to capture](#enable-cdc-on-the-tables-you-want-to-capture)\n- [ ] [Add the Debezium user to the CDC role](#add-the-debezium-user-to-the-cdc-role)", "section_id": "3-enable-cdc-for-the-tables-you-want-to-capture" }, { "id": "3-enable-cdc-for-the-tables-you-want-to-capture-ex1", "language": "sql", "code": "USE MyDB\n GO\n\n EXEC sys.sp_cdc_enable_table\n @source_schema = N'dbo',\n @source_name = N'MyTable', \n @role_name = N'MyRole', \n @supports_net_changes = 0\n GO", "section_id": "3-enable-cdc-for-the-tables-you-want-to-capture" }, { "id": "3-enable-cdc-for-the-tables-you-want-to-capture-ex2", "language": "sql", "code": "USE MyDB\n GO\n EXEC sp_addrolemember N'MyRole', N'MyUser'\n GO", "section_id": "3-enable-cdc-for-the-tables-you-want-to-capture" }, { "id": "4-check-that-you-have-access-to-the-cdc-table-ex0", "language": "checklist {id=\"sqlserver-check-cdc-table\" nointeractive=\"true\" }", "code": "- [ ] [Run the stored procedure to query the CDC configuration](#run-the-stored-procedure-to-query-the-cdc-configuration)\n- [ ] [Check the results](#check-the-results)", "section_id": "4-check-that-you-have-access-to-the-cdc-table" }, { "id": "4-check-that-you-have-access-to-the-cdc-table-ex1", "language": "sql", "code": "USE MyDB;\n GO\n EXEC sys.sp_cdc_help_change_data_capture\n GO", "section_id": "4-check-that-you-have-access-to-the-cdc-table" }, { "id": "troubleshooting-ex0", "language": "sql", "code": "IF EXISTS (SELECT 1 \n FROM master.dbo.sysprocesses \n WHERE program_name = N'SQLAgent - Generic Refresher')\nBEGIN\n SELECT @@SERVERNAME AS 'InstanceName', 1 AS 'SQLServerAgentRunning'\nEND\nELSE \nBEGIN\n SELECT @@SERVERNAME AS 'InstanceName', 0 AS 'SQLServerAgentRunning'\nEND", "section_id": "troubleshooting" }, { "id": "troubleshooting-ex1", "language": "sql", "code": "EXEC xp_servicecontrol N'START',N'SQLServerAGENT';\nGO", "section_id": "troubleshooting" }, { "id": "5-using-sql-server-with-custom-data-fields-ex0", "language": "sql", "code": "USE ;\n\nGRANT REFERENCES ON TYPE::. TO ;", "section_id": "5-using-sql-server-with-custom-data-fields" }, { "id": "sql-server-on-azure-ex0", "language": "checklist {id=\"sqlserverazurelist\"}", "code": "- [ ] [Confirm your Azure SQL product and service tier](#supported-azure-sql-products)\n- [ ] [Configure network access](#configure-network-access)\n- [ ] [Enable CDC on the database](#enable-cdc-on-the-database-azure)\n- [ ] [Create a database user for Debezium](#create-a-database-user-for-debezium)\n- [ ] [Configure the RDI source for Azure SQL](#configure-the-rdi-source-for-azure-sql)\n- [ ] [Verify the connection](#verify-the-connection)", "section_id": "sql-server-on-azure" }, { "id": "enable-cdc-on-the-database-azure-ex0", "language": "sql", "code": "EXEC sys.sp_cdc_enable_db\nGO", "section_id": "enable-cdc-on-the-database-azure" }, { "id": "enable-cdc-on-the-database-azure-ex1", "language": "text", "code": "loop until shutdown:\n start = now()\n try:\n EXEC sys.sp_cdc_scan # command timeout: 30s\n log(\"scan_ok\", now() - start)\n catch SqlException identifying \"scan already active\":\n log(\"scan_already_running\", now() - start)\n catch any other exception as e:\n log(\"scan_error\", e)\n reconnect with exponential backoff\n sleep(max(0, interval - (now() - start)))", "section_id": "enable-cdc-on-the-database-azure" }, { "id": "enable-cdc-on-the-database-azure-ex2", "language": "yaml", "code": "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: azure-sql-cdc-scan-worker\nspec:\n replicas: 1\n selector:\n matchLabels:\n app: azure-sql-cdc-scan-worker\n template:\n metadata:\n labels:\n app: azure-sql-cdc-scan-worker\n spec:\n containers:\n - name: worker\n image: /:\n env:\n - name: SQL_HOST\n value: .database.windows.net\n - name: SQL_DATABASE\n value: \n - name: SCAN_INTERVAL_MS\n value: \"2000\"\n envFrom:\n - secretRef:\n name: ", "section_id": "enable-cdc-on-the-database-azure" }, { "id": "enable-cdc-on-the-database-azure-ex3", "language": "sql", "code": "-- Recent CDC log-scan sessions (manual and automatic combined)\nSELECT TOP (10)\n session_id, start_time, end_time, duration, scan_phase,\n latency, tran_count, last_commit_cdc_time\nFROM sys.dm_cdc_log_scan_sessions\nWHERE session_id > 0\nORDER BY session_id DESC", "section_id": "enable-cdc-on-the-database-azure" }, { "id": "enable-cdc-on-the-database-azure-ex4", "language": "sql", "code": "-- Replace with the capture instance name shown by\n-- sys.sp_cdc_help_change_data_capture\nSELECT sys.fn_cdc_map_lsn_to_time(MAX(__$start_lsn)) AS latest_captured_commit_time\nFROM cdc._CT", "section_id": "enable-cdc-on-the-database-azure" }, { "id": "create-a-database-user-for-debezium-ex0", "language": "sql", "code": "CREATE USER WITH PASSWORD = ''\nGO\nALTER ROLE db_datareader ADD MEMBER \nGO\nGRANT VIEW DATABASE STATE TO \nGO", "section_id": "create-a-database-user-for-debezium" }, { "id": "create-a-database-user-for-debezium-ex1", "language": "sql", "code": "EXEC sp_addrolemember N'', N''\nGO", "section_id": "create-a-database-user-for-debezium" }, { "id": "create-a-database-user-for-debezium-ex2", "language": "sql", "code": "CREATE USER [] FROM EXTERNAL PROVIDER\n GO\n ALTER ROLE db_datareader ADD MEMBER []\n GO\n GRANT VIEW DATABASE STATE TO []\n GO", "section_id": "create-a-database-user-for-debezium" }, { "id": "create-a-database-user-for-debezium-ex3", "language": "sql", "code": "EXEC sp_addrolemember N'', N''\n GO", "section_id": "create-a-database-user-for-debezium" }, { "id": "configure-the-rdi-source-for-azure-sql-ex0", "language": "yaml", "code": "sources:\n sqlserver:\n type: cdc\n connection:\n type: sqlserver\n host: .database.windows.net\n port: 1433\n database: \n user: ${SQLSERVER_DB_USERNAME}\n password: ${SQLSERVER_DB_PASSWORD}\n logging:\n level: info\n schemas:\n - dbo\n tables:\n :\n columns:\n - \n - \n keys:\n - \n advanced:\n source:\n driver.authentication: ActiveDirectoryServicePrincipal\n database.encrypt: \"true\"\n database.hostNameInCertificate: \"*.database.windows.net\"\n database.trustServerCertificate: \"false\"\n database.applicationIntent: ReadOnly\n snapshot.mode: initial", "section_id": "configure-the-rdi-source-for-azure-sql" }, { "id": "configure-the-rdi-source-for-azure-sql-ex1", "language": "bash", "code": "redis-di set-secret USERNAME --db sqlserver \nredis-di set-secret PASSWORD --db sqlserver ", "section_id": "configure-the-rdi-source-for-azure-sql" }, { "id": "verify-the-connection-ex0", "language": "sql", "code": "-- Check whether CDC is enabled on the database\nSELECT name, is_cdc_enabled FROM sys.databases WHERE name = ''\nGO\n\n-- Check which tables in the current database have CDC enabled\nSELECT name, is_tracked_by_cdc FROM sys.tables WHERE is_tracked_by_cdc = 1\nGO", "section_id": "verify-the-connection" }, { "id": "handling-changes-to-the-schema-ex0", "language": "checklist {id=\"sqlserver-schema-changes\" nointeractive=\"true\" }", "code": "- [ ] [Make your changes to the source table schema](#make-your-changes-to-the-source-table-schema)\n- [ ] [Create a new capture table for the updated source table](#create-a-new-capture-table-for-the-updated-source-table)\n- [ ] [Drop the old capture table](#drop-the-old-capture-table)", "section_id": "handling-changes-to-the-schema" }, { "id": "handling-changes-to-the-schema-ex1", "language": "sql", "code": "EXEC sys.sp_cdc_enable_table\n @source_schema = N'dbo',\n @source_name = N'MyTable',\n @role_name = N'MyRole',\n @capture_instance = N'dbo_MyTable_v2',\n @supports_net_changes = 0\n GO", "section_id": "handling-changes-to-the-schema" }, { "id": "handling-changes-to-the-schema-ex2", "language": "sql", "code": "EXEC sys.sp_cdc_disable_table\n @source_schema = N'dbo',\n @source_name = N'MyTable',\n @capture_instance = N'dbo_MyTable'\n GO", "section_id": "handling-changes-to-the-schema" }, { "id": "update-the-schema-offline-ex0", "language": "bash", "code": "redis-di describe ", "section_id": "update-the-schema-offline" }, { "id": "update-the-schema-offline-ex1", "language": "bash", "code": "redis-di stop --source ", "section_id": "update-the-schema-offline" }, { "id": "update-the-schema-offline-ex2", "language": "bash", "code": "redis-di start --source ", "section_id": "update-the-schema-offline" } ] }