{
  "schema_version": 2,
  "id": "integrate/redis-data-integration/data-pipelines/pipeline-config",
  "title": "Pipeline configuration file",
  "url": "https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config/",
  "summary": "Learn how to specify the main configuration details for an RDI pipeline.",
  "aliases": [
    "/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines/"
  ],
  "tags": [
    "docs",
    "integrate",
    "rs",
    "rdi"
  ],
  "last_updated": "2026-10-13T08:44:20+03:00",
  "page_type": "content",
  "content_hash": "f08f10dc1ef3a05197e89048539a029db7d26b9188014521ac0688d4cc3592c9",
  "sections": [
    {
      "id": "overview",
      "title": "Overview",
      "role": "overview",
      "text": "The main configuration details for an RDI pipeline are in the `config.yaml` file.\nThis file specifies the connection details for the source and target databases,\nand also the set of tables you want to capture. You can also add one or more\n[job files](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples)\nif you want to apply custom transformations to the captured data.\n\nEach section explains one part of the file. Start with the minimal example, then\nadd only the optional properties that you need. See the\n[configuration file reference](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference)\nfor all supported properties."
    },
    {
      "id": "before-you-start",
      "title": "Before you start",
      "role": "content",
      "text": "Before you create `config.yaml`:\n\n1. [Prepare the source database](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs) for change data capture.\n1. [Install RDI](https://redis.io/docs/latest/integrate/redis-data-integration/installation).\n1. [Set the secrets](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#set-secrets) that the file references."
    },
    {
      "id": "start-with-a-minimal-file",
      "title": "Start with a minimal file",
      "role": "content",
      "text": "The following example shows the required structure of a `config.yaml` file. Values of the\nform \"`${name}`\" refer to secrets that you should set as described in\n[Set secrets](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#set-secrets).\nIn particular, you should normally use secrets as shown to set the source\nand target username and password rather than storing them in plain text in this file.\n\n[code example]\n\nThe processor properties all have defaults, so add them only when you need to change the\ndefault behavior. In particular, the pipeline runs on the Flink processor unless you set\n`type` to `classic`."
    },
    {
      "id": "build-the-file-with-an-ai-assistant",
      "title": "Build the file with an AI assistant",
      "role": "content",
      "text": "Copy the following prompt into your AI assistant. The prompt tells the assistant\nto use the RDI documentation as its source of truth and to flag unsupported requests.\n\n[code example]"
    },
    {
      "id": "sections",
      "title": "Sections",
      "role": "content",
      "text": "The main sections of the file configure [`sources`](#sources), [`targets`](#targets),\nand [`processors`](#processors)."
    },
    {
      "id": "sources",
      "title": "Sources",
      "role": "content",
      "text": "The `sources` section has one subsection per source database. Each subsection starts\nwith the source name, which identifies the source and must be unique (in the example,\nthe source is called `mysql`).\n\nRDI also derives the environment variables that contain the source's credentials from the\nsource name, for example `${MYSQL_DB_USERNAME}` and `${MYSQL_DB_PASSWORD}` for a source\nnamed `mysql`. See\n[Multiple sources in one pipeline](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/multiple-sources)\nfor the source naming rules and for capturing from more than one source database. For a source\nthat existed before RDI supported multiple sources, these variables are still named\n`SOURCE_DB_*`; see\n[Existing names are kept after an upgrade](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade).\n\nThe example configuration contains the following data:\n\n- `type`: The collector to use for the pipeline. Use `cdc` for MariaDB, MySQL,\n  MongoDB, Oracle, PostgreSQL, or SQL Server. Use `flink` for Google Cloud\n  Spanner. Use `riotx` for Snowflake. Use `external` when you provide and manage\n  the collector. RDI doesn't create collector resources for an `external` source,\n  so omit the other properties in the source section.\n- `connection`: The connection details for the source database: `type`, `host`, `port`,\n  and credentials (`user` and `password`).\n  See the [configuration file reference](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#sourcesconnection)\n  for the required fields for each source database type.\n  - If you use [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security)/\n    or [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) to connect\n    to the source database, you may need to specify additional properties in the\n    `advanced` section with references to the corresponding certificates depending\n    on the source database type. Note that these properties **must** be references to\n    secrets that you should set as described in [Set secrets](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#set-secrets).\n- `databases`: List of all databases to collect data from for source database types\n  that support multiple databases, such as `mysql` and `mariadb`.\n- `schemas`: List of all schemas to collect data from for source database types\n  that support multiple schemas, such as `oracle`, `postgresql`, and `sqlserver`.\n- `tables`: List of all tables to collect data from. Each table is identified by its\n  full name, including a database or schema prefix. If there is a single\n  database or schema, this prefix can be omitted.\n  For each table, you can specify:\n  - `columns`: A list of the columns you are interested in (the default is to\n    include all columns)\n  - `keys`: A list of columns to create a composite key if your table\n    doesn't already have a [`PRIMARY KEY`](https://www.w3schools.com/sql/sql_primarykey.asp) or\n    [`UNIQUE`](https://www.w3schools.com/sql/sql_unique.asp) constraint.\n  - `snapshot_sql`: A query to be used when performing the initial snapshot.\n    By default, a query that contains all listed columns of all listed tables will be used.\n- `advanced`: These optional properties configure other Debezium-specific features.\n  The available sub-sections are:\n  - `source`: Properties for reading from the source database.\n    See the Debezium [Source connectors](https://debezium.io/documentation/reference/stable/connectors/)\n    pages for more information about the properties available for each database type.\n  - `sink`: Properties for writing to Redis streams in the RDI database.\n    See the Debezium [Redis stream properties](https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream)\n    page for the full set of available properties.\n  - `quarkus`: Properties for the Debezium server, such as the log level. See the\n    Quarkus [Configuration options](https://quarkus.io/guides/all-config)\n    docs for the full set of available properties.\n  - `java_options`: controls the JAVA_OPTS environment variable (for RDI 1.15.1 and above). Use it to modify the default values for Java heap size and other Java options for the Debezium server.\n    For example, set it to `\"-Xmx2g -Xms512m\"` to set the maximum heap size to 2 GB and the initial heap size to 512 MB."
    },
    {
      "id": "choose-a-snapshot-mode",
      "title": "Choose a snapshot mode",
      "role": "content",
      "text": "Use `sources.<source-name>.advanced.source.snapshot.mode` to control when a\nDebezium collector loads existing source data before streaming changes. This property\napplies to sources with `type: cdc`, including MariaDB, MongoDB, MySQL, Oracle,\nPostgreSQL, and SQL Server. It does not configure Spanner, Snowflake, or external\ncollectors.\n\nAdd the property under the source you want to configure. For example, this file\nuses `initial` mode for a MySQL source:\n\n[code example]\n\nRDI 1.19.1 and RDI 2.0.0 bundle Debezium 3.5. The comparison and connector\nnotes in this section apply to those releases. RDI 2.2.0 is planned to use\nDebezium 3.7.\n\nThe collector saves an *offset*, which is its position in the source's change history.\nA missing offset means there is no saved position, as on first deployment or after a\nreset. An unavailable position means an offset exists, but the source no longer has\nthe log history needed to resume from it.\n\nWith the default `initial` mode, the collector snapshots when there is no saved\nposition, then streams changes. A restart with a completed snapshot and a valid saved\nposition resumes streaming. Deployment or restart alone does not require a new snapshot.\nIf the saved position is no longer available, `initial` does not automatically\nre-snapshot to recover, and the collector stops and reports an error.\n\nWith `when_needed` mode, the collector also takes a new snapshot when it detects that the\nsaved position is no longer available. This can reload all selected data after an\noutage without a manual reset. The snapshot captures current rows, rather than\nreconstructing every missed change.\n\nIf the collector restarts during a snapshot, the snapshot starts again. Rows deleted\nwhile the snapshot is interrupted can remain as stale records in the target because\nthe new snapshot does not replay their delete events.\n\nUse `when_needed` with monitoring and alerting. Recovery can start a long-running\nsnapshot, which can increase latency. Missed deletes can leave stale target records\neven after the snapshot finishes.\n\n#### Missed deletes can leave stale target records {#missed-deletes}\n\nA new snapshot copies existing source rows. It does not replay missed deletes or remove\ntarget records for rows absent from the snapshot.\n\nThe timeline shows one row whose delete event is lost before collection resumes.\nTime progresses from top to bottom.\n\n[code example]\n\nAn interruption alone does not cause this gap. If the saved position and the required\nlog history are still available, the collector can resume and capture the delete.\nIn this example, the history expires before the collector resumes, and the new snapshot\ncontains no row 42 to overwrite or delete its target record. Streaming resumes after\nthe snapshot, but the stale record can remain in Redis.\n\n\n#### Compare snapshot modes\n\nThe available modes and their exact behavior depend on the source connector and its\nversion. The following table describes the six values listed in the RDI configuration\nreference. A listed value is not necessarily supported by every connector.\n\n| Mode | Existing source data | Restart with a saved position | After a reset clears the position |\n| :-- | :-- | :-- | :-- |\n| `initial` (default) | Snapshot when no offset exists, then stream changes. | Resume after a completed snapshot if the position is valid. No automatic snapshot recovery for unavailable log history. | Snapshot, then stream changes. |\n| `when_needed` | Snapshot when no offset exists, then stream changes. | Resume if the position is valid. Snapshot again if the connector detects an unavailable position. | Snapshot, then stream changes. |\n| `always` | Snapshot on every collector start, then stream changes. | Take a new snapshot even if the position is valid. | Snapshot, then stream changes. |\n| `initial_only` | Take an initial snapshot without streaming subsequent changes. | Restart an interrupted initial snapshot; do not capture subsequent changes after it completes. | Snapshot without ongoing change capture. |\n| `no_data` | Skip existing rows. Initialize any required schema or source position, then stream changes. | Resume streaming if the position is valid. No data snapshot to recover unavailable history. | Skip existing rows and start streaming according to the connector's rules. |\n| `never` | Skip the data snapshot where this mode is supported. | Resume according to the connector's rules; no data snapshot recovery. | Connector-specific startup behavior; existing rows are not reloaded. |\n\n\nOn Kubernetes deployments, the collector exits after completing an `initial_only`\nsnapshot. Kubernetes restarts it, but the collector sees that the snapshot is complete\nand exits again. This restart loop can make the source appear disconnected in\nmonitoring. Check snapshot completion separately from collector connectivity;\n`initial_only` mode does not capture subsequent changes.\n\n\nAn interrupted initial snapshot can restart when the collector starts again, including\nwith `initial` and `initial_only`.\n\nFor Debezium 3.5, note these connector differences:\n\n- PostgreSQL `no_data` performs no snapshot. It resumes from a saved log sequence\n  number (LSN), or from the replication slot's creation position when no LSN is saved.\n  This requires the relevant write-ahead log (WAL) history to remain available.\n- Other connectors can capture schema or initialize a position without copying\n  existing rows in `no_data` mode.\n- `never` is a deprecated alias for `no_data` in PostgreSQL and MongoDB. MySQL and\n  MariaDB retain `never` with connector-specific prerequisites. Oracle and SQL Server\n  do not offer `never`; use `no_data` to skip existing rows.\n\nSee the Debezium 3.5 connector references for [MariaDB](https://debezium.io/documentation/reference/3.5/connectors/mariadb.html),\n[MongoDB](https://debezium.io/documentation/reference/3.5/connectors/mongodb.html),\n[MySQL](https://debezium.io/documentation/reference/3.5/connectors/mysql.html),\n[Oracle](https://debezium.io/documentation/reference/3.5/connectors/oracle.html),\n[PostgreSQL](https://debezium.io/documentation/reference/3.5/connectors/postgresql.html),\nand [SQL Server](https://debezium.io/documentation/reference/3.5/connectors/sqlserver.html).\n\n\nA reset clears saved positions without changing `snapshot.mode` or flushing the target\nRedis database. If you need to reload existing rows, select a mode that takes a data\nsnapshot. A snapshot can overwrite target records, but it does not remove stale records\nfor rows that are absent from the source. See\n[Recover from unavailable source log history](https://redis.io/docs/latest/integrate/redis-data-integration/troubleshooting#unavailable-source-history)."
    },
    {
      "id": "targets",
      "title": "Targets",
      "role": "content",
      "text": "Use this section to provide the connection details for the target Redis\ndatabase. RDI supports one target database. Name the target `target`.\nIn the `connection` section, you can specify the\n`type` of the target database, which must be `redis`, along with\nconnection details such as `host`, `port`, and credentials (`user` and `password`).\nIf you use [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security)/\nor [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) to connect\nto the target database, you must specify the CA certificate (for TLS),\nand the client certificate and private key (for mTLS) in `cacert`, `cert`, and `key`.\nNote that these certificates **must** be references to secrets\nthat you should set as described in [Set secrets](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#set-secrets)\n(it is not possible to include these certificates as plain text in the file)."
    },
    {
      "id": "processors",
      "title": "Processors",
      "role": "content",
      "text": "The `processors` section selects the stream processor and configures its behavior. The default\nis the Flink processor, which is what we recommend. To run a pipeline on the classic processor instead,\nset `type` to `classic`:\n\n[code example]\n\nSee [Differences between the classic and Flink processors](https://redis.io/docs/latest/integrate/redis-data-integration/architecture/classic-vs-flink)\nand [Migrate from the classic processor to the Flink processor](https://redis.io/docs/latest/integrate/redis-data-integration/installation/migration-classic-to-flink)\nfor existing pipelines."
    },
    {
      "id": "tune-classic-processor-performance",
      "title": "Tune Classic processor performance",
      "role": "content",
      "text": "The Classic processor uses the top-level batch, queue, initial-sync, and stream\npolling properties. Larger batches can improve throughput but use more memory and\ncan increase latency while RDI waits for a batch to fill.\n\n[code example]"
    },
    {
      "id": "tune-flink-processor-performance",
      "title": "Tune Flink processor performance",
      "role": "content",
      "text": "The Flink processor uses `processors.advanced` for batch behavior, parallelism,\nand memory. Don't use Classic queue and initial-sync properties to tune Flink.\n\n[code example]\n\nFor Kubernetes installations, the number of available task slots is the number\nof TaskManager replicas multiplied by `taskmanager.numberOfTaskSlots`. When you\nomit `parallelism.default`, Flink uses the available task slots. Adding task slots\ncan increase initial snapshot throughput. Size `taskmanager.memory.process.size`\nfor the work done by each TaskManager, especially when jobs use transformations.\n\nThe `advanced.source.batch.size`, `advanced.source.batch.timeout.ms`, and\n`advanced.target.batch.size` properties override their top-level aliases when\nboth forms are present. Change other Flink settings only when instructed by Redis\nsupport. See the [configuration file reference](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#processorsadvanced)\nfor all Flink processor properties."
    },
    {
      "id": "choose-the-redis-data-type",
      "title": "Choose the Redis data type",
      "role": "content",
      "text": "Set `target_data_type` to `hash` (the default) or `json`. The `json` option\nrequires JSON support in the target database. A job file can override this\nsetting for its output.\n\n[code example]\n\nSee [Job files](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples)\nfor the data types available to job outputs."
    },
    {
      "id": "confirm-writes-reached-a-replica",
      "title": "Confirm writes reached a replica",
      "role": "content",
      "text": "Use these properties only when target database replication is enabled and a\nhealthy replica is available:\n\n[code example]\n\nFor the Flink processor, the corresponding properties under\n`processors.advanced.target` take priority over these top-level properties.\n\nSee also the\n[RDI configuration file reference](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#processors)\nfor full details of the other available properties."
    },
    {
      "id": "extended-configuration-example",
      "title": "Extended configuration example",
      "role": "content",
      "text": "The example below is the configuration that\n[`redis-di scaffold`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/cli/redis-di-scaffold)\ngenerates for a MySQL source named `mysql`, with every property documented inline. Uncomment what\nyou need and delete the rest. See the\n[configuration file reference](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference)\nfor every supported property.\n\n[code example]"
    }
  ],
  "examples": [
    {
      "id": "start-with-a-minimal-file-ex0",
      "language": "yaml",
      "code": "sources:\n  mysql:\n    type: cdc\n    logging:\n      level: info\n    connection:\n      type: mysql\n      host: <DB_HOST>\n      port: 3306\n      user: ${MYSQL_DB_USERNAME}\n      password: ${MYSQL_DB_PASSWORD}\n\ntargets:\n  target:\n    connection:\n      type: redis\n      host: <TARGET_DB_HOST>\n      port: 6379\n      password: ${TARGET_DB_PASSWORD}\n\nprocessors:\n  target_data_type: hash",
      "section_id": "start-with-a-minimal-file"
    },
    {
      "id": "build-the-file-with-an-ai-assistant-ex0",
      "language": "text",
      "code": "Help me create a valid Redis Data Integration (RDI) config.yaml file.\n\nUse only these pages as sources for configuration properties and behavior:\n- https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config/\n- https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference/\n- https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/prepare-dbs/\n\nDo not invent property names. If a requested property is not documented, tell me.\nUse ${NAME} secret references for credentials and certificates. Do not include secret\nvalues in the file. Configure one target Redis database named `target`. Always use the\nFlink processor by setting `processors.type` to `flink`.\n\nAsk me for the following information one question at a time:\n1. Source database type, host, and port.\n2. Databases or schemas to capture.\n3. Tables and columns to capture, including keys for tables without a primary key\n   or unique constraint.\n4. Whether the initial snapshot needs a row filter.\n5. Target Redis host and port, and whether the connection uses TLS or mTLS.\n6. Redis hash or JSON output.\n\nAfter I answer, generate config.yaml. Then list the required secrets and link me to\nthe documented commands to set the secrets and deploy the pipeline.",
      "section_id": "build-the-file-with-an-ai-assistant"
    },
    {
      "id": "choose-a-snapshot-mode-ex0",
      "language": "yaml",
      "code": "sources:\n  mysql:\n    type: cdc\n    connection:\n      type: mysql\n      host: <your-source-host>\n      port: 3306\n      user: ${MYSQL_DB_USERNAME}\n      password: ${MYSQL_DB_PASSWORD}\n    advanced:\n      source:\n        snapshot.mode: initial\n\ntargets:\n  target:\n    connection:\n      type: redis\n      host: <your-target-host>\n      port: 6379\n      password: ${TARGET_DB_PASSWORD}",
      "section_id": "choose-a-snapshot-mode"
    },
    {
      "id": "choose-a-snapshot-mode-ex1",
      "language": "mermaid",
      "code": "%%{init: {\"sequence\": {\"actorMargin\": 15, \"diagramMarginX\": 5, \"width\": 90, \"mirrorActors\": false}}}%%\nsequenceDiagram\n    participant Source as Source<br/>database\n    participant RDI as RDI<br/>collector\n    participant Redis as Target<br/>Redis\n\n    Source->>RDI: Capture<br/>row 42\n    RDI->>Redis: Write<br/>row 42\n    Note over RDI: Collection stops\n    Note over Source: Delete row 42\n    Note over Source,RDI: Required log<br/>history expires\n    RDI->>Source: Restart with<br/>when_needed\n    Note over Source,RDI: Saved position unavailable<br/>New snapshot\n    Source->>RDI: Current rows<br/>(without row 42)\n    RDI->>Redis: Write snapshot<br/>rows\n    Note over Redis: Row 42 is stale<br/>Delete missed",
      "section_id": "choose-a-snapshot-mode"
    },
    {
      "id": "processors-ex0",
      "language": "yaml",
      "code": "processors:\n  type: classic",
      "section_id": "processors"
    },
    {
      "id": "tune-classic-processor-performance-ex0",
      "language": "yaml",
      "code": "processors:\n  type: classic\n  read_batch_size: 2000\n  read_batch_timeout_ms: 100\n  write_batch_size: 200\n  enable_async_processing: true\n  batch_queue_size: 3\n  ack_queue_size: 10\n  initial_sync_processes: 4\n  idle_sleep_time_ms: 200\n  idle_streams_check_interval_ms: 1000\n  busy_streams_check_interval_ms: 5000",
      "section_id": "tune-classic-processor-performance"
    },
    {
      "id": "tune-flink-processor-performance-ex0",
      "language": "yaml",
      "code": "processors:\n  type: flink\n  advanced:\n    source:\n      batch.size: 2000\n      batch.timeout.ms: 100\n      discovery.interval.ms: 1000\n    target:\n      batch.size: 200\n      flush.interval.ms: 100\n    flink:\n      taskmanager.numberOfTaskSlots: 1\n      taskmanager.memory.process.size: 2048m\n    resources:\n      taskManager:\n        replicas: 2",
      "section_id": "tune-flink-processor-performance"
    },
    {
      "id": "choose-the-redis-data-type-ex0",
      "language": "yaml",
      "code": "processors:\n  target_data_type: hash",
      "section_id": "choose-the-redis-data-type"
    },
    {
      "id": "confirm-writes-reached-a-replica-ex0",
      "language": "yaml",
      "code": "processors:\n  wait_enabled: true\n  wait_timeout: 1000\n  retry_on_replica_failure: true",
      "section_id": "confirm-writes-reached-a-replica"
    },
    {
      "id": "extended-configuration-example-ex0",
      "language": "yaml",
      "code": "# Configuration of the RDI sources, targets, and processor. Commented-out properties are optional, and where a default\n# exists the value shown is that default.\n# For an introduction to the pipeline configuration, see\n# https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config/\n# For a reference of every configuration property, see\n# https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference/\n\n# Source databases that are used to capture changes from. Each key is a unique source name.\nsources:\n  mysql:\n    # Type of the source collector. Use `cdc` for Debezium, `flink` for Spanner, or `riotx` for Snowflake.\n    type: cdc\n\n    # Log verbosity of the source collector, one of `trace`, `debug`, `info`, `warn`, or `error`.\n    logging:\n      level: info\n\n    # Connection to the source database.\n    connection:\n      type: mysql\n      # Hostname or IP address of the source database server, for example `localhost`.\n      host: <DB_HOST>\n      # Port that the source database server listens on.\n      port: 3306\n      # User and password are resolved from the source database secret.\n      user: ${MYSQL_DB_USERNAME}\n      password: ${MYSQL_DB_PASSWORD}\n\n    # Databases to capture from the source database. When omitted, all databases are captured.\n    # databases:\n      # - <DATABASE_NAME>\n\n    # Tables to capture from the source database, keyed by table name. A table with no properties, or with an empty\n    # mapping, captures all of its columns. If only one database is listed above, table names can omit the\n    # database prefix.\n    # tables:\n      # <DATABASE_NAME>.<TABLE_NAME1>: {}\n      # <DATABASE_NAME>.<TABLE_NAME2>:\n        # Columns to capture. When omitted, all columns are captured.\n        # columns:\n          # - <COLUMN_NAME>\n        # Columns that form a unique identifier. Only needed when the table has no primary key or unique constraint.\n        # keys:\n          # - <COLUMN_NAME>\n\n    # Advanced properties that override the collector defaults. Only needed for non-standard tuning.\n    # advanced:\n      # Properties of the RDI Collector stream writer. See the full list in\n      # https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream\n      # sink:\n        # Number of records that the sink writes to Redis in a single batch. Raise it, together with the source\n        # `max.batch.size` and `max.queue.size`, for higher snapshot throughput.\n        # redis.batch.size: 2048\n\n        # Back pressure on the RDI streams. The sink stops writing once the used memory of the RDI database reaches\n        # `redis.memory.threshold.percentage` of its `maxmemory`, or of `redis.memory.limit.mb` when that is set.\n        # redis.memory.limit.mb: 0\n        # redis.memory.threshold.percentage: 85\n\n        # Whether every write to the RDI streams is verified to have reached a replica shard, which prevents losing\n        # captured changes when the primary shard fails over. It costs write latency, and it has no effect unless the\n        # RDI database is replicated. `redis.wait.retry.enabled` keeps retrying when the verification times out.\n        # redis.wait.enabled: false\n        # redis.wait.timeout.ms: 1000\n        # redis.wait.retry.enabled: false\n        # redis.wait.retry.delay.ms: 1000\n\n      # Properties of the source database connection and of Debezium. See the full list for your source database in\n      # https://debezium.io/documentation/reference/stable/connectors/\n      # source:\n        # When and whether the collector takes an initial snapshot of the source database.\n        # snapshot.mode: initial\n\n        # Performance tuning of the Debezium collector. Rows read from the source database are buffered in a queue of\n        # `max.queue.size` records, serialized by `record.processing.threads` worker threads, and passed on in batches\n        # of `max.batch.size` records, which the sink then writes to Redis in batches of `sink.redis.batch.size`.\n        # Lower this to 2 or 1 when fewer CPUs are available to the collector.\n        # record.processing.threads: 4\n\n        # Interval between two polls of the source database for new changes in milliseconds. Lower it to 100 or below\n        # for lower CDC latency.\n        # poll.interval.ms: 500\n\n        # Raise both sizes, together with `sink.redis.batch.size`, for higher snapshot throughput, and keep the queue\n        # about four times the batch. A larger queue needs a larger heap, so raise `advanced.resources.memory` and set\n        # `-Xmx` in `advanced.java_options` accordingly.\n        # max.batch.size: 2048\n        # max.queue.size: 8192\n\n        # Restrict the snapshot of a table to the rows that the given statement selects. This affects the snapshot only,\n        # for example to skip orders that are still pending.\n        # snapshot.select.statement.overrides: <DATABASE_NAME>.<TABLE_NAME>\n        # snapshot.select.statement.overrides.<DATABASE_NAME>.<TABLE_NAME>: <SELECT_STATEMENT>\n\n        # Let the client retrieve the public key of the server for the TLS handshake. Uncomment when the source\n        # database user authenticates with `caching_sha2_password` over a connection that is not encrypted.\n        # database.allowPublicKeyRetrieval: true\n\n      # Properties of the Quarkus runtime that hosts Debezium Server. See the full list in\n      # https://quarkus.io/guides/all-config\n      # quarkus:\n        # banner.enabled: \"false\"\n\n# Target Redis databases that the processed records are written to. RDI currently only supports a single target that\n# must be named `target`.\ntargets:\n  target:\n    # Connection to the target Redis database.\n    connection:\n      type: redis\n      # Hostname or IP address of the target Redis database, for example `localhost`.\n      host: <TARGET_DB_HOST>\n      # Port that the target Redis database listens on.\n      port: 6379\n      # User of the target Redis database, uncomment when not using the default user.\n      # user: ${TARGET_DB_USERNAME}\n      # Password of the target Redis database.\n      password: ${TARGET_DB_PASSWORD}\n      # TLS configuration, uncomment to connect securely. The key and the certificate must be set together, and the key\n      # password only when the key is protected by one.\n      # cacert: ${TARGET_DB_CACERT}\n      # cert: ${TARGET_DB_CERT}\n      # key: ${TARGET_DB_KEY}\n      # key_password: ${TARGET_DB_KEY_PASSWORD}\n\n# Settings that control how the processor writes the captured records to the targets.\nprocessors:\n  # Processor implementation to run, one of `classic` or `flink`.\n  # The default is `flink`, which is the recommended processor.\n  # See https://redis.io/docs/latest/integrate/redis-data-integration/faq/#which-processor-should-i-use\n  # type: flink\n\n  # Maximum number of records read from the source streams in a single batch.\n  # read_batch_size: 2000\n\n  # Maximum time in milliseconds to wait for a read batch to fill before processing it.\n  # read_batch_timeout_ms: 100\n\n  # Maximum number of records written to a target database in a single batch. Must not exceed read_batch_size.\n  # write_batch_size: 200\n\n  # Whether batches are processed asynchronously, which improves throughput and reduces latency. Applies to the classic\n  # processor only.\n  # enable_async_processing: true\n\n  # Maximum number of batches queued for processing. Applies to the classic processor only.\n  # batch_queue_size: 3\n\n  # Maximum number of batches queued for asynchronous acknowledgement. Applies to the classic processor only.\n  # ack_queue_size: 10\n\n  # Whether incoming records are deduplicated. Applies to the classic processor only.\n  # dedup: false\n\n  # Maximum number of entries kept in the deduplication set. Applies to the classic processor only.\n  # dedup_max_size: 1024\n\n  # How failed records are handled. `ignore` drops them, and `dlq` writes them to the dead letter queue.\n  # error_handling: dlq\n\n  # Maximum number of messages stored per dead letter queue stream.\n  # dlq_max_messages: 1000\n\n  # Data type that the records are stored as. `hash` writes a Redis hash, and `json` writes a RedisJSON document, which\n  # requires the RedisJSON module in the target database.\n  # target_data_type: hash\n\n  # How existing JSON documents are updated. `replace` overwrites the whole document, and `merge` merges the incoming\n  # fields into it.\n  # json_update_strategy: replace\n\n  # Whether JSON documents are merged with the native JSON.MERGE command rather than with Lua scripts. Applies to the\n  # classic processor only.\n  # use_native_json_merge: true\n\n  # Number of parallel processes that perform the initial synchronization. Applies to the classic processor only.\n  # initial_sync_processes: 4\n\n  # Time in milliseconds to sleep between batches while idle. Applies to the classic processor only.\n  # idle_sleep_time_ms: 200\n\n  # Time in milliseconds between two checks for new streams while idle. Applies to the classic processor only.\n  # idle_streams_check_interval_ms: 1000\n\n  # Time in milliseconds between two checks for new streams while busy. Applies to the classic processor only.\n  # busy_streams_check_interval_ms: 5000\n\n  # Maximum number of attempts for a failed write to a target database before giving up.\n  # retry_max_attempts: 5\n\n  # Initial delay in milliseconds before the first retry of a failed write.\n  # retry_initial_delay_ms: 1000\n\n  # Maximum delay in milliseconds between two retries of a failed write.\n  # retry_max_delay_ms: 10000\n\n  # Whether every write is verified to have reached the replica shards of the target database. Enable this only when the\n  # target database is replicated and a healthy replica is available.\n  # wait_enabled: false\n\n  # Maximum time in milliseconds to wait for the replica write verification.\n  # wait_timeout: 1000\n\n  # Whether a write is retried until the replica verification succeeds, rather than given up on after the first failure.\n  # retry_on_replica_failure: true\n\n  # Logging settings of the processor. Applies to the Flink processor only.\n  # logging:\n    # Log verbosity of the processor, one of `trace`, `debug`, `info`, `warn`, or `error`.\n    # level: info",
      "section_id": "extended-configuration-example"
    }
  ]
}