{ "schema_version": 2, "id": "integrate/redis-data-integration/reference/config-yaml-reference", "title": "Redis Data Integration configuration file", "url": "https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference/", "summary": "Redis Data Integration configuration file reference", "aliases": [ "/integrate/redis-data-integration/ingest/reference/config-yaml-reference/" ], "tags": [ "redis-di" ], "last_updated": "2026-10-09T12:52:02+03:00", "page_type": "content", "content_hash": "1bdad068ae1eefb24121a72fef1babc78aa371c2a727187827955a95f0a60048", "sections": [ { "id": "overview", "title": "Overview", "role": "overview", "text": "Configuration file for Redis Data Integration (RDI) source collectors and target connections.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|[**sources**](#sources)
(Source collectors)|`object`|Source collectors that capture changes from upstream databases. Each key is a unique source identifier; the value configures one collector.
||\n|[**targets**](#targets)
(Target connections)|`object`|Target Redis databases where processed records are written. Each key is a target identifier; the value configures the connection.
||\n|[**processors**](#processors)
(Data processing configuration)|`object`, `null`|Settings that control how data is processed, including batch sizes, error handling, and performance tuning.
||\n|[**secret\\-providers**](#secret-providers)
(Secret providers)|`object`|External secret providers used to resolve `${...}` references in the configuration.
||\n|[**metadata**](#metadata)
(Pipeline metadata)|`object`|Optional metadata describing this pipeline, such as a display name and description.
||\n\n**Additional Properties:** not allowed \n" }, { "id": "sources-source-collectors", "title": "sources: Source collectors", "role": "content", "text": "Source collectors that capture changes from upstream databases. Each key is a unique source identifier; the value configures one collector.\n\n\n**Properties** (key: `.*`)\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|[**connection**](#sourcesconnection)
(Source database connection)|`object`|Connection configuration for a non-Redis source database. The exact set of properties depends on the database type.
|no|\n|**name**
(Source name)|`string`|Human-readable name for the source collector. Maximum 100 characters.
Maximal Length: `100`
|no|\n|**type**
(Collector type)|`string`|Type of the source collector. Use `cdc` (default) for change data capture using [Debezium](https://debezium.io/). Use `flink` for Spanner change streams using the Apache Flink-based collector. Use `riotx` for Snowflake CDC using [RIOT-X](https://redis.github.io/riotx/).
Default: `\"cdc\"`
Enum: `\"cdc\"`, `\"flink\"`, `\"riotx\"`
|yes|\n|**active**
(Collector enabled)|`boolean`|When `true`, the collector runs; when `false`, the collector is disabled and produces no events.
Default: `true`
|no|\n|[**logging**](#sourceslogging)
(Logging configuration)|`object`|Logging settings for this source collector.
|no|\n|[**tables**](#sourcestables)
(Tables to capture)|`object`|Tables to capture from the source database, keyed by table name. The value configures column selection and key handling for that table.
|no|\n|[**schemas**](#sourcesschemas)
(Schema names)|`string[]`|Schema names to capture from the source database. Maps to the underlying connector's `schema.include.list`.
|no|\n|[**databases**](#sourcesdatabases)
(Database names)|`string[]`|Database names to capture from the source database. Maps to the underlying connector's `database.include.list`. Applies only to MySQL, MariaDB, and MongoDB connections.
|no|\n|[**advanced**](#sourcesadvanced)
(Advanced configuration)|`object`|Advanced configuration that overrides the underlying engine's defaults. Only required for non-standard tuning.
|no|\n\n\n**Property Name Pattern:** `^[a-z]([a-z0-9-]*[a-z0-9])?$` \n" }, { "id": "sourcesconnection-source-database-connection", "title": "sources\\.connection: Source database connection", "role": "content", "text": "Connection configuration for a non-Redis source database. The exact set of properties depends on the database type.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|[**SQL database**](#sourcesconnectionsqldatabase)
(SQL database)|`object`|Connection configuration for a supported SQL database.
||\n|[**MongoDB**](#sourcesconnectionmongodb)|`object`|Connection configuration for a MongoDB database.
|yes|\n|[**Spanner**](#sourcesconnectionspanner)|`object`|Connection configuration for a Google Cloud Spanner database.
|yes|\n|[**Snowflake**](#sourcesconnectionsnowflake)|`object`|Connection configuration for a Snowflake database.
|yes|\n\n**Example**\n\n[code example]\n\n\n#### sources\\.connection\\.SQL database: SQL database\n\nConnection configuration for a supported SQL database.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**type**
(Database type)|`string`|SQL database engine.
Enum: `\"mariadb\"`, `\"mysql\"`, `\"oracle\"`, `\"postgresql\"`, `\"sqlserver\"`
||\n|**host**
(Database host)|`string`|Hostname or IP address of the SQL database server.
||\n|**port**
(Database port)|`integer`|Network port on which the SQL database server is listening.
Minimum: `1`
Maximum: `65535`
||\n|**database**
(Database name)|`string`|Name of the database to connect to.
Minimal Length: `1`
||\n|**user**
(Database user)|`string`|Username for authentication to the SQL database.
||\n|**password**
(Database password)|`string`|Password for authentication to the SQL database.
||\n\n**Additional Properties:** not allowed \n**Example**\n\n[code example]\n\n**Example**\n\n[code example]\n\n\n#### sources\\.connection\\.MongoDB: MongoDB\n\nConnection configuration for a MongoDB database.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**type**
(Database type)|`string`|Database type identifier. Always `mongodb` for this connection.
Constant Value: `\"mongodb\"`
|yes|\n|**connection\\_string**|`string`|MongoDB connection URI including host, port, and any connection options.
|yes|\n|**user**
(MongoDB user)|`string`|Username for authentication to MongoDB.
|no|\n|**password**
(MongoDB password)|`string`|Password for authentication to MongoDB.
|no|\n|**database**
(MongoDB databases)|`string`|Comma-separated list of MongoDB databases to monitor.
|no|\n\n**Additional Properties:** not allowed \n**Example**\n\n[code example]\n\n\n#### sources\\.connection\\.Spanner: Spanner\n\nConnection configuration for a Google Cloud Spanner database.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**type**
(Database type)|`string`|Database type identifier. Always `spanner` for this connection.
Constant Value: `\"spanner\"`
|yes|\n|**project\\_id**
(Spanner project ID)|`string`|Google Cloud project ID that hosts the Spanner instance.
|yes|\n|**instance\\_id**
(Spanner instance ID)|`string`|Spanner instance identifier within the project.
|yes|\n|**database\\_id**
(Spanner database ID)|`string`|Spanner database identifier within the instance.
|yes|\n|**emulator\\_host**
(Spanner emulator host)|`string`|Host and port of the Spanner emulator. Used for local development; leave unset against real Spanner.
|no|\n|**use\\_credentials\\_file**|`boolean`|When `true`, RDI authenticates using a service account credentials file; when `false`, it uses application default credentials.
Default: `false`
|no|\n|[**change\\_streams**](#sourcesconnectionspannerchange_streams)
(Change streams configuration)|`object`|Spanner change streams to capture, keyed by change stream name.
|yes|\n\n**Additional Properties:** not allowed \n**Example**\n\n[code example]\n\n\n##### sources\\.connection\\.Spanner\\.change\\_streams: Change streams configuration\n\nSpanner change streams to capture, keyed by change stream name.\n\n\n**Additional Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|[**Additional Properties**](#sourcesconnectionspannerchange_streamsadditionalproperties)|`object`, `null`|||\n\n**Minimal Properties:** 1 \n\n###### sources\\.connection\\.Spanner\\.change\\_streams\\.additionalProperties: object,null\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**retention\\_period\\_hours**
(Change stream retention period hours)|`integer`, `string`|Retention period for the change stream, in hours.
Pattern: `^\\${.*}$`
Minimum: `1`
||\n\n**Additional Properties:** not allowed \n\n#### sources\\.connection\\.Snowflake: Snowflake\n\nConnection configuration for a Snowflake database.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**type**
(Database type)|`string`|Database type identifier. Always `snowflake` for this connection.
Constant Value: `\"snowflake\"`
|yes|\n|**url**
(JDBC URL)|`string`|Snowflake JDBC connection URL, for example `jdbc:snowflake://account.snowflakecomputing.com/`.
|yes|\n|**user**
(Snowflake user)|`string`|Username for authentication to Snowflake.
|yes|\n|**password**
(Snowflake password)|`string`|Password for authentication to Snowflake. For key-pair authentication, omit this field and provide the private key via the `-db-ssl` secret (`client.key` field).
|no|\n|**database**
(Snowflake database)|`string`|Name of the Snowflake database to connect to.
|yes|\n|**warehouse**
(Snowflake warehouse)|`string`|Name of the Snowflake warehouse used for compute.
|yes|\n|**role**
(Snowflake role)|`string`|Snowflake role used for the connection.
|no|\n|**cdcDatabase**
(CDC database)|`string`|Database hosting the CDC streams. Defaults to the main `database` if not set.
|no|\n|**cdcSchema**
(CDC schema)|`string`|Schema hosting the CDC streams. Defaults to the main schema if not set.
|no|\n\n**Additional Properties:** not allowed \n**Example**\n\n[code example]\n\n" }, { "id": "sourceslogging-logging-configuration", "title": "sources\\.logging: Logging configuration", "role": "content", "text": "Logging settings for this source collector.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**level**
(Logging level)|`string`|Log verbosity for the source collector.
Default: `\"info\"`
Enum: `\"trace\"`, `\"debug\"`, `\"info\"`, `\"warn\"`, `\"error\"`
||\n\n**Additional Properties:** not allowed \n**Example**\n\n[code example]\n\n" }, { "id": "sourcestables-tables-to-capture", "title": "sources\\.tables: Tables to capture", "role": "content", "text": "Tables to capture from the source database, keyed by table name. The value configures column selection and key handling for that table.\n\n\n**Additional Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|[**Additional Properties**](#sourcestablesadditionalproperties)|`object`, `null`|||\n\n**Minimal Properties:** 1 \n\n#### sources\\.tables\\.additionalProperties: object,null\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**snapshot\\_sql**|`string`|Custom SQL statement used during the initial snapshot, giving fine-grained control over the data captured.
||\n|[**columns**](#sourcestablesadditionalpropertiescolumns)
(Columns to capture)|`string[]`|List of specific columns to capture for changes. If not specified, all columns will be captured. For RIOTX Snowflake sources, this is rendered as per-table `--table-columns` projection. Note: This property cannot be used for MongoDB connections
||\n|[**exclude\\_columns**](#sourcestablesadditionalpropertiesexclude_columns)
(Columns to exclude)|`string[]`|Specific columns to exclude from capture. When omitted, no columns are excluded. Only supported for MongoDB connections.
||\n|[**keys**](#sourcestablesadditionalpropertieskeys)
(Message keys)|`string[]`|Optional list of columns to use as a composite unique identifier. For RIOTX Snowflake sources, this is rendered as per-table `--table-keys`. Only required when the table lacks a primary key or unique constraint. Must form a unique combination of fields
||\n\n**Additional Properties:** not allowed \n\n##### sources\\.tables\\.additionalProperties\\.columns\\[\\]: Columns to capture\n\nList of specific columns to capture for changes. If not specified, all columns will be captured. For RIOTX Snowflake sources, this is rendered as per-table `--table-columns` projection. Note: This property cannot be used for MongoDB connections\n\n\n\n##### sources\\.tables\\.additionalProperties\\.exclude\\_columns\\[\\]: Columns to exclude\n\nSpecific columns to exclude from capture. When omitted, no columns are excluded. Only supported for MongoDB connections.\n\n\n\n##### sources\\.tables\\.additionalProperties\\.keys\\[\\]: Message keys\n\nOptional list of columns to use as a composite unique identifier. For RIOTX Snowflake sources, this is rendered as per-table `--table-keys`. Only required when the table lacks a primary key or unique constraint. Must form a unique combination of fields\n\n\n" }, { "id": "sourcesschemas-schema-names", "title": "sources\\.schemas\\[\\]: Schema names", "role": "content", "text": "Schema names to capture from the source database. Maps to the underlying connector's `schema.include.list`.\n\n\n" }, { "id": "sourcesdatabases-database-names", "title": "sources\\.databases\\[\\]: Database names", "role": "content", "text": "Database names to capture from the source database. Maps to the underlying connector's `database.include.list`. Applies only to MySQL, MariaDB, and MongoDB connections.\n\n\n" }, { "id": "sourcesadvanced-advanced-configuration", "title": "sources\\.advanced: Advanced configuration", "role": "content", "text": "Advanced configuration that overrides the underlying engine's defaults. Only required for non-standard tuning.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|[**sink**](#sourcesadvancedsink)
(RDI Collector stream writer configuration)|`object`|Advanced configuration properties for the RDI Collector stream writer connection and behaviour. **Applies to the `cdc` and `flink` collector types.**
||\n|[**source**](#sourcesadvancedsource)
(Advanced source settings)|`object`|Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**
||\n|[**quarkus**](#sourcesadvancedquarkus)
(Quarkus runtime settings)|`object`|Advanced configuration properties for the Quarkus runtime that hosts Debezium Server. **Only applies to the `cdc` collector type.** See the [Debezium Server documentation](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) for runtime configuration options. When using a property from that page, omit the `quarkus.` prefix.
||\n|[**flink**](#sourcesadvancedflink)
(Advanced Flink settings)|`object`|Advanced configuration properties forwarded to the Flink runtime that hosts the collector. Any property listed in the [Flink configuration documentation](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/deployment/config/) can be set here and will override the RDI default. **Only applies to the `flink` collector type.**
||\n|[**resources**](#sourcesadvancedresources)
(Collector resource settings)|`object`|Compute resources allocated to the collector. **Only applies to the `cdc` collector type.**
||\n|[**riotx**](#sourcesadvancedriotx)
(Advanced RIOT\\-X settings)|`object`|Advanced configuration properties for the RIOT-X Snowflake collector. **Only applies to the `riotx` collector type.**
||\n|**java\\_options**
(Advanced Java options)|`string`|These Java options will be passed to the command line command when launching the source collector. **Only applies to the `cdc` collector type.**
||\n|[**diagnostics**](#sourcesadvanceddiagnostics)
(Collector diagnostics)|`object`|Opt-in JVM diagnostics for the source collector. Each artifact is written to the `/opt/rdi/diagnostics` directory on the host and included in the `dump-support-package` bundle. **Only applies to the `cdc` collector type on VM installations, and is ignored otherwise.**
||\n\n**Additional Properties:** not allowed \n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n#### sources\\.advanced\\.sink: RDI Collector stream writer configuration\n\nAdvanced configuration properties for the RDI Collector stream writer connection and behaviour. **Applies to the `cdc` and `flink` collector types.**

For the `cdc` collector type, see the full list of properties at [Debezium Server — Redis Stream sink](https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream). When using a property from that page, omit the `debezium.sink.` prefix.

**The properties listed below only apply to the `flink` collector type.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**redis\\.batch\\.size**
(Sink batch size)|`integer`|Maximum number of records the collector sink writes to Redis in a single batch.
Default: `1000`
Minimum: `1`
||\n|**redis\\.flush\\.interval\\.ms**
(Sink flush interval)|`integer`|Maximum time in milliseconds the collector sink waits to fill a batch before flushing it to Redis.
Default: `100`
Minimum: `1`
||\n|**redis\\.connection\\.timeout\\.ms**
(Sink connection timeout)|`integer`|Connection timeout in milliseconds for the target Redis client used by the collector sink.
Default: `2000`
Minimum: `1`
||\n|**redis\\.socket\\.timeout\\.ms**
(Sink socket timeout)|`integer`|Socket read/write timeout in milliseconds for the target Redis client used by the collector sink.
Default: `2000`
Minimum: `1`
||\n|**redis\\.retry\\.max\\.attempts**
(Sink retry max attempts)|`integer`|Maximum number of retry attempts for failed Redis operations.
Default: `5`
Minimum: `1`
||\n|**redis\\.retry\\.initial\\.delay\\.ms**
(Sink retry initial delay)|`integer`|Initial delay in milliseconds before the first retry of a failed Redis operation.
Default: `100`
Minimum: `1`
||\n|**redis\\.retry\\.max\\.delay\\.ms**
(Sink retry max delay)|`integer`|Maximum delay in milliseconds between retry attempts for failed Redis operations.
Default: `3000`
Minimum: `1`
||\n|**redis\\.retry\\.backoff\\.multiplier**
(Sink retry backoff multiplier)|`number`|Exponential backoff multiplier between retry attempts for failed Redis operations.
Default: `2`
Minimum: `1`
||\n|**redis\\.oom\\.retry\\.initial\\.delay\\.ms**
(Sink OOM retry initial delay)|`integer`|Initial delay in milliseconds before the first retry after a Redis out-of-memory error.
Default: `1000`
Minimum: `1`
||\n|**redis\\.oom\\.retry\\.max\\.delay\\.ms**
(Sink OOM retry max delay)|`integer`|Maximum delay in milliseconds between retry attempts after a Redis out-of-memory error.
Default: `10000`
Minimum: `1`
||\n|**redis\\.oom\\.retry\\.backoff\\.multiplier**
(Sink OOM retry backoff multiplier)|`number`|Exponential backoff multiplier between retry attempts after a Redis out-of-memory error.
Default: `2`
Minimum: `1`
||\n|**redis\\.wait\\.enabled**
(Sink replica wait enabled)|`boolean`|When `true`, the collector verifies that each write has been replicated to the configured number of Redis replica shards before acknowledging it.
Default: `false`
||\n|**redis\\.wait\\.timeout\\.ms**
(Sink replica wait timeout)|`integer`|Maximum time in milliseconds to wait for replica write acknowledgements.
Default: `1000`
Minimum: `1`
||\n|**redis\\.wait\\.write\\.timeout\\.ms**
(Sink replica wait timeout)|`integer`|Maximum time in milliseconds to wait for replica write acknowledgements.
Default: `1000`
Minimum: `1`
||\n|**redis\\.wait\\.retry\\.enabled**
(Sink replica wait retry enabled)|`boolean`|When `true`, the collector keeps retrying a write until replica acknowledgement succeeds; when `false`, it gives up after the first failure.
Default: `false`
||\n|**redis\\.wait\\.retry\\.delay\\.ms**
(Sink replica wait retry delay)|`integer`|Delay in milliseconds between replica wait retry attempts.
Default: `1000`
Minimum: `1`
||\n\n**Additional Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**Additional Properties**|`string`, `number`, `boolean`|||\n\n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n#### sources\\.advanced\\.source: Advanced source settings\n\nAdvanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**

For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [Db2](https://debezium.io/documentation/reference/stable/connectors/db2.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.

**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair, except for `topic.prefix`, which RDI derives from the source name, and the keystore properties, which RDI configures from a source's TLS secret.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**record\\.processing\\.threads**
(Record processing threads)|`integer`|Controls how many worker threads process captured records before they are written downstream.
Minimum: `1`
||\n|**snapshot\\.mode**
(Snapshot mode)|`string`|Determines when a Debezium collector snapshots selected source data. Available modes and restart behavior depend on the source connector. A reset clears the saved offset without changing this mode or flushing the target. A new snapshot can leave stale target records after missed deletes. See [Choose a snapshot mode](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config#choose-a-snapshot-mode) for the mode comparison.
Default: `\"initial\"`
Enum: `\"always\"`, `\"initial\"`, `\"initial_only\"`, `\"never\"`, `\"no_data\"`, `\"when_needed\"`
||\n|**snapshot\\.max\\.threads**
(Snapshot max threads)|`integer`|Sets the maximum number of threads used while taking the initial snapshot.
Default: `1`
Minimum: `1`
||\n|**poll\\.interval\\.ms**
(Poll interval ms)|`integer`|Defines how often the collector polls the source for new changes.
Default: `500`
Minimum: `1`
||\n|**snapshot\\.fetch\\.size**
(Snapshot fetch size)|`integer`|Defines how many rows are fetched per batch during the initial snapshot.
Minimum: `0`
||\n|**max\\.batch\\.size**
(Max batch size)|`integer`|Caps how many records are processed together in a single batch.
Default: `2048`
Minimum: `1`
||\n|**max\\.queue\\.size**
(Max queue size)|`integer`|Limits how many records can be buffered in memory before processing catches up.
Default: `8192`
Minimum: `1`
||\n|**heartbeat\\.interval\\.ms**
(Heartbeat interval ms)|`integer`|Sets how often heartbeat events are emitted to keep change tracking active. Use 0 to disable them.
Default: `0`
Minimum: `0`
||\n|**heartbeat\\.action\\.query**
(Heartbeat action query)|`string`|SQL query executed on the source whenever a heartbeat is emitted.
||\n|**database\\.trustServerCertificate**
(Trust SQL Server certificate)|`boolean`|Controls whether the RDI collector skips validation of the SQL Server TLS certificate. Enable this only when the server certificate cannot be validated through a trusted certificate authority.
Default: `false`
||\n|**driver\\.applicationIntent**
(SQL Server application intent)|`string`|Declares the SQL Server workload type for the RDI collector. Set to `ReadOnly` when connecting to an Always On read-only replica. CDC must be enabled on the primary SQL Server node.
Default: `\"ReadWrite\"`
Enum: `\"ReadOnly\"`, `\"ReadWrite\"`
||\n|**lob\\.enabled**
(Lob enabled)|`boolean`|Determines whether large object columns are included in change capture.
Default: `false`
||\n|**gtid\\.source\\.includes**
(GTID source includes)|`string`|Restricts MySQL GTID processing to the listed source UUIDs.
||\n|**publication\\.autocreate\\.mode**
(Publication autocreate mode)|`string`|Controls whether and how the PostgreSQL publication is created or updated automatically.
Default: `\"all_tables\"`
Enum: `\"all_tables\"`, `\"filtered\"`, `\"disabled\"`
||\n|**publication\\.name**
(Publication name)|`string`|Sets the PostgreSQL logical replication publication name used by the collector.
Default: `\"dbz_publication\"`
||\n|**slot\\.name**
(Slot name)|`string`|Sets the PostgreSQL replication slot name the collector reads from.
Default: `\"debezium\"`
||\n|**capture\\.mode**
(MongoDB capture mode)|`string`|Determines whether an update event carries the full document and its pre-image. Jobs that set an output key, and the set output types, need a `*_with_pre_image` mode and pre-images on the collections.
Default: `\"change_streams_update_full\"`
Enum: `\"change_streams\"`, `\"change_streams_update_full\"`, `\"change_streams_update_full_with_pre_image\"`, `\"change_streams_with_pre_image\"`
||\n|**capture\\.mode\\.full\\.update\\.type**
(MongoDB full update type)|`string`|Determines how the collector fetches the full document for an update event. `lookup` queries the source per event and can yield a null `after`; `post_image` reads the post-image and needs MongoDB 6.0.
Default: `\"lookup\"`
Enum: `\"lookup\"`, `\"post_image\"`
||\n|**cursor\\.oversize\\.handling\\.mode**
(MongoDB oversize event handling)|`string`|Determines what happens to a change event above the MongoDB 16 MB limit: `fail` stops the collector, `skip` drops it, `split` reassembles it. The limit covers the document and its pre-image.
Default: `\"fail\"`
Enum: `\"fail\"`, `\"skip\"`, `\"split\"`
||\n|**cursor\\.oversize\\.skip\\.threshold**
(MongoDB oversize skip threshold)|`integer`|Maximum size in bytes of the document, and of its pre-image, for which change events are processed. Only applies when `cursor.oversize.handling.mode` is `skip`, which requires a value above 0.
Default: `0`
Minimum: `0`
||\n|**mongodb\\.authsource**
(MongoDB authentication database)|`string`|Database that holds the credentials the collector authenticates with. Only applies to the connection's `user` and `password`; an `authSource` in `connection_string` takes precedence.
Default: `\"admin\"`
||\n|**mongodb\\.ssl\\.invalid\\.hostname\\.allowed**
(Allow invalid MongoDB hostname)|`boolean`|Controls whether the RDI collector skips hostname verification of the MongoDB TLS certificate. Enable this only when the certificate cannot cover the hostnames the replica set advertises.
Default: `false`
||\n|**mongodb\\.connect\\.timeout\\.ms**
(MongoDB connect timeout)|`integer`|Time in milliseconds the driver waits before it aborts a new connection attempt. The `connectTimeoutMS` option of `connection_string` takes precedence.
Default: `10000`
Minimum: `0`
||\n|**mongodb\\.socket\\.timeout\\.ms**
(MongoDB socket timeout)|`integer`|Time in milliseconds a send or receive on the socket may take before it times out, where 0 disables the timeout. The `socketTimeoutMS` option of `connection_string` takes precedence.
Default: `0`
Minimum: `0`
||\n|**mongodb\\.server\\.selection\\.timeout\\.ms**
(MongoDB server selection timeout)|`integer`|Time in milliseconds the driver waits to select a server before it fails with an error. The `serverSelectionTimeoutMS` option of `connection_string` takes precedence.
Default: `30000`
Minimum: `0`
||\n|**mongodb\\.heartbeat\\.frequency\\.ms**
(MongoDB heartbeat frequency)|`integer`|Interval in milliseconds at which the cluster monitor reaches each MongoDB server. The `heartbeatFrequencyMS` option of `connection_string` takes precedence.
Default: `10000`
Minimum: `0`
||\n|**spanner\\.version\\.retention\\.period\\.hours**
(Spanner version retention period)|`integer`|Retention period in hours for Spanner change stream versions. Determines how far back the collector can resume after an outage.
Default: `1`
Minimum: `1`
||\n|**spanner\\.fetch\\.timeout\\.ms**
(Spanner fetch timeout)|`integer`|Timeout in milliseconds for a single change stream fetch request to Spanner.
Default: `500`
Minimum: `1`
||\n|**spanner\\.fetch\\.heartbeat\\.ms**
(Spanner fetch heartbeat interval)|`integer`|Interval in milliseconds at which Spanner sends heartbeat records when no data changes are available.
Default: `100`
Minimum: `1`
||\n|**spanner\\.max\\.rows\\.per\\.partition**
(Spanner max rows per partition)|`integer`|Maximum number of rows the collector reads from a single Spanner change stream partition before yielding.
Default: `10000`
Minimum: `1`
||\n|**spanner\\.dialect**
(Spanner SQL dialect)|`string`|SQL dialect of the Spanner database. Use `GOOGLESQL` for Google Standard SQL or `POSTGRESQL` for the PostgreSQL interface.
Default: `\"GOOGLESQL\"`
Enum: `\"GOOGLESQL\"`, `\"POSTGRESQL\"`
||\n\n**Additional Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**Additional Properties**|`string`, `number`, `boolean`|||\n\n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n#### sources\\.advanced\\.quarkus: Quarkus runtime settings\n\nAdvanced configuration properties for the Quarkus runtime that hosts Debezium Server. **Only applies to the `cdc` collector type.** See the [Debezium Server documentation](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) for runtime configuration options. When using a property from that page, omit the `quarkus.` prefix.\n\n\n**Additional Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**Additional Properties**|`string`, `number`, `boolean`|||\n\n**Minimal Properties:** 1 \n\n#### sources\\.advanced\\.flink: Advanced Flink settings\n\nAdvanced configuration properties forwarded to the Flink runtime that hosts the collector. Any property listed in the [Flink configuration documentation](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/deployment/config/) can be set here and will override the RDI default. **Only applies to the `flink` collector type.**

The properties listed below are the ones most likely to require adjustment. **Changing any other Flink property is not recommended unless instructed by Redis support.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**parallelism\\.default**
(Default parallelism)|`integer`|Default parallelism for Flink jobs and operators. When unset, Flink uses the number of available task slots across all task managers (`taskManager.replicas × taskmanager.numberOfTaskSlots`). See [parallel execution](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/dev/datastream/execution/parallel/).
Minimum: `1`
||\n|**taskmanager\\.numberOfTaskSlots**
(Task slots per task manager)|`integer`|Number of parallel task slots per task manager pod. Each slot can run one parallel pipeline instance, so this caps the parallelism a single task manager can absorb. See [task slots and resources](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/concepts/flink-architecture/#task-slots-and-resources).
Default: `1`
Minimum: `1`
||\n|**taskmanager\\.memory\\.process\\.size**
(Task manager process memory)|`string`|Total memory budget for each task manager JVM process, expressed with a unit suffix such as `2048m` or `4g`. See [task manager memory configuration](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/deployment/memory/mem_setup_tm/).
||\n\n**Additional Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**Additional Properties**|`string`, `number`, `boolean`|||\n\n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n#### sources\\.advanced\\.resources: Collector resource settings\n\nCompute resources allocated to the collector. **Only applies to the `cdc` collector type.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**cpu**
(CPU resource value)|`string`|CPU request for the collector container, for example `1` or `500m`.
||\n|**memory**
(Memory resource value)|`string`|Memory request for the collector container, for example `1024Mi` or `2Gi`.
||\n\n**Additional Properties:** not allowed \n**Minimal Properties:** 1 \n\n#### sources\\.advanced\\.riotx: Advanced RIOT\\-X settings\n\nAdvanced configuration properties for the RIOT-X Snowflake collector. **Only applies to the `riotx` collector type.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**poll**
(Polling interval)|`string`|Interval between polls for new stream changes, for example `30s` or `PT30S`.
Default: `\"30s\"`
||\n|**snapshot**
(Snapshot mode)|`string`|Initial-load behavior. `INITIAL` performs a one-time snapshot before streaming; `NEVER` skips the snapshot.
Default: `\"INITIAL\"`
Enum: `\"INITIAL\"`, `\"NEVER\"`
||\n|**streamPrefix**
(Redis stream key prefix)|`string`|Prefix used when constructing Redis stream keys, for example `data:`.
Default: `\"data:\"`
||\n|**streamLimit**
(Maximum stream length)|`integer`|Maximum number of entries kept in each Redis stream before older entries are trimmed.
Minimum: `1`
||\n|[**keyColumns**](#sourcesadvancedriotxkeycolumns)
(Key columns)|`string[]`|Deprecated RIOTX global fallback list of columns to use as message keys for every captured table. Prefer `tables..keys`
||\n|**clearOffset**
(Clear existing offset)|`boolean`|When `true`, the stored offset is cleared on startup, forcing a fresh read.
Default: `false`
||\n|**count**
(Record count limit)|`integer`|Maximum number of records to process. Set to `0` for unlimited.
Default: `0`
Minimum: `0`
||\n\n**Additional Properties:** not allowed \n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n##### sources\\.advanced\\.riotx\\.keyColumns\\[\\]: Key columns\n\nDeprecated RIOTX global fallback list of columns to use as message keys for every captured table. Prefer `tables..keys`\n\n\n\n#### sources\\.advanced\\.diagnostics: Collector diagnostics\n\nOpt-in JVM diagnostics for the source collector. Each artifact is written to the `/opt/rdi/diagnostics` directory on the host and included in the `dump-support-package` bundle. **Only applies to the `cdc` collector type on VM installations, and is ignored otherwise.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|[**heap\\_dump**](#sourcesadvanceddiagnosticsheap_dump)
(Heap dump on out\\-of\\-memory)|`object`|Writes a heap dump if the JVM runs out of memory, then exits so the collector restarts cleanly.
||\n|[**java\\_flight\\_recorder**](#sourcesadvanceddiagnosticsjava_flight_recorder)
(Java Flight Recorder recording)|`object`|Records a Java Flight Recorder profile of the collector at startup. RDI controls `filename` and `dumponexit`, and defaults `settings=profile`, `duration=5m`, and `maxsize=128m` unless set under `options`.
||\n\n**Additional Properties:** not allowed \n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n##### sources\\.advanced\\.diagnostics\\.heap\\_dump: Heap dump on out\\-of\\-memory\n\nWrites a heap dump if the JVM runs out of memory, then exits so the collector restarts cleanly.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**enabled**
(Enable heap dump)|`boolean`|When `true`, the collector writes a heap dump if the JVM runs out of memory, then exits so it restarts cleanly.
Default: `false`
||\n\n**Additional Properties:** not allowed \n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n##### sources\\.advanced\\.diagnostics\\.java\\_flight\\_recorder: Java Flight Recorder recording\n\nRecords a Java Flight Recorder profile of the collector at startup. RDI controls `filename` and `dumponexit`, and defaults `settings=profile`, `duration=5m`, and `maxsize=128m` unless set under `options`.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**enabled**
(Enable flight recording)|`boolean`|When `true`, starts a Java Flight Recorder recording when the collector starts.
Default: `false`
||\n|**options**
(Flight Recorder options)|`string`|Comma-separated `-XX:StartFlightRecording` options, for example `duration=10m,maxsize=256m,path-to-gc-roots=true`.
||\n\n**Additional Properties:** not allowed \n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n" }, { "id": "targets-target-connections", "title": "targets: Target connections", "role": "content", "text": "Target Redis databases where processed records are written. Each key is a target identifier; the value configures the connection.\n\n\n**Properties** (key: `.*`)\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|[**connection**](#targetsconnection)
(Database connection)|`object`|Connection configuration for a Redis database.
|yes|\n|**name**
(Target name)|`string`|Human-readable name for the target connection. Maximum 100 characters.
Maximal Length: `100`
|no|\n\n\n" }, { "id": "targetsconnection-database-connection", "title": "targets\\.connection: Database connection", "role": "content", "text": "Connection configuration for a Redis database.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**type**
(Database type)||Database type identifier. Always `redis` for this connection.
Constant Value: `\"redis\"`
|yes|\n|**host**
(Database host)|`string`|Hostname or IP address of the Redis server.
|yes|\n|**port**
(Database port)|`integer`|Network port on which the Redis server is listening.
Minimum: `1`
Maximum: `65535`
|yes|\n|**user**
(Database user)|`string`|Username for authentication to the Redis database.
|no|\n|**password**
(Database password)|`string`|Password for authentication to the Redis database.
|no|\n|**key**
(Private key file)|`string`|Path to the private key file used for SSL/TLS client authentication.
|no|\n|**key\\_password**
(Private key password)|`string`|Password used to decrypt the private key file.
|no|\n|**cert**
(Client certificate)|`string`|Path to the client certificate file used for SSL/TLS client authentication.
|no|\n|**cacert**
(CA certificate)|`string`|Path to the Certificate Authority (CA) certificate file used to verify the server's TLS certificate.
|no|\n\n**Additional Properties:** not allowed \n**Minimal Properties:** 3 \n**If property *key* is defined**, property/ies *cert* is/are required. \n**If property *cert* is defined**, property/ies *key* is/are required. \n**If property *key_password* is defined**, property/ies *key* is/are required. \n" }, { "id": "processors-data-processing-configuration", "title": "processors: Data processing configuration", "role": "content", "text": "Settings that control how data is processed, including batch sizes, error handling, and performance tuning.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**type**
(Processor type)|`string`|Processor implementation to run. Use `flink` (default) for the Apache Flink-based processor. Use `classic` for the classic processor.
Default: `\"flink\"`
Enum: `\"classic\"`, `\"flink\"`
||\n|**read\\_batch\\_size**|`integer`, `string`|Maximum number of records read from the source streams in a single batch.
Default: `2000`
Pattern: `^\\${.*}$`
Minimum: `1`
||\n|**read\\_batch\\_timeout\\_ms**
(Read batch timeout)|`integer`|Maximum time in milliseconds to wait for a batch to fill before processing it.
Default: `100`
Minimum: `1`
||\n|**duration**
(Batch duration limit)|`integer`, `string`|(DEPRECATED)
This property has no effect; use `read_batch_timeout_ms` instead.
Default: `100`
Pattern: `^\\${.*}$`
Minimum: `1`
||\n|**write\\_batch\\_size**|`integer`, `string`|Maximum number of records written to the target Redis database in a single batch.
Default: `200`
Pattern: `^\\${.*}$`
Minimum: `1`
||\n|**enable\\_async\\_processing**|`boolean`|When `true`, the processor handles batches asynchronously to improve throughput. **Classic processor only.**
Default: `true`
||\n|**batch\\_queue\\_size**|`integer`|Maximum number of batches queued for processing. **Classic processor only.**
Default: `3`
Minimum: `1`
||\n|**ack\\_queue\\_size**|`integer`|Maximum number of batches queued for asynchronous acknowledgement. **Classic processor only.**
Default: `10`
Minimum: `1`
||\n|**dedup**
(Enable deduplication)|`boolean`|When `true`, the processor deduplicates incoming records. **Classic processor only.**
Default: `false`
||\n|**dedup\\_max\\_size**
(Deduplication set size)|`integer`|Maximum number of entries kept in the deduplication set. **Classic processor only.**
Default: `1024`
Minimum: `1`
||\n|**dedup\\_strategy**
(Deduplication strategy)|`string`|(DEPRECATED)
This property has no effect — the only supported strategy is `ignore`. Remove it from the configuration. **Classic processor only.**
Default: `\"ignore\"`
Enum: `\"reject\"`, `\"ignore\"`
||\n|**error\\_handling**
(Error handling strategy)|`string`|Strategy for handling failed records. `ignore` silently drops them; `dlq` writes them to the dead-letter queue.
Default: `\"dlq\"`
||\n|**dlq\\_max\\_messages**
(DLQ message limit)|`integer`, `string`|Maximum number of messages stored per dead-letter queue stream.
Default: `1000`
Pattern: `^\\${.*}$`
Minimum: `1`
||\n|**target\\_data\\_type**
(Target Redis data type)|`string`|Data type used to store target records in Redis. `hash` writes a Redis Hash; `json` writes a RedisJSON document and requires the RedisJSON module.
Default: `\"hash\"`
||\n|**json\\_update\\_strategy**|`string`|Strategy for updating existing JSON documents in Redis. `replace` overwrites the entire document; `merge` merges incoming fields into it.
Default: `\"replace\"`
||\n|**use\\_native\\_json\\_merge**
(Use native JSON merge from RedisJSON module)|`boolean`|Controls whether JSON merge operations use the native `JSON.MERGE` command (when `true`) or Lua scripts (when `false`). Introduced in RDI 1.15.0. The native command provides 2x performance improvement but handles null values differently:

**Previous behavior (Lua merge)**: When merging `{\"field1\": \"value1\", \"field2\": \"value2\"}` with `{\"field2\": null, \"field3\": \"value3\"}`, the result was `{\"field1\": \"value1\", \"field2\": null, \"field3\": \"value3\"}` (null value is preserved).

**New behavior (JSON.MERGE)**: The same merge produces `{\"field1\": \"value1\", \"field3\": \"value3\"}` (null value removes the field, following [RFC 7396](https://datatracker.ietf.org/doc/html/rfc7396)).

**Note**: The native `JSON.MERGE` command requires RedisJSON 2.6.0 or higher. If the target database has an older version of RedisJSON, RDI automatically falls back to Lua-based merge operations regardless of this setting.

**Impact**: If your application logic distinguishes between a field with a `null` value and a missing field, you may need to adjust your data handling. This follows the JSON Merge Patch RFC standard but differs from the previous Lua implementation. Set to `false` to revert to the previous Lua-based merge behavior if needed.

The Flink processor always uses the native `JSON.MERGE` command when the target database supports it. **Classic processor only.**
Default: `true`
||\n|**initial\\_sync\\_processes**|`integer`, `string`|Number of parallel processes used to perform the initial data synchronization. For the Flink processor, parallelism is controlled by Flink properties instead. **Classic processor only.**
Default: `4`
Pattern: `^\\${.*}$`
Minimum: `1`
Maximum: `32`
||\n|**idle\\_sleep\\_time\\_ms**
(Idle sleep interval)|`integer`, `string`|Time in milliseconds to sleep between processing batches when idle. **Classic processor only.**
Default: `200`
Pattern: `^\\${.*}$`
Minimum: `1`
Maximum: `999999`
||\n|**idle\\_streams\\_check\\_interval\\_ms**
(Idle streams check interval)|`integer`, `string`|Time in milliseconds between checks for new streams when the processor is idle. For the Flink processor, use `processors.advanced.source.discovery.interval.ms` instead to configure a single discovery interval regardless of load. **Classic processor only.**
Default: `1000`
Pattern: `^\\${.*}$`
Minimum: `1`
Maximum: `999999`
||\n|**busy\\_streams\\_check\\_interval\\_ms**
(Busy streams check interval)|`integer`, `string`|Time in milliseconds between checks for new streams when the processor is busy. For the Flink processor, use `processors.advanced.source.discovery.interval.ms` instead to configure a single discovery interval regardless of load. **Classic processor only.**
Default: `5000`
Pattern: `^\\${.*}$`
Minimum: `1`
Maximum: `999999`
||\n|**retry\\_max\\_attempts**
(Maximum retry attempts)|`integer`, `string`|Maximum number of attempts for a failed write to the target Redis database before giving up.
Default: `5`
Pattern: `^\\${.*}$`
Minimum: `1`
||\n|**retry\\_initial\\_delay\\_ms**
(Initial retry delay)|`integer`, `string`|Initial delay in milliseconds before the first retry of a failed write.
Default: `1000`
Pattern: `^\\${.*}$`
Minimum: `1`
Maximum: `999999`
||\n|**retry\\_max\\_delay\\_ms**
(Maximum retry delay)|`integer`, `string`|Maximum delay in milliseconds between retry attempts.
Default: `10000`
Pattern: `^\\${.*}$`
Minimum: `1`
Maximum: `999999`
||\n|**wait\\_enabled**
(Enable replica wait)|`boolean`|When `true`, RDI verifies that each write has been replicated to the target database's replica shards before acknowledging it. Enable this only when target database replication is enabled and a healthy replica is available. For the Flink processor, `processors.advanced.target.wait.enabled` takes priority.
Default: `false`
||\n|**wait\\_timeout**
(Replica wait timeout)|`integer`, `string`|Maximum time in milliseconds to wait for replica write verification on the target database.
Default: `1000`
Pattern: `^\\${.*}$`
Minimum: `1`
||\n|**retry\\_on\\_replica\\_failure**|`boolean`|When `true`, RDI keeps retrying a write until replica replication is confirmed; when `false`, it gives up after the first failure.
Default: `true`
||\n|**on\\_failed\\_retry\\_interval**
(Retry interval on failure)|`integer`, `string`|(DEPRECATED)
This property has no effect; remove it from the configuration.
Default: `5`
Pattern: `^\\${.*}$`
Minimum: `1`
||\n|[**logging**](#processorslogging)
(Logging configuration)|`object`|Logging settings for the processor. **Flink processor only.**
||\n|[**advanced**](#processorsadvanced)
(Advanced configuration)|`object`|Advanced configuration for fine-tuning the processor. **All properties under `advanced` apply to the Flink processor only and are silently ignored by the classic processor.**
||\n\n**Additional Properties:** not allowed \n" }, { "id": "processorslogging-logging-configuration", "title": "processors\\.logging: Logging configuration", "role": "content", "text": "Logging settings for the processor. **Flink processor only.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**level**
(Logging level)|`string`|Log verbosity for the processor.
Default: `\"info\"`
Enum: `\"trace\"`, `\"debug\"`, `\"info\"`, `\"warn\"`, `\"error\"`
||\n\n**Additional Properties:** not allowed \n**Example**\n\n[code example]\n\n" }, { "id": "processorsadvanced-advanced-configuration", "title": "processors\\.advanced: Advanced configuration", "role": "content", "text": "Advanced configuration for fine-tuning the processor. **All properties under `advanced` apply to the Flink processor only and are silently ignored by the classic processor.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|[**source**](#processorsadvancedsource)
(Advanced source settings)|`object`|Advanced configuration properties for the source Redis client and streams reader. **Flink processor only.**
||\n|[**target**](#processorsadvancedtarget)
(Advanced target settings)|`object`|Advanced configuration properties for the target Redis client and sink. **Flink processor only.**
||\n|[**dlq**](#processorsadvanceddlq)
(Advanced DLQ settings)|`object`|Advanced configuration properties for the DLQ Redis client and sink. **Flink processor only.**
||\n|[**processor**](#processorsadvancedprocessor)
(Advanced processor settings)|`object`|Advanced configuration properties for the processor. **Flink processor only.**
||\n|[**flink**](#processorsadvancedflink)
(Advanced Flink settings)|`object`|Advanced configuration properties forwarded to the underlying Flink runtime. Any property listed in the [Flink configuration documentation](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/deployment/config/) can be set here and will override the RDI default. **Flink processor only.**
||\n|[**resources**](#processorsadvancedresources)
(Advanced resource settings)|`object`|Compute resources allocated to the Flink job, such as the number of task manager pods. **Flink processor only.**
||\n\n**Additional Properties:** not allowed \n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n#### processors\\.advanced\\.source: Advanced source settings\n\nAdvanced configuration properties for the source Redis client and streams reader. **Flink processor only.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**stream\\.name\\.pattern**
(Source stream name pattern)|`string`|Glob pattern used to discover input streams in the source Redis database, for example `data:*`.
Default: `\"data:*\"`
||\n|**discovery\\.interval\\.ms**
(Stream discovery interval)|`integer`|Time in milliseconds between checks for new input streams. Replaces the classic `processors.idle_streams_check_interval_ms` and `processors.busy_streams_check_interval_ms` properties.
Default: `1000`
Minimum: `0`
||\n|**batch\\.size**
(Source batch size)|`integer`|Maximum number of records the source operator reads in a single batch. Alias for `processors.read_batch_size`; takes priority when both are set.
Default: `2000`
Minimum: `1`
||\n|**batch\\.timeout\\.ms**
(Source batch timeout)|`integer`|Maximum time in milliseconds to wait for a source batch to fill before processing. Alias for `processors.read_batch_timeout_ms`; takes priority when both are set.
Default: `100`
Minimum: `1`
||\n|**connection\\.timeout\\.ms**
(Source connection timeout)|`integer`|Connection timeout in milliseconds for the source Redis client.
Default: `2000`
Minimum: `1`
||\n|**socket\\.timeout\\.ms**
(Source socket timeout)|`integer`|Socket read/write timeout in milliseconds for the source Redis client.
Default: `2000`
Minimum: `1`
||\n|**retry\\.max\\.attempts**
(Source retry max attempts)|`integer`|Maximum number of retry attempts for failed source Redis operations.
Default: `5`
Minimum: `1`
||\n|**retry\\.initial\\.delay\\.ms**
(Source retry initial delay)|`integer`|Initial delay in milliseconds before the first retry of a failed source Redis operation.
Default: `100`
Minimum: `1`
||\n|**retry\\.max\\.delay\\.ms**
(Source retry max delay)|`integer`|Maximum delay in milliseconds between retry attempts for source Redis operations.
Default: `3000`
Minimum: `1`
||\n|**retry\\.backoff\\.multiplier**
(Source retry backoff multiplier)|`number`|Exponential backoff multiplier between retry attempts for source Redis operations.
Default: `2`
Minimum: `1`
||\n\n**Additional Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**Additional Properties**|`string`, `number`, `boolean`|||\n\n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n#### processors\\.advanced\\.target: Advanced target settings\n\nAdvanced configuration properties for the target Redis client and sink. **Flink processor only.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**batch\\.size**
(Target sink batch size)|`integer`|Maximum number of records the target sink writes in a single batch. Alias for `processors.write_batch_size`; takes priority when both are set.
Default: `200`
Minimum: `1`
||\n|**flush\\.interval\\.ms**
(Target sink flush interval)|`integer`|Maximum time in milliseconds the target sink waits to fill a batch before flushing it to Redis.
Default: `100`
Minimum: `1`
||\n|**connection\\.timeout\\.ms**
(Target connection timeout)|`integer`|Connection timeout in milliseconds for the target Redis client.
Default: `2000`
Minimum: `1`
||\n|**socket\\.timeout\\.ms**
(Target socket timeout)|`integer`|Socket read/write timeout in milliseconds for the target Redis client.
Default: `2000`
Minimum: `1`
||\n|**retry\\.max\\.attempts**
(Target retry max attempts)|`integer`|Maximum number of retry attempts for failed target Redis operations. Alias for `processors.retry_max_attempts`; takes priority when both are set.
Default: `5`
Minimum: `1`
||\n|**retry\\.initial\\.delay\\.ms**
(Target retry initial delay)|`integer`|Initial delay in milliseconds before the first retry of a failed target Redis operation. Alias for `processors.retry_initial_delay_ms`; takes priority when both are set.
Default: `1000`
Minimum: `1`
||\n|**retry\\.max\\.delay\\.ms**
(Target retry max delay)|`integer`|Maximum delay in milliseconds between retry attempts for target Redis operations. Alias for `processors.retry_max_delay_ms`; takes priority when both are set.
Default: `10000`
Minimum: `1`
||\n|**retry\\.backoff\\.multiplier**
(Target retry backoff multiplier)|`number`|Exponential backoff multiplier between retry attempts for target Redis operations.
Default: `2`
Minimum: `1`
||\n|**wait\\.enabled**
(Target replica wait enabled)|`boolean`|When `true`, RDI verifies that each write has been replicated to the target database's replica shards before acknowledging it. Enable this only when target database replication is enabled and a healthy replica is available. Alias for `processors.wait_enabled`; takes priority when both are set.
Default: `false`
||\n|**wait\\.write\\.timeout\\.ms**
(Target replica wait timeout)|`integer`|Maximum time in milliseconds to wait for target replica write verification. Alias for `processors.wait_timeout`; takes priority when both are set.
Default: `1000`
Minimum: `1`
||\n|**wait\\.retry\\.enabled**
(Target replica wait retry enabled)|`boolean`|When `true`, RDI keeps retrying a target write until replica replication is confirmed; when `false`, it gives up after the first failure. Alias for `processors.retry_on_replica_failure`; takes priority when both are set. When enabled, the Flink processor retries indefinitely. Failed checkpoints can restart the job, after which the retries resume. The classic processor retries once.
Default: `true`
||\n|**wait\\.retry\\.delay\\.ms**
(Target replica wait retry delay)|`integer`|Delay in milliseconds between target replica wait retry attempts.
Default: `1000`
Minimum: `1`
||\n\n**Additional Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**Additional Properties**|`string`, `number`, `boolean`|||\n\n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n#### processors\\.advanced\\.dlq: Advanced DLQ settings\n\nAdvanced configuration properties for the DLQ Redis client and sink. **Flink processor only.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**max\\.len**
(DLQ sink max length)|`integer`|Maximum number of messages stored per dead letter queue stream. Alias for `processors.dlq_max_messages`; takes priority when both are set.
Default: `1000`
Minimum: `1`
||\n|**batch\\.size**
(DLQ sink batch size)|`integer`|Maximum number of records the DLQ sink writes in a single batch.
Default: `100`
Minimum: `1`
||\n|**flush\\.interval\\.ms**
(DLQ sink flush interval)|`integer`|Maximum time in milliseconds the DLQ sink waits to fill a batch before flushing it to Redis.
Default: `100`
Minimum: `1`
||\n|**connection\\.timeout\\.ms**
(DLQ connection timeout)|`integer`|Connection timeout in milliseconds for the DLQ Redis client.
Default: `2000`
Minimum: `1`
||\n|**socket\\.timeout\\.ms**
(DLQ socket timeout)|`integer`|Socket read/write timeout in milliseconds for the DLQ Redis client.
Default: `2000`
Minimum: `1`
||\n|**retry\\.max\\.attempts**
(DLQ retry max attempts)|`integer`|Maximum number of retry attempts for failed DLQ Redis operations.
Default: `1`
Minimum: `1`
||\n|**retry\\.initial\\.delay\\.ms**
(DLQ retry initial delay)|`integer`|Initial delay in milliseconds before the first retry of a failed DLQ Redis operation.
Default: `100`
Minimum: `1`
||\n|**retry\\.max\\.delay\\.ms**
(DLQ retry max delay)|`integer`|Maximum delay in milliseconds between retry attempts for DLQ Redis operations.
Default: `3000`
Minimum: `1`
||\n|**retry\\.backoff\\.multiplier**
(DLQ retry backoff multiplier)|`number`|Exponential backoff multiplier between retry attempts for DLQ Redis operations.
Default: `2`
Minimum: `1`
||\n|**wait\\.enabled**
(DLQ replica wait enabled)|`boolean`|When `true`, RDI verifies that each DLQ write has been replicated to the DLQ database's replica shards before acknowledging it.
Default: `false`
||\n|**wait\\.write\\.timeout\\.ms**
(DLQ replica wait timeout)|`integer`|Maximum time in milliseconds to wait for DLQ replica write verification.
Default: `1000`
Minimum: `1`
||\n|**wait\\.retry\\.enabled**
(DLQ replica wait retry enabled)|`boolean`|When `true`, RDI keeps retrying a DLQ write until replica replication is confirmed; when `false`, it gives up after the first failure.
Default: `false`
||\n|**wait\\.retry\\.delay\\.ms**
(DLQ replica wait retry delay)|`integer`|Delay in milliseconds between DLQ replica wait retry attempts.
Default: `1000`
Minimum: `1`
||\n\n**Additional Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**Additional Properties**|`string`, `number`, `boolean`|||\n\n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n#### processors\\.advanced\\.processor: Advanced processor settings\n\nAdvanced configuration properties for the processor. **Flink processor only.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**default\\.data\\.type**
(Default target data type)|`string`|Data type to use in Redis when not overridden per job: `hash` for Redis Hash, `json` for RedisJSON. Alias for `processors.target_data_type`; takes priority when both are set.
Default: `\"hash\"`
Enum: `\"hash\"`, `\"json\"`
||\n|**default\\.json\\.update\\.strategy**
(Default JSON update strategy)|`string`|Strategy for updating JSON data in Redis: `replace` to overwrite the entire JSON object, `merge` to merge new data with the existing JSON object. Alias for `processors.json_update_strategy`; takes priority when both are set.
Default: `\"replace\"`
Enum: `\"replace\"`, `\"merge\"`
||\n|**dlq\\.enabled**
(Enable DLQ)|`boolean`|When `true`, rejected messages are stored in the dead-letter queue; when `false`, errors are silently skipped. Alias for `processors.error_handling`; takes priority when both are set.
Default: `true`
||\n\n**Additional Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**Additional Properties**|`string`, `number`, `boolean`|||\n\n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n#### processors\\.advanced\\.flink: Advanced Flink settings\n\nAdvanced configuration properties forwarded to the underlying Flink runtime. Any property listed in the [Flink configuration documentation](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/deployment/config/) can be set here and will override the RDI default. **Flink processor only.**

The properties listed below are the ones most likely to require adjustment. **Changing any other Flink property is not recommended unless instructed by Redis support.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**parallelism\\.default**
(Default parallelism)|`integer`|Default parallelism for jobs and operators. When unset, Flink uses the number of available task slots across all task managers (`taskManager.replicas × taskmanager.numberOfTaskSlots`). Increase to fan out work across more task slots; see [parallel execution](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/dev/datastream/execution/parallel/).
Minimum: `1`
||\n|**taskmanager\\.numberOfTaskSlots**
(Task slots per task manager)|`integer`|Number of parallel task slots per task manager pod. Each slot can run one parallel pipeline instance, so this caps the parallelism a single task manager can absorb. See [task slots and resources](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/concepts/flink-architecture/#task-slots-and-resources).
Default: `1`
Minimum: `1`
||\n|**taskmanager\\.memory\\.process\\.size**
(Task manager process memory)|`string`|Total memory budget for each task manager JVM process (heap + managed + network + metaspace + JVM overhead), expressed with a unit suffix such as `2048m` or `4g`. See [task manager memory configuration](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/deployment/memory/mem_setup_tm/).
Default: `\"2048m\"`
||\n\n**Additional Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**Additional Properties**|`string`, `number`, `boolean`|||\n\n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n#### processors\\.advanced\\.resources: Advanced resource settings\n\nCompute resources allocated to the Flink job, such as the number of task manager pods. **Flink processor only.**\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|[**taskManager**](#processorsadvancedresourcestaskmanager)
(Task manager resource settings)|`object`|Resource settings for Flink task manager pods.
||\n\n**Additional Properties:** not allowed \n**Minimal Properties:** 1 \n**Example**\n\n[code example]\n\n\n##### processors\\.advanced\\.resources\\.taskManager: Task manager resource settings\n\nResource settings for Flink task manager pods.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**replicas**
(Task manager replicas)|`integer`|Number of Flink task manager pods to run.
Minimum: `1`
||\n\n**Additional Properties:** not allowed \n**Minimal Properties:** 1 \n" }, { "id": "secret-providers-secret-providers", "title": "secret\\-providers: Secret providers", "role": "content", "text": "External secret providers used to resolve `${...}` references in the configuration.\n\n\n**Properties** (key: `.*`)\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**type**
(Provider type)|`string`|Secret provider backend. `aws` uses AWS Secrets Manager; `vault` uses HashiCorp Vault.
Enum: `\"aws\"`, `\"vault\"`
|yes|\n|[**parameters**](#secret-providersparameters)
(Provider parameters)|`object`|Configuration parameters for the secret provider.
|yes|\n\n\n" }, { "id": "secret-providersparameters-provider-parameters", "title": "secret\\-providers\\.parameters: Provider parameters", "role": "content", "text": "Configuration parameters for the secret provider.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|[**objects**](#secret-providersparametersobjects)
(Secrets objects array)|`object[]`|Secret objects to fetch from the provider.
|yes|\n\n**Example**\n\n[code example]\n\n\n#### secret\\-providers\\.parameters\\.objects\\[\\]: Secrets objects array\n\nSecret objects to fetch from the provider.\n\n\n**Items: Secret object**\n\n**No properties.**\n\n**Example**\n\n[code example]\n\n" }, { "id": "metadata-pipeline-metadata", "title": "metadata: Pipeline metadata", "role": "content", "text": "Optional metadata describing this pipeline, such as a display name and description.\n\n\n**Properties**\n\n|Name|Type|Description|Required|\n|----|----|-----------|--------|\n|**name**
(Pipeline name)|`string`|Human-readable name for the pipeline. Maximum 100 characters.
Maximal Length: `100`
||\n|**description**
(Pipeline description)|`string`|Free-form description of what the pipeline does. Maximum 500 characters.
Maximal Length: `500`
||\n|**revision**
(Pipeline revision)|`integer`|Pipeline revision number. Must be a non-negative integer.
Minimum: `0`
||\n|[**tags**](#metadatatags)
(Pipeline tags)|`string[]`|Array of pipeline tags. Each tag must be a string of up to 50 characters, containing only alphanumeric characters, dots, dashes, or underscores, and must start and end with an alphanumeric character. Tags must be unique.
||\n\n**Additional Properties:** not allowed \n" }, { "id": "metadatatags-pipeline-tags", "title": "metadata\\.tags\\[\\]: Pipeline tags", "role": "content", "text": "Array of pipeline tags. Each tag must be a string of up to 50 characters, containing only alphanumeric characters, dots, dashes, or underscores, and must start and end with an alphanumeric character. Tags must be unique.\n\n\n**Unique Items:** yes" } ], "examples": [ { "id": "sourcesconnection-source-database-connection-ex0", "language": "yaml", "code": "SQL database:\n hr:\n type: postgresql\n host: localhost\n port: 5432\n database: postgres\n user: postgres\n password: postgres\nMongoDB:\n mongodb-source:\n type: mongodb\n connection_string: mongodb://localhost:27017/?replicaSet=rs0\n user: debezium\n password: dbz\n database: db1,db2\nSpanner:\n spanner-source:\n type: spanner\n project_id: example-12345\n instance_id: example\n database_id: example\n change_streams:\n change_stream_all:\n retention_period_hours: 24\nSnowflake:\n snowflake:\n type: snowflake\n url: jdbc:snowflake://myaccount.snowflakecomputing.com/\n user: myuser\n password: mypassword\n database: MYDB\n warehouse: COMPUTE_WH", "section_id": "sourcesconnection-source-database-connection" }, { "id": "sourcesconnection-source-database-connection-ex1", "language": "yaml", "code": "hr:\n type: postgresql\n host: localhost\n port: 5432\n database: postgres\n user: postgres\n password: postgres", "section_id": "sourcesconnection-source-database-connection" }, { "id": "sourcesconnection-source-database-connection-ex2", "language": "yaml", "code": "my-oracle:\n type: oracle\n host: 172.17.0.4\n port: 1521\n user: c##dbzuser\n password: dbz", "section_id": "sourcesconnection-source-database-connection" }, { "id": "sourcesconnection-source-database-connection-ex3", "language": "yaml", "code": "mongodb-source:\n type: mongodb\n connection_string: mongodb://localhost:27017/?replicaSet=rs0\n user: debezium\n password: dbz\n database: db1,db2", "section_id": "sourcesconnection-source-database-connection" }, { "id": "sourcesconnection-source-database-connection-ex4", "language": "yaml", "code": "spanner-source:\n type: spanner\n project_id: example-12345\n instance_id: example\n database_id: example\n change_streams:\n change_stream_all:\n retention_period_hours: 24", "section_id": "sourcesconnection-source-database-connection" }, { "id": "sourcesconnection-source-database-connection-ex5", "language": "yaml", "code": "snowflake:\n type: snowflake\n url: jdbc:snowflake://myaccount.snowflakecomputing.com/\n user: myuser\n password: mypassword\n database: MYDB\n warehouse: COMPUTE_WH", "section_id": "sourcesconnection-source-database-connection" }, { "id": "sourceslogging-logging-configuration-ex0", "language": "yaml", "code": "level: info", "section_id": "sourceslogging-logging-configuration" }, { "id": "sourcesadvanced-advanced-configuration-ex0", "language": "yaml", "code": "sink:\n redis.batch.size: 1000\n redis.flush.interval.ms: 100\n redis.connection.timeout.ms: 2000\n redis.socket.timeout.ms: 2000\n redis.retry.max.attempts: 5\n redis.retry.initial.delay.ms: 100\n redis.retry.max.delay.ms: 3000\n redis.retry.backoff.multiplier: 2\n redis.oom.retry.initial.delay.ms: 1000\n redis.oom.retry.max.delay.ms: 10000\n redis.oom.retry.backoff.multiplier: 2\n redis.wait.enabled: false\n redis.wait.timeout.ms: 1000\n redis.wait.write.timeout.ms: 1000\n redis.wait.retry.enabled: false\n redis.wait.retry.delay.ms: 1000\nsource:\n snapshot.mode: initial\n snapshot.max.threads: 1\n poll.interval.ms: 500\n max.batch.size: 2048\n max.queue.size: 8192\n heartbeat.interval.ms: 0\n database.trustServerCertificate: false\n driver.applicationIntent: ReadWrite\n lob.enabled: false\n publication.autocreate.mode: all_tables\n publication.name: dbz_publication\n slot.name: debezium\n capture.mode: change_streams_update_full\n capture.mode.full.update.type: lookup\n cursor.oversize.handling.mode: fail\n cursor.oversize.skip.threshold: 0\n mongodb.authsource: admin\n mongodb.ssl.invalid.hostname.allowed: false\n mongodb.connect.timeout.ms: 10000\n mongodb.socket.timeout.ms: 0\n mongodb.server.selection.timeout.ms: 30000\n mongodb.heartbeat.frequency.ms: 10000\n spanner.version.retention.period.hours: 1\n spanner.fetch.timeout.ms: 500\n spanner.fetch.heartbeat.ms: 100\n spanner.max.rows.per.partition: 10000\n spanner.dialect: GOOGLESQL\nquarkus: {}\nflink:\n taskmanager.numberOfTaskSlots: 1\nresources: {}\nriotx:\n poll: 30s\n snapshot: INITIAL\n streamPrefix: 'data:'\n clearOffset: false\n count: 0\ndiagnostics:\n heap_dump:\n enabled: false\n java_flight_recorder:\n enabled: false", "section_id": "sourcesadvanced-advanced-configuration" }, { "id": "sourcesadvanced-advanced-configuration-ex1", "language": "yaml", "code": "redis.batch.size: 1000\nredis.flush.interval.ms: 100\nredis.connection.timeout.ms: 2000\nredis.socket.timeout.ms: 2000\nredis.retry.max.attempts: 5\nredis.retry.initial.delay.ms: 100\nredis.retry.max.delay.ms: 3000\nredis.retry.backoff.multiplier: 2\nredis.oom.retry.initial.delay.ms: 1000\nredis.oom.retry.max.delay.ms: 10000\nredis.oom.retry.backoff.multiplier: 2\nredis.wait.enabled: false\nredis.wait.timeout.ms: 1000\nredis.wait.write.timeout.ms: 1000\nredis.wait.retry.enabled: false\nredis.wait.retry.delay.ms: 1000", "section_id": "sourcesadvanced-advanced-configuration" }, { "id": "sourcesadvanced-advanced-configuration-ex2", "language": "yaml", "code": "snapshot.mode: initial\nsnapshot.max.threads: 1\npoll.interval.ms: 500\nmax.batch.size: 2048\nmax.queue.size: 8192\nheartbeat.interval.ms: 0\ndatabase.trustServerCertificate: false\ndriver.applicationIntent: ReadWrite\nlob.enabled: false\npublication.autocreate.mode: all_tables\npublication.name: dbz_publication\nslot.name: debezium\ncapture.mode: change_streams_update_full\ncapture.mode.full.update.type: lookup\ncursor.oversize.handling.mode: fail\ncursor.oversize.skip.threshold: 0\nmongodb.authsource: admin\nmongodb.ssl.invalid.hostname.allowed: false\nmongodb.connect.timeout.ms: 10000\nmongodb.socket.timeout.ms: 0\nmongodb.server.selection.timeout.ms: 30000\nmongodb.heartbeat.frequency.ms: 10000\nspanner.version.retention.period.hours: 1\nspanner.fetch.timeout.ms: 500\nspanner.fetch.heartbeat.ms: 100\nspanner.max.rows.per.partition: 10000\nspanner.dialect: GOOGLESQL", "section_id": "sourcesadvanced-advanced-configuration" }, { "id": "sourcesadvanced-advanced-configuration-ex3", "language": "yaml", "code": "taskmanager.numberOfTaskSlots: 1", "section_id": "sourcesadvanced-advanced-configuration" }, { "id": "sourcesadvanced-advanced-configuration-ex4", "language": "yaml", "code": "poll: 30s\nsnapshot: INITIAL\nstreamPrefix: 'data:'\nclearOffset: false\ncount: 0", "section_id": "sourcesadvanced-advanced-configuration" }, { "id": "sourcesadvanced-advanced-configuration-ex5", "language": "yaml", "code": "heap_dump:\n enabled: false\njava_flight_recorder:\n enabled: false", "section_id": "sourcesadvanced-advanced-configuration" }, { "id": "sourcesadvanced-advanced-configuration-ex6", "language": "yaml", "code": "enabled: false", "section_id": "sourcesadvanced-advanced-configuration" }, { "id": "sourcesadvanced-advanced-configuration-ex7", "language": "yaml", "code": "enabled: false", "section_id": "sourcesadvanced-advanced-configuration" }, { "id": "processorslogging-logging-configuration-ex0", "language": "yaml", "code": "level: info", "section_id": "processorslogging-logging-configuration" }, { "id": "processorsadvanced-advanced-configuration-ex0", "language": "yaml", "code": "source:\n stream.name.pattern: data:*\n discovery.interval.ms: 1000\n batch.size: 2000\n batch.timeout.ms: 100\n connection.timeout.ms: 2000\n socket.timeout.ms: 2000\n retry.max.attempts: 5\n retry.initial.delay.ms: 100\n retry.max.delay.ms: 3000\n retry.backoff.multiplier: 2\ntarget:\n batch.size: 200\n flush.interval.ms: 100\n connection.timeout.ms: 2000\n socket.timeout.ms: 2000\n retry.max.attempts: 5\n retry.initial.delay.ms: 1000\n retry.max.delay.ms: 10000\n retry.backoff.multiplier: 2\n wait.enabled: false\n wait.write.timeout.ms: 1000\n wait.retry.enabled: true\n wait.retry.delay.ms: 1000\ndlq:\n max.len: 1000\n batch.size: 100\n flush.interval.ms: 100\n connection.timeout.ms: 2000\n socket.timeout.ms: 2000\n retry.max.attempts: 1\n retry.initial.delay.ms: 100\n retry.max.delay.ms: 3000\n retry.backoff.multiplier: 2\n wait.enabled: false\n wait.write.timeout.ms: 1000\n wait.retry.enabled: false\n wait.retry.delay.ms: 1000\nprocessor:\n default.data.type: hash\n default.json.update.strategy: replace\n dlq.enabled: true\nflink:\n taskmanager.numberOfTaskSlots: 1\n taskmanager.memory.process.size: 2048m\nresources:\n taskManager: {}", "section_id": "processorsadvanced-advanced-configuration" }, { "id": "processorsadvanced-advanced-configuration-ex1", "language": "yaml", "code": "stream.name.pattern: data:*\ndiscovery.interval.ms: 1000\nbatch.size: 2000\nbatch.timeout.ms: 100\nconnection.timeout.ms: 2000\nsocket.timeout.ms: 2000\nretry.max.attempts: 5\nretry.initial.delay.ms: 100\nretry.max.delay.ms: 3000\nretry.backoff.multiplier: 2", "section_id": "processorsadvanced-advanced-configuration" }, { "id": "processorsadvanced-advanced-configuration-ex2", "language": "yaml", "code": "batch.size: 200\nflush.interval.ms: 100\nconnection.timeout.ms: 2000\nsocket.timeout.ms: 2000\nretry.max.attempts: 5\nretry.initial.delay.ms: 1000\nretry.max.delay.ms: 10000\nretry.backoff.multiplier: 2\nwait.enabled: false\nwait.write.timeout.ms: 1000\nwait.retry.enabled: true\nwait.retry.delay.ms: 1000", "section_id": "processorsadvanced-advanced-configuration" }, { "id": "processorsadvanced-advanced-configuration-ex3", "language": "yaml", "code": "max.len: 1000\nbatch.size: 100\nflush.interval.ms: 100\nconnection.timeout.ms: 2000\nsocket.timeout.ms: 2000\nretry.max.attempts: 1\nretry.initial.delay.ms: 100\nretry.max.delay.ms: 3000\nretry.backoff.multiplier: 2\nwait.enabled: false\nwait.write.timeout.ms: 1000\nwait.retry.enabled: false\nwait.retry.delay.ms: 1000", "section_id": "processorsadvanced-advanced-configuration" }, { "id": "processorsadvanced-advanced-configuration-ex4", "language": "yaml", "code": "default.data.type: hash\ndefault.json.update.strategy: replace\ndlq.enabled: true", "section_id": "processorsadvanced-advanced-configuration" }, { "id": "processorsadvanced-advanced-configuration-ex5", "language": "yaml", "code": "taskmanager.numberOfTaskSlots: 1\ntaskmanager.memory.process.size: 2048m", "section_id": "processorsadvanced-advanced-configuration" }, { "id": "processorsadvanced-advanced-configuration-ex6", "language": "yaml", "code": "taskManager: {}", "section_id": "processorsadvanced-advanced-configuration" }, { "id": "secret-providersparameters-provider-parameters-ex0", "language": "yaml", "code": "objects:\n - {}", "section_id": "secret-providersparameters-provider-parameters" }, { "id": "secret-providersparameters-provider-parameters-ex1", "language": "yaml", "code": "- {}", "section_id": "secret-providersparameters-provider-parameters" } ] }