{
  "schema_version": 2,
  "id": "integrate/redis-data-integration/installation/migration-classic-to-flink",
  "title": "Migrate from the classic processor to the Flink processor",
  "url": "https://redis.io/docs/latest/integrate/redis-data-integration/installation/migration-classic-to-flink/",
  "summary": "Learn how to migrate an existing RDI pipeline from the classic processor to the Apache Flink-based processor.",
  "tags": [
    "docs",
    "integrate",
    "rs",
    "rdi"
  ],
  "last_updated": "2026-09-18T16:39:21+01:00",
  "page_type": "content",
  "content_hash": "bbd4eb2fbe4152f9cab7f276ab02b1a77e95c9334d46343509f6d7f60adc3c5c",
  "sections": [
    {
      "id": "overview",
      "title": "Overview",
      "role": "overview",
      "text": "RDI ships with two stream processor implementations. The *classic*\nprocessor is implemented in Python. The default *Flink* processor is built on top of\n[Apache Flink](https://flink.apache.org/). Both run on VM and Kubernetes\ninstallations. The Flink processor can achieve much higher throughput\nduring snapshots, scales horizontally by changing the number of TaskManager replicas,\nand uses Flink checkpointing for fault tolerance. See [Stream processor implementations](https://redis.io/docs/latest/integrate/redis-data-integration/architecture#stream-processor-implementations)\nfor an overview.\n\nThe classic processor is the default in RDI 1.19.0. The Flink processor is the\ndefault starting with RDI 2.0.0.\n\nThis page describes how to migrate an existing pipeline from the classic\nprocessor to the Flink processor. The steps are the same on VMs and Kubernetes,\nexcept for the optional Helm-level tuning in [Step 1](#step-1-configure-the-flink-processor-at-the-helm-chart-level-kubernetes),\nwhich applies to Kubernetes only."
    },
    {
      "id": "before-you-migrate",
      "title": "Before you migrate",
      "role": "content",
      "text": "This procedure migrates the pipeline processor on RDI 1.19.0. It does not\nupgrade RDI.\n\nBefore you start, save your existing configuration and jobs. Wait for the\ninitial snapshot to finish. Interrupting it causes the snapshot to restart\nfrom the beginning.\n\n> [!WARNING]\n> Switching processors with records still in the RDI input streams can leave\n> records unprocessed. Stop the collector and let the classic processor empty\n> the streams before switching.\n\nConfirm that your pipeline is compatible with the Flink processor:\n\n-   `JSON.MERGE` semantics differ from the classic processor's Lua-based merge\n    when null values are involved (see\n    [`use_native_json_merge`](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration)).\n    The Flink processor always uses the native `JSON.MERGE` command when the\n    target database supports it.\n-   Ensure your Kubernetes cluster or VM has enough capacity for the Flink JobManager\n    and TaskManager pods (see\n    [Configure the Flink processor](https://redis.io/docs/latest/integrate/redis-data-integration/installation/install-k8s#configure-the-flink-processor)\n    for the default sizing)."
    },
    {
      "id": "step-1-configure-the-flink-processor-at-the-helm-chart-level-kubernetes",
      "title": "Step 1: Configure the Flink processor at the Helm chart level (Kubernetes)",
      "role": "content",
      "text": "This step applies to **Kubernetes** installations only. On VM installations,\ncontinue with [Step 2](#step-2-disable-source-collection).\n\nThe Flink processor is always available — no opt-in is required at the Helm\nchart level. The defaults are sized for typical workloads, so you can skip\nthis step if you don't need to override them. To adjust the JobManager and\nTaskManager defaults, add an `operator.dataPlane.flinkProcessor` block to\nyour `rdi-values.yaml` file and run `helm upgrade` as described in\n[Configure the Flink processor](https://redis.io/docs/latest/integrate/redis-data-integration/installation/install-k8s#configure-the-flink-processor).\nExisting pipelines continue to run on the classic processor until you switch\nthem in [Step 4](#step-4-switch-processors-and-resume-collection).\n\nFor VM installations, skip this step. You can configure per-pipeline Flink\nresources in [Step 6](#step-6-tune-the-flink-processor-optional)."
    },
    {
      "id": "step-2-disable-source-collection",
      "title": "Step 2: Disable source collection",
      "role": "content",
      "text": "In your existing `config.yaml`, add `active: false` under the source and set\n`processors.type` to `classic`. Use your existing source name and preserve\nall other source, target, processor, and job settings. This example shows\nonly the fields to change:\n\n[code example]\n\nDeploy the complete configuration directory, including the existing jobs:\n\n[code example]\n\nWait for the deployment to finish and the source collector to stop. Keep\nthe pipeline active so the classic processor can process the remaining input records.\n\nApplications can continue writing to the source database while collection\nis disabled. When the collector restarts, it resumes from the saved source\nposition and processes those changes."
    },
    {
      "id": "step-3-wait-for-the-input-streams-to-empty",
      "title": "Step 3: Wait for the input streams to empty",
      "role": "content",
      "text": "After the collector has stopped, wait for every input stream to have a length\nof `0` in three complete checks, five seconds apart. An error, missing\nstatistics, or an unexpectedly empty stream list does not count as `0`.\nDo not include DLQ streams. Use any of the following methods."
    },
    {
      "id": "check-with-redis-di",
      "title": "Check with `redis-di`",
      "role": "content",
      "text": "Run:\n\n[code example]\n\nIn the **Statistics** table, the **Pending** value for each classic processor\nstream is its current length. Confirm that every input stream is listed."
    },
    {
      "id": "check-with-redis-commands",
      "title": "Check with Redis commands",
      "role": "content",
      "text": "Connect an authenticated Redis client to the **RDI database** that stores\nthe pipeline's input streams, not the target database. Find the input stream\nkeys with\n[`SCAN`](https://redis.io/docs/latest/commands/scan):\n\n[code example]\n\nIf the returned cursor is not `0`, pass it to the next command:\n\n[code example]\n\nRepeat with each new cursor until the returned cursor is `0`, even if an\nintermediate result contains no keys.\n\nFor every input stream returned, run\n[`XLEN`](https://redis.io/docs/latest/commands/xlen):\n\n[code example]\n\nRun a complete `SCAN` and all `XLEN` commands in each of the three checks."
    },
    {
      "id": "step-4-switch-processors-and-resume-collection",
      "title": "Step 4: Switch processors and resume collection",
      "role": "content",
      "text": "After the drain check passes, remove the source's `active: false` setting\nfrom the existing `config.yaml` and set\n[`processors.type`](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config#processors)\nto `flink` (or remove this property, since `flink` is the default):\n\n[code example]\n\nRDI 1.19.x requires this setting because its default processor is `classic`.\nOn RDI 2.0.0 the default processor is `flink`, so this setting is no longer\nrequired.\n\nKeep the remaining configuration and jobs, then redeploy the complete\nconfiguration directory:\n\n[code example]\n\nWait for the classic processor to terminate and the Flink JobManager and\nTaskManager workloads to become healthy. Confirm that collection resumes\nfrom the saved source position and changes committed during the pause reach\nthe target. Verify new inserts, updates, and deletes. The processor migration\nis complete after these checks pass."
    },
    {
      "id": "step-5-adapt-deprecated-and-classic-only-properties",
      "title": "Step 5: Adapt deprecated and classic-only properties",
      "role": "content",
      "text": "Some `processors` properties are no-ops, classic-only, or have moved to\n`processors.advanced` for the Flink processor. The following table lists the\nproperties that need attention when migrating.\n\n| Property | Action when migrating to Flink |\n| :-- | :-- |\n| `on_failed_retry_interval` | No-op. Remove. |\n| `duration` | No-op. Use `read_batch_timeout_ms` instead. |\n| `dedup`, `dedup_max_size`, `dedup_strategy` | Classic-only. Remove. |\n| `enable_async_processing`, `batch_queue_size`, `ack_queue_size` | Classic-only. Remove. |\n| `initial_sync_processes` | Classic-only. Configure parallelism through `advanced.flink.taskmanager.numberOfTaskSlots` and `advanced.resources.taskManager.replicas` instead. |\n| `idle_streams_check_interval_ms`, `busy_streams_check_interval_ms` | Classic-only. Use `processors.advanced.source.discovery.interval.ms` for a single discovery interval. |\n| `idle_sleep_time_ms` | Classic-only. Remove. |\n| `use_native_json_merge` | Classic-only. The Flink processor always uses `JSON.MERGE` when the target supports it. |\n\nThe classic processor silently ignores `processors.advanced`,\nand the Flink processor silently ignores classic-only top-level properties, so keeping\nboth top-level properties and their `processors.advanced` equivalents lets\nyou switch back without further edits."
    },
    {
      "id": "step-6-tune-the-flink-processor-optional",
      "title": "Step 6: Tune the Flink processor (optional)",
      "role": "content",
      "text": "Fine-tune the Flink processor through the `processors.advanced` section.\nFor example:\n\n[code example]\n\nSee the\n[`processors.advanced` reference](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#processorsadvanced-advanced-configuration)\nfor the full set of available properties."
    },
    {
      "id": "step-7-update-observability",
      "title": "Step 7: Update observability",
      "role": "content",
      "text": "The Flink processor exposes Prometheus metrics directly\nfrom the Flink JobManager and TaskManager pods.\nSee\n[Flink processor metrics](https://redis.io/docs/latest/integrate/redis-data-integration/observability#flink-processor-metrics)\nfor the `ServiceMonitor` configuration and the available metrics."
    },
    {
      "id": "rolling-back",
      "title": "Rolling back",
      "role": "content",
      "text": "To revert a pipeline to the classic processor, set `processors.type` to\n`classic` and redeploy the pipeline. This setting is required on RDI 2.0.0,\nwhere the default is `flink`. The classic processor silently ignores\n`processors.advanced`, so you don't need to remove it before switching back."
    }
  ],
  "examples": [
    {
      "id": "step-2-disable-source-collection-ex0",
      "language": "yaml",
      "code": "sources:\n  <existing-source-name>:\n    active: false\nprocessors:\n  type: classic",
      "section_id": "step-2-disable-source-collection"
    },
    {
      "id": "step-2-disable-source-collection-ex1",
      "language": "bash",
      "code": "redis-di deploy --dir <pipeline-config-directory>",
      "section_id": "step-2-disable-source-collection"
    },
    {
      "id": "check-with-redis-di-ex0",
      "language": "bash",
      "code": "redis-di describe",
      "section_id": "check-with-redis-di"
    },
    {
      "id": "check-with-redis-commands-ex0",
      "language": "text",
      "code": "SCAN 0 MATCH data:{rdi}:* COUNT 1000 TYPE stream",
      "section_id": "check-with-redis-commands"
    },
    {
      "id": "check-with-redis-commands-ex1",
      "language": "text",
      "code": "SCAN <returned-cursor> MATCH data:{rdi}:* COUNT 1000 TYPE stream",
      "section_id": "check-with-redis-commands"
    },
    {
      "id": "check-with-redis-commands-ex2",
      "language": "text",
      "code": "XLEN <input-stream-key>",
      "section_id": "check-with-redis-commands"
    },
    {
      "id": "step-4-switch-processors-and-resume-collection-ex0",
      "language": "yaml",
      "code": "processors:\n  type: flink",
      "section_id": "step-4-switch-processors-and-resume-collection"
    },
    {
      "id": "step-4-switch-processors-and-resume-collection-ex1",
      "language": "bash",
      "code": "redis-di deploy --dir <pipeline-config-directory>",
      "section_id": "step-4-switch-processors-and-resume-collection"
    },
    {
      "id": "step-6-tune-the-flink-processor-optional-ex0",
      "language": "yaml",
      "code": "processors:\n  type: flink\n  advanced:\n    source:\n      # Time between checks for new input streams.\n      discovery.interval.ms: 1000\n    flink:\n      # Number of parallel task slots per TaskManager pod.\n      taskmanager.numberOfTaskSlots: 2\n      # Total memory budget for each TaskManager JVM process.\n      taskmanager.memory.process.size: 4096m\n    resources:\n      taskManager:\n        # Number of TaskManager pods.\n        replicas: 2",
      "section_id": "step-6-tune-the-flink-processor-optional"
    }
  ]
}