{
  "schema_version": 2,
  "id": "operate/rc/rdi/view-edit",
  "title": "View and edit data pipeline",
  "url": "https://redis.io/docs/latest/operate/rc/rdi/view-edit/",
  "summary": "Observe and change your data pipeline.",
  "aliases": [
    "/operate/rc/databases/rdi/view-edit/",
    "/operate/rc/databases/rdi/view-edit"
  ],
  "tags": [
    "docs",
    "operate",
    "rc"
  ],
  "last_updated": "2026-10-18T03:32:17+03:00",
  "page_type": "content",
  "content_hash": "77e658c4a86a33ff6a5ef3752c6755f18aca043a3ec360db6befeec9890ef1a2",
  "sections": [
    {
      "id": "overview",
      "title": "Overview",
      "role": "overview",
      "text": "To manage a pipeline, select it from your workspace on the **Data Integration** page or from the **Data Integration** tab of your subscription or database.\n\nThe pipeline page has the following tabs:\n\n- [Dashboard](#dashboard)\n- [Metrics](#metrics)\n- [Settings](#settings)\n- [Configuration](#configuration)\n- [Dataset](#dataset)\n- [Transformations](#transformations)\n\nUse the **Sources** list in **Metrics**, **Configuration**, and **Dataset** to select the source you want to inspect. **Settings** applies to the whole pipeline. **Transformations** lists jobs with their source assignments."
    },
    {
      "id": "dashboard",
      "title": "Dashboard",
      "role": "content",
      "text": "The **Dashboard** shows pipeline health, processor and target status, throughput, pending records, record health, and processor load. It also shows the number of sources, selected tables, and transformation jobs.\n\n![Pipeline dashboard with throughput, pending records, record health, and processor load.](https://redis.io/docs/latest/images/rc/rdi/rdi-2-dashboard.png)\n\nEach source has a card showing its database type, name, activity, and status. Expand a card for more details. Open the source's **More actions** menu to view its configuration, edit its dataset or transformations, or manage the source.\n\n![Three source cards: one PostgreSQL source and two MySQL sources, all streaming.](https://redis.io/docs/latest/images/rc/rdi/rdi-2-sources.png)\n\n![Source actions for configuration, dataset, transformations, stop, reset, and deletion.](https://redis.io/docs/latest/images/rc/rdi/rdi-2-source-actions.png)\n\nA pipeline can be streaming, stopped, or in an error or transitional state. If there is a problem, you can use the source cards and **Metrics** tab to identify the affected source before taking remedial action. Pipeline health and individual source status can differ, for example when one source is stopped."
    },
    {
      "id": "add-source",
      "title": "Add a source",
      "role": "content",
      "text": "1. On **Dashboard**, select **Add source**.\n1. In **Add sources**, select **Add source** in the **Sources** list, choose the new database type, and enter a unique source name. Existing sources remain listed with their connection details read-only.\n1. Complete **Configure source**, **Select data**, and **Add transformations** for the new source. See [Create data pipeline](https://redis.io/docs/latest/operate/rc/rdi/define) for the configuration steps.\n1. Review the changes in **Review & deploy** and select **Deploy pipeline**.\n\n![Add-source wizard showing existing sources and the Add source control.](https://redis.io/docs/latest/images/rc/rdi/rdi-2-add-sources.png)\n\nAdding a source keeps the existing pipeline's shared target and settings, so the wizard starts at **Add sources**. A source with an incomplete setup appears with **Pending setup** in the workspace. You can resume an existing draft setup at any time to complete it. If you remove a source, wait for the removal process to finish before adding another source.\n\nWhen extending an older single-source pipeline, preserve the original source name and its existing job assignments. You do not need to recreate the pipeline to add a source."
    },
    {
      "id": "stop-and-start-source",
      "title": "Stop and start a source",
      "role": "content",
      "text": "1. On **Dashboard**, open the source card's **More actions** menu.\n1. Select **Stop source**, then confirm with **Stop source**.\n\nStopping one source pauses data capture from that source without stopping the other sources. When you start a source again, its behavior depends on the source type:\n\n- Debezium sources follow their [snapshot mode](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config#choose-a-snapshot-mode) and the availability of their saved position.\n- Snowflake sources use RIOT-X and Snowflake Streams. They resume from the saved stream offset when the stream retains the required changes. If the retention window expires while the source is stopped, changes can be lost.\n\nTo recover a Snowflake source with an unusable saved position, set `clearOffset` to\n`true` in that source's **Collector source properties**, then apply the changes. In the\npipeline configuration, the property is\n`sources.<source-name>.advanced.riotx.clearOffset`. The Snowflake `snapshot` property\ncontrols whether the collector reloads existing rows:\n\n- `INITIAL` (default): Load current source rows, then capture new changes.\n- `NEVER`: Skip existing rows and capture only new changes.\n\nClearing the offset does not replay missed events or remove stale target records.\nAfter recovery, set `clearOffset` back to `false` so later collector starts preserve\nthe saved position. See [Edit collector properties](#edit-collector-properties).\n\nTo resume a source, open its **More actions** menu and select **Start source**, then confirm with **Start source**. A source processes data only while the pipeline is running. See [How does stopping sources affect billing?](https://redis.io/docs/latest/operate/rc/rdi/faq#stopping-and-billing)."
    },
    {
      "id": "reset-source",
      "title": "Reset a source",
      "role": "content",
      "text": "Resetting one source clears its saved position. With the default `initial` mode, a Debezium source takes a new snapshot and reprocesses its selected data. Other [snapshot modes](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config#choose-a-snapshot-mode) can skip existing rows or omit ongoing change capture.\n\n1. On **Dashboard**, open the source card's **More actions** menu.\n1. Select **Reset source**.\n1. Review the effect and confirm with **Reset source**.\n\n![Reset source confirmation explaining that other sources retain their data and the pipeline temporarily stops.](https://redis.io/docs/latest/images/rc/rdi/rdi-2-reset-source.png)\n\nThe reset clears that source's internal RDI streams, offsets, schema history, rejected records, and processing counters. The whole pipeline and all its sources restart during the reset. Other sources keep their saved positions. Debezium sources restart according to their snapshot modes; for example, `always` mode takes another snapshot even when its saved position is preserved. All records already in the shared target database remain, including those from the reset source. The new snapshot can overwrite records for that source. See [What happens when I reset one source?](https://redis.io/docs/latest/operate/rc/rdi/faq#reset-one-source)."
    },
    {
      "id": "remove-source",
      "title": "Remove a source",
      "role": "content",
      "text": "Before deleting a source, remove or reassign transformation jobs that refer to it and apply the changes. RDI rejects deletion while a transformation job still refers to the source.\n\n1. On **Dashboard**, open the source card's **More actions** menu.\n1. Select **Delete source**.\n1. Review the confirmation and select **Delete source**.\n\n![Delete source confirmation for the selected PostgreSQL source.](https://redis.io/docs/latest/images/rc/rdi/rdi-2-delete-source.png)\n\nDeleting a source removes its data selection and internal RDI state, including streams, offsets, schema history, rejected records, and processing counters. Records already written to the target Redis database remain there. Other sources retain their data. The whole pipeline stops while RDI cleans up the removed source and starts again afterwards.\n\nSource deletion cannot be undone. To remove its records from the target, see [Can I flush data for a single source?](https://redis.io/docs/latest/operate/rc/rdi/faq#flush-one-source)."
    },
    {
      "id": "change-target-database",
      "title": "Change target database",
      "role": "content",
      "text": "You can change the target to another database in the same subscription.\n\n1. On **Dashboard**, open the target card's **More actions** menu and select **Change pipeline target**.\n1. Select the new target database.\n1. Confirm with **Change target**.\n\nAll sources share the new target. Changing the target restarts the pipeline. Whether a Debezium source reloads existing rows depends on its [snapshot mode](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config#choose-a-snapshot-mode). To copy existing source data to the new target, select modes that take data snapshots, then [reset the pipeline](#reset-data-pipeline)."
    },
    {
      "id": "metrics",
      "title": "Metrics",
      "role": "content",
      "text": "Select a source in the **Sources** list to see its connection status, snapshot progress, queue usage, errors, and table-level record counts. Available collector diagnostics depend on the source type. Expand **Diagnostics** to see additional metrics for the selected source.\n\n![Metrics for the selected PostgreSQL source, including collector diagnostics and table-level counts.](https://redis.io/docs/latest/images/rc/rdi/rdi-2-metrics.png)\n\n| Metric | Description |\n|--------|-------------|\n| **Source table** | Name of the data stream. Each stream corresponds to a table from the source database.  |\n| **Total** | Total number of records that arrived from the source table. |\n| **Pending** | Number of records from the source table that are waiting to be processed. |\n| **Inserted** | Number of new records from the source table that have been written to the target database. |\n| **Updated** | Number of updated records from the source table that have been updated in the target database. |\n| **Deleted** | Number of deleted records from the source table that have been deleted in the target database. |\n| **Filtered** | Number of records from the source table that were filtered from being inserted into the target database. |\n| **Rejected** | Number of records from the source table that could not be parsed or inserted into the target database. Select a rejected count to open the [Rejected records](#rejected-records) view for that table. |"
    },
    {
      "id": "view-metrics-endpoints",
      "title": "View metrics endpoints",
      "role": "content",
      "text": "On **Metrics**, select **Connect to Prometheus** to view the available collector and processor endpoints. Add the required endpoints to your Prometheus configuration. Select the relevant source when inspecting per-source metrics.\n\nPrometheus endpoints are exposed on Redis Cloud's internal network. To access this network, enable [VPC peering](https://redis.io/docs/latest/operate/rc/security/vpc-peering) or [AWS Transit Gateway](https://redis.io/docs/latest/operate/rc/security/aws-transit-gateway). See [Prometheus and Grafana with Redis Cloud](https://redis.io/docs/latest/integrate/prometheus-with-redis-cloud) for more information.\n\n> [!NOTE]\n> VPC peering and AWS Transit Gateway are the tested and supported methods for accessing Prometheus endpoints. AWS PrivateLink support for this feature is under evaluation and is not currently supported.\n\nFor more information about available RDI metrics, see [Observability](https://redis.io/docs/latest/integrate/redis-data-integration/observability)."
    },
    {
      "id": "rejected-records",
      "title": "Rejected records",
      "role": "content",
      "text": "The **Rejected records** view shows records that RDI sent to the dead letter queue (DLQ) because processing failed. Open it from the rejected count on the **Dashboard** tab or from a table-level rejected count on the **Metrics** tab.\n\nThe view shows:\n\n- The total number of rejected records.\n- The number of affected tables.\n- The affected tables and their rejected counts.\n- Rejected record IDs and rejection times.\n- Safe troubleshooting metadata, such as the rejection reason, operation, affected table, and transformation job details when available. See [Using the operation code](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples/redis-opcode-example) for the operation labels.\n\nRedis Cloud uses the RDI DLQ API to show a sanitized view of rejected records. It\ndoes not show the original source record payload or every field stored in the\nDLQ stream. To inspect the full DLQ entry, connect to the RDI database and read\nthe corresponding DLQ stream directly.\n\nFor more information about why records are rejected and how RDI stores them, see [Rejected records](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/rejected-records)."
    },
    {
      "id": "settings",
      "title": "Settings",
      "role": "configuration",
      "text": "The **Settings** tab contains the default data structure (**Hash** or **JSON**) and advanced processor properties. These settings apply to all sources in the pipeline.\n\n![Pipeline-wide Settings tab with default data structure and processor properties.](https://redis.io/docs/latest/images/rc/rdi/rdi-2-settings.png)\n\nSelect **Edit** to change these settings. Select **Save changes**, then **Apply and restart**. Every Cloud RDI pipeline uses the Flink processor. Review [processor properties](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration) before changing them. To change processor capacity, see [Scale a Cloud RDI pipeline](https://redis.io/docs/latest/operate/rc/rdi/scale-pipeline).\n\n![The processor advanced properties editor with key and value fields.](https://redis.io/docs/latest/images/rc/rdi/rdi-processor-advanced-properties.png)"
    },
    {
      "id": "configuration",
      "title": "Configuration",
      "role": "configuration",
      "text": "Select a source in the **Sources** list to view its connectivity, secret references, and collector configuration. The connection details and secrets of a deployed source are read-only in this view."
    },
    {
      "id": "edit-collector-properties",
      "title": "Edit collector properties",
      "role": "content",
      "text": "1. Select the source in **Configuration**.\n1. Select **Edit collector properties**.\n1. Update the [collector source properties](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsource-advanced-source-settings) or [collector sink properties](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsink-rdi-collector-stream-writer-configuration) for that source.\n\n    ![The advanced properties dialog with separate collector source and sink properties.](https://redis.io/docs/latest/images/rc/rdi/rdi-advanced-properties.png)\n\n1. Save the properties and review the restart confirmation before applying the changes."
    },
    {
      "id": "dataset",
      "title": "Dataset",
      "role": "content",
      "text": "Select a source in the **Sources** list to view the schemas, tables, columns, and keys selected for ingestion from that source.\n\n![Dataset tab with a source selector and selected schema, tables, and columns.](https://redis.io/docs/latest/images/rc/rdi/rdi-2-dataset.png)"
    },
    {
      "id": "edit-dataset",
      "title": "Edit dataset",
      "role": "content",
      "text": "1. Select the source, then select **Edit**.\n\n    ![The Edit button.](https://redis.io/docs/latest/images/rc/rdi/rdi-view-edit-button.png)\n\n1. Select the schemas, tables, and columns to ingest. Review the record key for each selected table and correct any missing-key warnings.\n\n    ![Selecting a schema shows its tables for ingestion.](https://redis.io/docs/latest/images/rc/rdi/rdi-dataset-schema-selected.png)\n\n    ![Selecting a table shows its columns and the columns selected for ingestion.](https://redis.io/docs/latest/images/rc/rdi/rdi-select-columns.png)\n\n    ![The missing unique key warning with the affected table and its columns.](https://redis.io/docs/latest/images/rc/rdi/rdi-dataset-missing-unique-key.png)\n\n    ![The key control beside the column used to identify a record.](https://redis.io/docs/latest/images/rc/rdi/rdi-unique-key-selected.png)\n\n1. Select **Save changes**.\n1. Review the restart warning and select **Apply and restart**.\n\nDataset changes belong to the selected source. Applying changes restarts the pipeline and can temporarily interrupt processing. To reload existing rows, select a snapshot mode that takes a data snapshot, then use [Reset a source](#reset-source)."
    },
    {
      "id": "transformations",
      "title": "Transformations",
      "role": "content",
      "text": "The **Transformations** tab lists the pipeline's jobs, their source assignments, matching databases, schemas and tables, and validation status.\n\n![Transformation jobs assigned to three different sources, with verified status.](https://redis.io/docs/latest/images/rc/rdi/rdi-2-transformation-jobs.png)"
    },
    {
      "id": "edit-transformations",
      "title": "Edit transformations",
      "role": "content",
      "text": "1. Select **Edit** on **Transformations**.\n1. Add, upload, or edit the [transformation jobs](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/transform-examples) needed for your tables.\n1. Select the **Source name** for each job.\n1. Resolve job validation errors and select **Save changes**.\n1. Review the restart warning and select **Apply and restart**.\n\nThe Flink processor accepts lists and `regex:` entries in source selection patterns, but rejects jobs where more than one job can apply to a given table. The default data structure and processor properties are in [Settings](#settings)."
    },
    {
      "id": "reset-data-pipeline",
      "title": "Reset data pipeline",
      "role": "content",
      "text": "Resetting the whole pipeline clears its internal RDI state for all sources. A running pipeline restarts each source according to its configured initial data load settings. With the default `initial` mode, Debezium sources take new snapshots. Other [snapshot modes](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config#choose-a-snapshot-mode) can skip existing rows or omit ongoing change capture. A stopped pipeline remains stopped until you [start it](#stop-and-restart-data-pipeline). When a source takes a data snapshot, RDI reprocesses the selected data using the current transformations. Reset alone does not delete target records, and a new snapshot can [leave stale records after missed deletes](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config#missed-deletes). See [What happens when I reset the pipeline?](https://redis.io/docs/latest/operate/rc/rdi/faq#reset-pipeline).\n\n1. Open **Pipeline actions** and select **Reset pipeline**.\n1. Review the confirmation and select **Reset data pipeline**.\n\nTo clear the saved position of one source while preserving other sources' internal data, see [Reset a source](#reset-source)."
    },
    {
      "id": "stop-and-restart-data-pipeline",
      "title": "Stop and restart data pipeline",
      "role": "content",
      "text": "1. Open **Pipeline actions** and select **Stop pipeline**.\n1. Confirm with **Stop pipeline**.\n\nStopping the pipeline pauses processing for all sources. To resume, open **Pipeline actions**, select **Start pipeline**, and confirm with **Start pipeline**. Use the [source actions](#stop-and-start-source) to control an individual source separately. Stopping all sources individually leaves the processor running. See [Billing](https://redis.io/docs/latest/operate/rc/rdi/faq#billing) for the difference in charges."
    },
    {
      "id": "flush-the-target-database",
      "title": "Flush the target database",
      "role": "content",
      "text": "Flushing permanently deletes **all data** from the target database, including records from every source and data written outside RDI. It does not reset source positions or automatically reload the data.\n\n1. [Stop the pipeline](#stop-and-restart-data-pipeline) and wait until it is stopped.\n1. Open **Pipeline actions** and select **Flush target database**. This action is disabled while the pipeline is running.\n1. Check the target database and confirm with **Flush target database**.\n\nTo refill the target, follow [How do I reload data after a flush?](https://redis.io/docs/latest/operate/rc/rdi/faq#reload-after-flush)."
    },
    {
      "id": "delete-pipeline",
      "title": "Delete pipeline",
      "role": "content",
      "text": "> [!NOTE]\n> You cannot delete a pipeline while the target database's Redis Cloud cluster is\n> in maintenance mode. Wait until maintenance is complete, then delete the\n> pipeline.\n\n1. Return to the workspace list on the **Data Integration** page or your database's **Data Integration** tab.\n1. Open the pipeline's actions menu and select **Delete pipeline**.\n1. Review and confirm the deletion.\n\nDeleted pipelines cannot be recovered. If the pipeline has an associated setup draft, deletion also removes that draft. You can then [delete the workspace](https://redis.io/docs/latest/operate/rc/rdi/create-workspace#delete-workspace) when it is no longer needed."
    }
  ],
  "examples": []
}