{
  "schema_version": 2,
  "id": "develop/data-types/streams/idempotency",
  "title": "Idempotent message processing",
  "url": "https://redis.io/docs/latest/develop/data-types/streams/idempotency/",
  "summary": "Idempotent message processing in Redis Streams",
  "tags": [
    "docs",
    "develop",
    "stack",
    "oss",
    "rs",
    "rc",
    "oss",
    "kubernetes",
    "clients"
  ],
  "last_updated": "2026-09-29T16:29:37+01:00",
  "page_type": "content",
  "content_hash": "adb2ec23491ba43f0eb6192e850ec704ee9388e0bb7fab5b7e8fae073cd61acf",
  "sections": [
    {
      "id": "overview",
      "title": "Overview",
      "role": "overview",
      "text": "In Redis 8.6, streams support idempotent message processing (at-most-once production) to prevent duplicate entries when using at-least-once delivery patterns. This feature enables reliable message submission with automatic deduplication.\n\nIdempotent message processing ensures that handling the same message multiple times produces the same system state as handling it once.\n\nBeginning with Redis 8.6, streams support idempotent message processing (at-most-once production) to prevent duplicate entries when producers resend messages.\n\nProducers may need to resend messages under two scenarios:\n\n1. Producer-Redis network issues (disconnection and reconnection).\n\n    If a disconnection occurs after the producer executes `XADD`, but before it receives the reply, the producer has no way of knowing if that message was delivered.\n\n1. The producer crashes and restarts.\n\n    If the producer crashes after calling `XADD` but before receiving the reply and marking a message as delivered, after a restart, the producer has no way of knowing if that message was delivered.\n\nIn both cases, to guarantee that the message is added to the stream, the producer must call `XADD` again with the same message. Without idempotent message processing, a retry may result in a message being delivered twice. With idempotent message processing, producers can guarantee at-most-once production even under such scenarios.\n\nA unique ID, called an idempotent ID or *iid*, is associated with each message that is added to a stream. \nThere are two ways to assign iids:\n\n1. Producers provide a unique iid for each message. An iid can be some identifier already associated with this message: a transaction ID, a counter, or a UUID.\n1. Redis generates an iid based on each message’s content.\n\nIf the same message is added to the stream more than once, the same iid would need to be provided by the producer.\nFor (1), this is the producer’s responsibility, and for (2), Redis will calculate the same iid, as long as the message content hasn’t changed."
    },
    {
      "id": "idempotency-modes",
      "title": "Idempotency modes",
      "role": "content",
      "text": "Use the [`XADD`](https://redis.io/docs/latest/commands/xadd) command with idempotency parameters, `IDMP` or `IDMPAUTO`:\n\n[code example]"
    },
    {
      "id": "manual-mode-idmp",
      "title": "Manual mode (`IDMP`)",
      "role": "content",
      "text": "Specify both producer ID (pid) and iid explicitly:\n\n[code example]\n\n- `pid`: Unique identifier for the message producer.\n- `iid`: Unique identifier for a specific message.\n- Performance: Faster processing (no hash calculation).\n- Control: Full control over ID generation and uniqueness."
    },
    {
      "id": "automatic-mode-idmpauto",
      "title": "Automatic mode (`IDMPAUTO`)",
      "role": "content",
      "text": "Specify only the pid; Redis generates the iid from message content:\n\n[code example]\n\n- `pid`: Unique identifier for the message producer.\n- Automatic deduplication: Redis calculates an iid from field-value pairs.\n- Content-based: The same content produces the same iid.\n- Performance: Slightly slower due to hash calculation.\n\nFor both IDMP and IDMPAUTO, each producer application is required to use the same pid after it restarts.\n\nFor IDMP, each producer application is responsible for:\n\n- Providing a unique iid for each entry (either globally, or just for each pid).\n- Reusing the same (pid, iid) when resending a message (even after it restarts).\n\nHere's an illustration of how message processing in Redis Streams works with and without idempotent production:\n\n![Idempotent message processing in Redis Streams](https://redis.io/docs/latest/images/dev/stream/stream-idempotency.png)"
    },
    {
      "id": "stream-configuration",
      "title": "Stream configuration",
      "role": "content",
      "text": "Configure idempotency settings for a stream using [`XCFGSET`](https://redis.io/docs/latest/commands/xcfgset):\n\n[code example]"
    },
    {
      "id": "parameters",
      "title": "Parameters",
      "role": "parameters",
      "text": "- `IDMP-DURATION`: How long (in seconds) to retain iids (1-86400 seconds, the default is 100).\n- `IDMP-MAXSIZE`: The maximum number of per-producer iids to track (1-10,000 entries, the default is 100)."
    },
    {
      "id": "expiration-behavior",
      "title": "Expiration behavior",
      "role": "content",
      "text": "Idempotent IDs are removed when either condition is met:\n\n- Time-based: iids expire after the configured `IDMP-DURATION`.\n- Capacity-based: Oldest iids are evicted when `IDMP-MAXSIZE` is reached. Redis never keeps more than `IDMP-MAXSIZE` iids per pid. In other words, `IDMP-MAXSIZE` is *stronger* than `IDMP-DURATION`."
    },
    {
      "id": "determine-optimal-configuration-values",
      "title": "Determine optimal configuration values",
      "role": "content",
      "text": "`IDMP-DURATION` is an operational guarantee: Redis will not discard a previously sent iid for the specified duration (unless reaching `IDMP-MAXSIZE` iids for that producer).\nIf a producer application crashes and stops sending messages to Redis, Redis will keep each iid for `IDMP-DURATION` seconds, after which they will be discarded.\nYou should know how long it may take your producer application to recover from a crash and start resending messages, so you should set `IDMP-DURATION` accordingly.\nIf `IDMP-DURATION` is set too high, Redis will waste memory by retaining iids for a longer duration than necessary.\n\n**Example**: if a producer crashes, it may take up to 1,000 seconds until it recovers and restarts sending messages. You should set `IDMP-DURATION` to 1000.\n\nWhen a producer application retrieves an `XADD` reply from Redis, it usually marks the message as *delivered* in a transaction database or log file. \nIf the application crashes, it needs to resend undelivered messages after recovering from the crash.\nSince a few messages may have not been marked as delivered as a result of the crash, the application will likely resend these messages.\nUsing iids will allow Redis to detect such duplicate messages and filter them.\nSetting `IDMP-MAXSIZE` correctly ensures that Redis retains a sufficient number of recent iids.\nIf `IDMP-MAXSIZE` is set too high, Redis will waste memory by retaining too many iids.\nUsually this number can be very small, and often, even *one* is enough.\nIf your application marks messages as delivered asynchronously, you should know how long it may take from the time it retrieved a `XADD` reply from Redis until the message is marked as delivered; this duration is called *mark-delay*. `IDMP-MAXSIZE` should be set to\n\n`mark-delay [in msec] * (messages / msec) + some margin.`\n\n**Example**: a producer is sending 1K msgs/sec (1 msg/msec), and takes up to 80 msec to mark each message as delivered, `IDMP-MAXSIZE` should be set to `1 * 80 + margin = 100` iids."
    },
    {
      "id": "producer-isolation",
      "title": "Producer isolation",
      "role": "content",
      "text": "Each producer maintains independent idempotency tracking:\n\n[code example]\n\nProducers can use the same iid without conflicts, as long as long as the pids are different."
    },
    {
      "id": "monitoring",
      "title": "Monitoring",
      "role": "content",
      "text": "Use [`XINFO STREAM`](https://redis.io/docs/latest/commands/xinfo-stream) to monitor idempotency metrics:\n\n[code example]\n\nReturns additional fields when idempotency is being used:\n\n- `idmp-duration`: Current duration setting.\n- `idmp-maxsize`: Current maxsize setting.\n- `pids-tracked`: The number of pids currently tracked in the stream\n- `iids-tracked`: Total number of iids currently tracked.\n- `iids-added`: Lifetime count of messages with idempotent IDs.\n- `iids-duplicates`: Lifetime count of duplicates prevented."
    },
    {
      "id": "best-practices",
      "title": "Best practices",
      "role": "content",
      "text": ""
    },
    {
      "id": "producer-id-selection",
      "title": "Producer ID selection",
      "role": "content",
      "text": "Use globally unique, persistent producer IDs:\n\n- Recommended: Use short producer IDs to save memory and increase performance.\n- Persistence: Use the same producer ID after restarts to maintain idempotency tracking."
    },
    {
      "id": "configuration-tuning",
      "title": "Configuration tuning",
      "role": "configuration",
      "text": "- Duration: Set based on your retry timeout patterns.\n- Maxsize: Balance memory usage with deduplication window needs.\n- Monitoring: Track `iids-duplicates` to verify deduplication effectiveness."
    },
    {
      "id": "error-handling",
      "title": "Error handling",
      "role": "errors",
      "text": "Handle these error conditions:\n\n- `WRONGTYPE`: Key exists but is not a stream.\n- `ERR no such key`: Stream doesn't exist (when using NOMKSTREAM).\n- `ERR syntax error`: Invalid command syntax."
    },
    {
      "id": "performance-characteristics",
      "title": "Performance characteristics",
      "role": "performance",
      "text": "Idempotency introduces minimal overhead:\n\n- Throughput: 2-5% reduction compared to standard XADD.\n- Memory: <1.5% additional memory usage.\n- Latency: Negligible impact on per-operation latency.\n\nManual mode (IDMP) is slightly faster than automatic mode (IDMPAUTO) since it avoids hash calculations."
    },
    {
      "id": "persistence",
      "title": "Persistence",
      "role": "content",
      "text": "Idempotency tracking persists across Redis restarts:\n\n- RDB/AOF: All producer-idempotent ID pairs are saved.\n- Recovery: Tracking remains active after restart.\n- Configuration: `IDMP-DURATION` and `IDMP-MAXSIZE` settings persist.\n- Important: Executing `XCFGSET` with different `IDMP-DURATION` or `IDMP-MAXSIZE` values than the current values for a particular key clears its IDMP map."
    }
  ],
  "examples": [
    {
      "id": "idempotency-modes-ex0",
      "language": "plaintext",
      "code": "XADD mystream IDMP producer-1 iid-1 * field value      # producer-1 (pid) and iid-1 (iid) are provided manually\nXADD mystream IDMPAUTO producer-2 * field value        # producer-2 (pid) is provided manually, Redis provides the iid",
      "section_id": "idempotency-modes"
    },
    {
      "id": "manual-mode-idmp-ex0",
      "language": "plaintext",
      "code": "XADD mystream IDMP producer1 msg1 * field value",
      "section_id": "manual-mode-idmp"
    },
    {
      "id": "automatic-mode-idmpauto-ex0",
      "language": "plaintext",
      "code": "XADD mystream IDMPAUTO producer1 * field value",
      "section_id": "automatic-mode-idmpauto"
    },
    {
      "id": "stream-configuration-ex0",
      "language": "plaintext",
      "code": "XCFGSET mystream IDMP-DURATION 300 IDMP-MAXSIZE 1000",
      "section_id": "stream-configuration"
    },
    {
      "id": "producer-isolation-ex0",
      "language": "plaintext",
      "code": "XADD mystream IDMP producer-1 iid-1 * field value    # producer-1 is tracking\nXADD mystream IDMP producer-2 iid-1 * field value    # producer-2 is tracking (independent)",
      "section_id": "producer-isolation"
    },
    {
      "id": "monitoring-ex0",
      "language": "plaintext",
      "code": "XINFO STREAM mystream",
      "section_id": "monitoring"
    }
  ]
}