# Prepare MongoDB for RDI

```json metadata
{
  "schema_version": 2,
  "title": "Prepare MongoDB for RDI",
  "description": "Prepare MongoDB databases to work with RDI",
  "categories": ["docs","integrate","rs","rdi"],
  "group": "di",
  "tableOfContents": {"sections":[{"id":"prerequisites","title":"Prerequisites"},{"id":"summary","title":"Summary"},{"id":"1-configure-oplog-size","title":"1. Configure oplog size"},{"id":"2-create-a-mongodb-user-for-rdi","title":"2. Create a MongoDB user for RDI"},{"id":"3-connection-string-format","title":"3. Connection string format"},{"id":"4-enable-pre-images-when-required","title":"4. Enable pre-images when required"},{"id":"5-mongodb-atlas-specific-requirements","title":"5. MongoDB Atlas specific requirements"},{"id":"6-self-hosted-mongodb-mtls-and-x509-authentication","title":"6. Self-hosted MongoDB mTLS and X.509 authentication"},{"id":"7-network-and-security","title":"7. Network and security"},{"id":"8-configuration-is-complete","title":"8. Configuration is complete"},{"id":"see-also","title":"See also"}]}

,
  "codeExamples": [{"id":"4-enable-change-streams-and-prepost-images-only-if-using-a-custom-key"}]
}
```


This guide describes the steps required to prepare a MongoDB database as a source for Redis Data Integration (RDI) pipelines.

## Prerequisites
- **MongoDB version:** 6.0 or later (replica set, sharded cluster, or MongoDB Atlas).
- **User privileges:** You must have a MongoDB user with sufficient privileges to read the oplog and collections, and to use change streams.
- **Network access:** The RDI Collector must be able to connect to all MongoDB nodes in your deployment.

> [!NOTE]
> The MongoDB connector is not capable of monitoring the changes of a standalone MongoDB server, since standalone servers do not have an oplog. The connector will work if the standalone server is converted to a replica set with one member.
## Summary

The following table summarizes the considerations to prepare a MongoDB database for RDI.

| Requirement         | Description                                                                 |
|---------------------|-----------------------------------------------------------------------------|
| MongoDB Topology    | Replica Set, Sharded Cluster, or MongoDB Atlas                              |
| User Roles          | readAnyDatabase, clusterMonitor                                             |
| Oplog               | Sufficient size for snapshot and streaming                                  |
| Pre/Post Images     | Enable when the pipeline needs old document values, including custom keys and set output types |
| Connection String   | Must include all hosts, replicaSet (if applicable), authSource, credentials |
| MongoDB Atlas       | **[SSL required](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-property-mongodb-ssl-enabled)**, provide root CA as the source's `CACERT` secret in RDI       |
| MongoDB mTLS        | X.509 authentication requires source TLS secrets and MongoDB SSL properties |
| Network             | RDI Collector must reach all MongoDB nodes on required ports                |

The following checklist shows the steps to prepare a MongoDB database for RDI,
with links to the sections that explain the steps in full detail.
You may find it helpful to track your progress with the checklist as you
complete each step.

```checklist {id="mongodblist"}
- [ ] [Configure oplog size](#1-configure-oplog-size)
- [ ] [Create a MongoDB user for RDI](#2-create-a-mongodb-user-for-rdi)
- [ ] [Connection string format](#3-connection-string-format)
- [ ] [Enable pre-images when required](#4-enable-pre-images-when-required)
- [ ] [MongoDB Atlas specific requirements](#5-mongodb-atlas-specific-requirements)
- [ ] [Self-hosted MongoDB mTLS and X.509 authentication](#6-self-hosted-mongodb-mtls-and-x509-authentication)
- [ ] [Network and security](#7-network-and-security)
```

## 1. Configure oplog size

The Debezium MongoDB connector relies on the [oplog](https://www.mongodb.com/docs/manual/core/replica-set-oplog/) to capture changes from a replica set. The oplog is a fixed-size, capped collection. When it reaches its maximum size, it overwrites the oldest entries. If the connector is stopped and restarted, it attempts to resume from its last recorded position in the oplog. If that position has been overwritten, the connector may fail to start and report an invalid resume token error.

To prevent this, ensure the oplog retains enough history for Debezium to resume streaming after interruptions. You can do this by:

- **Increasing the oplog size:** Set the oplog size based on your workload, ensuring it can store more than the peak number of oplog entries generated per hour.
- **Setting a minimum oplog retention period (MongoDB 4.4+):** Configure MongoDB to retain oplog entries for a minimum number of hours, guaranteeing availability even if the oplog reaches its maximum size. This is generally preferred, but for high-throughput clusters nearing capacity, you may need to increase the oplog size instead.

For detailed guidance, see the Debezium [oplog configuration documentation](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-optimal-oplog-config).

## 2. Create a MongoDB user for RDI

Create a user with the following roles on the source database:
- `readAnyDatabase` (optional) OR grant `read` for the specific database(s) you will use with RDI
- `clusterMonitor`

Example:
```javascript
use admin;
db.createUser({
  user: "rdi_user",
  pwd: "rdi_password",
  roles: [
     // You can have multiple read roles, one per database.
    { role: "read", db: "your_database" },
    // Use the role below if you don't want to grant the `read` role for each database.
    // { role: "readAnyDatabase", db: "admin" },
    { role: "clusterMonitor", db: "admin" }
  ]
});
```

## 3. Connection string format

The RDI Collector requires a MongoDB connection string that includes all relevant hosts and authentication details.

The credential references come from the source name, which is `mongodb` in the examples on
this page. See
[Multiple sources in one pipeline](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/multiple-sources)
for the source naming rules.

Example (Replica Set):
```
mongodb://${MONGODB_DB_USERNAME}:${MONGODB_DB_PASSWORD}@host1:27017,host2:27017,host3:27017/?replicaSet=rs0&authSource=admin
```
Example (Sharded Cluster):
```
mongodb://${MONGODB_DB_USERNAME}:${MONGODB_DB_PASSWORD}@host:30000
```
- For Atlas, adjust the connection string accordingly (see example below).
- Set `replicaSet` and `authSource` as appropriate for your deployment.

<a id="4-enable-change-streams-and-prepost-images-only-if-using-a-custom-key"></a>

## 4. Enable pre-images when required

RDI uses MongoDB change streams for ongoing replication, with both default and custom Redis keys. Change streams are available on replica sets, sharded clusters, and MongoDB Atlas.

A **pre-image** contains a document's values before an update or deletion. For example, if a custom Redis key uses an email address, changing that address requires RDI to remove the old Redis key and write the new one. The pre-image supplies the old address. A **post-image** contains the document after an insert or update; a deleted document has no post-image.

Jobs that use custom output keys or set output types require pre-images. Enable document pre- and post-images on the relevant collections with this command. It enables the extra document images, not change streams themselves:

```javascript
db.runCommand({
  collMod: "your_collection",
  changeStreamPreAndPostImages: { enabled: true }
});
```

Also set `capture.mode` in the source's advanced settings to a mode ending in `_with_pre_image`, such as `change_streams_update_full_with_pre_image`. Enabling images on the collection alone does not make the collector request them. See the [source configuration reference](https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsource-advanced-source-settings).

Keep required pre-images available for the lag and interruptions you need to tolerate. A change event can remain available after its pre-image is gone. See [MongoDB history and old values](https://redis.io/docs/latest/integrate/redis-data-integration/cdc-offsets-and-retention#mongodb-account-for-write-volume-and-old-values) and [MongoDB pre- and post-images](https://www.mongodb.com/docs/manual/changestreams/#change-streams-with-document-pre--and-post-images).

## 5. MongoDB Atlas specific requirements

MongoDB Atlas only supports secure connections via SSL.
The root CA certificate for MongoDB Atlas must be added as the source's `CACERT` secret in RDI.

- Download the MongoDB Atlas root CA certificate.
- In RDI, add this certificate with
  `redis-di set-secret CACERT --db mongodb /path/to/atlas-ca.crt`.
- Ensure that the `mongodb.ssl.enabled: true` setting is present in your RDI configuration for the corresponding source DB collector.

Example connection string for Atlas:
```
mongodb+srv://${MONGODB_DB_USERNAME}:${MONGODB_DB_PASSWORD}@cluster0.mongodb.net/?authSource=admin
```

## 6. Self-hosted MongoDB mTLS and X.509 authentication

For self-hosted MongoDB deployments that require TLS, set the source's `CACERT` secret to
the source CA certificate. For X.509 client certificate authentication, also set the
source's `CERT` and `KEY` secrets. See
[Set secrets](https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/deploy#set-secrets)
for the full list of source database TLS and mTLS secrets.

When you use MongoDB X.509 authentication, enable TLS in the source `advanced.source` section:

```yaml
advanced:
  source:
    mongodb.ssl.enabled: true
```

RDI builds the keystore that presents the client certificate to MongoDB from the source's
`CERT` and `KEY` secrets, and configures the collector to use it.

> [!NOTE]
> Do not set `mongodb.ssl.keystore` or `mongodb.ssl.keystore.password` yourself. RDI
> manages the keystore, so these properties are not only unnecessary, they are rejected when you
> deploy the pipeline.

For X.509 authentication, the MongoDB connection string must also include the
required authentication options, such as `authMechanism=MONGODB-X509` and
`authSource=%24external`.

## 7. Network and security

- Ensure the RDI Collector can connect to all MongoDB nodes on the required ports (default: 27017, or as provided by Atlas).
- If using TLS/SSL, provide the necessary certificates and connection options in the connection string.

## 8. Configuration is complete
Once you have followed the steps above, your MongoDB database is ready for Debezium to use.

## See also

- [MongoDB Replica Set Documentation](https://www.mongodb.com/docs/manual/replication/)
- [MongoDB Sharded Cluster Documentation](https://www.mongodb.com/docs/manual/sharding/)
- [MongoDB Change Streams](https://www.mongodb.com/docs/manual/changeStreams/)
- [MongoDB User Management](https://www.mongodb.com/docs/manual/core/security-users/)
- [Debezium MongoDB Connector Documentation](https://debezium.io/documentation/reference/stable/connectors/mongodb.html)
- [MongoDB Atlas SSL Setup](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-in-the-cloud)