{ "schema_version": 2, "id": "develop/clients/patterns/bulk-loading", "title": "Bulk loading", "url": "https://redis.io/docs/latest/develop/clients/patterns/bulk-loading/", "summary": "Writing data in bulk using the Redis protocol", "aliases": [ "/develop/use/patterns/bulk-loading", "/manual/patterns/bulk-loading/" ], "tags": [ "docs", "develop", "stack", "oss", "rs", "rc", "oss", "kubernetes", "clients" ], "last_updated": "2026-09-09T10:55:35+01:00", "page_type": "content", "content_hash": "8ec4f7878d83160cfacf6caf366f07433faa8f72a3c4b24e780216e19ab23ff2", "sections": [ { "id": "overview", "title": "Overview", "role": "overview", "text": "Bulk loading is the process of loading Redis with a large amount of pre-existing data. Ideally, you want to perform this operation quickly and efficiently. This document describes some strategies for bulk loading data in Redis." }, { "id": "bulk-loading-using-the-redis-protocol", "title": "Bulk loading using the Redis protocol", "role": "content", "text": "Using a normal Redis client to perform bulk loading is not a good idea\nfor a few reasons: the naive approach of sending one command after the other\nis slow because you have to pay for the round trip time for every command.\nIt is possible to use pipelining, but for bulk loading of many records\nyou need to write new commands while you read replies at the same time to\nmake sure you are inserting as fast as possible.\n\nOnly a small percentage of clients support non-blocking I/O, and not all the\nclients are able to parse the replies in an efficient way in order to maximize\nthroughput. For all of these reasons the preferred way to mass import data into\nRedis is to generate a text file containing the Redis protocol, in raw format,\nin order to call the commands needed to insert the required data.\n\nFor instance if I need to generate a large data set where there are billions\nof keys in the form: `keyN -> ValueN' I will create a file containing the\nfollowing commands in the Redis protocol format:\n\n SET Key0 Value0\n SET Key1 Value1\n ...\n SET KeyN ValueN\n\nOnce this file is created, the remaining action is to feed it to Redis\nas fast as possible. In the past the way to do this was to use the\n`netcat` with the following command:\n\n (cat data.txt; sleep 10) | nc localhost 6379 > /dev/null\n\nHowever this is not a very reliable way to perform mass import because netcat\ndoes not really know when all the data was transferred and can't check for\nerrors. In 2.6 or later versions of Redis the `redis-cli` utility\nsupports a new mode called **pipe mode** that was designed in order to perform\nbulk loading.\n\nUsing the pipe mode the command to run looks like the following:\n\n cat data.txt | redis-cli --pipe\n\nThat will produce an output similar to this:\n\n All data transferred. Waiting for the last reply...\n Last reply received from server.\n errors: 0, replies: 1000000\n\nThe redis-cli utility will also make sure to only redirect errors received\nfrom the Redis instance to the standard output." }, { "id": "generating-redis-protocol", "title": "Generating Redis Protocol", "role": "content", "text": "The Redis protocol is extremely simple to generate and parse, and is\n[Documented here](https://redis.io/docs/latest/develop/reference/protocol-spec). However in order to generate protocol for\nthe goal of bulk loading you don't need to understand every detail of the\nprotocol, but just that every command is represented in the following way:\n\n *\n $\n \n \n ...\n \n\nWhere `` means \"\\r\" (or ASCII character 13) and `` means \"\\n\" (or ASCII character 10).\n\nFor instance the command **SET key value** is represented by the following protocol:\n\n *3\n $3\n SET\n $3\n key\n $5\n value\n\nOr represented as a quoted string:\n\n \"*3\\r\\n$3\\r\\nSET\\r\\n$3\\r\\nkey\\r\\n$5\\r\\nvalue\\r\\n\"\n\nThe file you need to generate for bulk loading is just composed of commands\nrepresented in the above way, one after the other.\n\nThe following Ruby function generates valid protocol:\n\n def gen_redis_proto(*cmd)\n proto = \"\"\n proto << \"*\"+cmd.length.to_s+\"\\r\\n\"\n cmd.each{|arg|\n proto << \"$\"+arg.to_s.bytesize.to_s+\"\\r\\n\"\n proto << arg.to_s+\"\\r\\n\"\n }\n proto\n end\n\n puts gen_redis_proto(\"SET\",\"mykey\",\"Hello World!\").inspect\n\nUsing the above function it is possible to easily generate the key value pairs\nin the above example, with this program:\n\n (0...1000).each{|n|\n STDOUT.write(gen_redis_proto(\"SET\",\"Key#{n}\",\"Value#{n}\"))\n }\n\nWe can run the program directly in pipe to redis-cli in order to perform our\nfirst mass import session.\n\n $ ruby proto.rb | redis-cli --pipe\n All data transferred. Waiting for the last reply...\n Last reply received from server.\n errors: 0, replies: 1000" }, { "id": "how-the-pipe-mode-works-under-the-hood", "title": "How the pipe mode works under the hood", "role": "content", "text": "The magic needed inside the pipe mode of redis-cli is to be as fast as netcat\nand still be able to understand when the last reply was sent by the server\nat the same time.\n\nThis is obtained in the following way:\n\n+ redis-cli --pipe tries to send data as fast as possible to the server.\n+ At the same time it reads data when available, trying to parse it.\n+ Once there is no more data to read from stdin, it sends a special **ECHO**\ncommand with a random 20 byte string: we are sure this is the latest command\nsent, and we are sure we can match the reply checking if we receive the same\n20 bytes as a bulk reply.\n+ Once this special final command is sent, the code receiving replies starts\nto match replies with these 20 bytes. When the matching reply is reached it\ncan exit with success.\n\nUsing this trick we don't need to parse the protocol we send to the server\nin order to understand how many commands we are sending, but just the replies.\n\nHowever while parsing the replies we take a counter of all the replies parsed\nso that at the end we are able to tell the user the amount of commands\ntransferred to the server by the mass insert session." } ], "examples": [] }