{ "schema_version": 2, "id": "develop/clients/patterns/indexes/index", "title": "Secondary indexing", "url": "https://redis.io/docs/latest/develop/clients/patterns/indexes/", "summary": "Building secondary indexes in Redis", "aliases": [ "/develop/use/patterns/indexes", "/manual/patterns/indexes/" ], "tags": [ "docs", "develop", "stack", "oss", "rs", "rc", "oss", "kubernetes", "clients" ], "last_updated": "2026-09-09T10:55:35+01:00", "page_type": "content", "content_hash": "57329342f853ef3bbe23520cc7bfab860f0b963d1b7fa5d597b879117c03013f", "sections": [ { "id": "overview", "title": "Overview", "role": "overview", "text": "Redis is not exactly a key-value store, since values can be complex data structures. However it has an external key-value shell: at API level data is addressed by the key name. It is fair to say that, natively, Redis only offers *primary key access*. However since Redis is a data structures server, its capabilities can be used for indexing, in order to create secondary indexes of different kinds, including composite (multi-column) indexes.\n\nThis document explains how it is possible to create indexes in Redis using the following data structures:\n\n* Hashes and JSON documents, using a variety of field types; used in conjunction with Redis Search.\n* Sorted sets to create secondary indexes by ID or other numerical fields.\n* Sorted sets with lexicographical ranges for creating more advanced secondary indexes, composite indexes and graph traversal indexes.\n* Sets for creating random indexes.\n* Lists for creating simple iterable indexes and last N items indexes.\n* Time series with labels.\n\nImplementing and maintaining indexes with Redis is an advanced topic, so most\nusers that need to perform complex queries on data should understand if they\nare better served by a relational store. However often, especially in caching\nscenarios, there is the explicit need to store indexed data into Redis in order to speedup common queries which require some form of indexing in order to be executed." }, { "id": "hashes-and-json-indexes", "title": "Hashes and JSON indexes", "role": "content", "text": "Redis Search provides capabilities to index and query both hash and JSON keys using a variety of field types:\n\n* `TEXT`\n* `TAG`\n* `NUMERIC`\n* `GEO`\n* `VECTOR`\n* `GEOSHAPE`\n\nOnce hash or JSON keys have been indexed using the [`FT.CREATE`](https://redis.io/docs/latest/commands/ft.create) command, all keys that use the prefix defined in the index can be queried using the [`FT.SEARCH`](https://redis.io/docs/latest/commands/ft.search) and [`FT.AGGREGATE`](https://redis.io/docs/latest/commands/ft.aggregate) commands.\n\nFor more information on creating hash and JSON indexes, see the following pages.\n\n* [Hash indexes](https://redis.io/docs/latest/develop/ai/search-and-query/indexing/schema-definition)\n* [JSON indexes](https://redis.io/docs/latest/develop/ai/search-and-query/indexing)" }, { "id": "simple-numerical-indexes-with-sorted-sets", "title": "Simple numerical indexes with sorted sets", "role": "content", "text": "The simplest secondary index you can create with Redis is by using the\nsorted set data type, which is a data structure representing a set of\nelements ordered by a floating point number which is the *score* of\neach element. Elements are ordered from the smallest to the highest score.\n\nSince the score is a double precision float, indexes you can build with\nvanilla sorted sets are limited to things where the indexing field is a number\nwithin a given range.\n\nThe two commands to build these kind of indexes are [`ZADD`](https://redis.io/docs/latest/commands/zadd) and\n[`ZRANGE`](https://redis.io/docs/latest/commands/zrange) with the `BYSCORE` argument to respectively add items and retrieve items within a\nspecified range.\n\nFor instance, it is possible to index a set of person names by their\nage by adding element to a sorted set. The element will be the name of the\nperson and the score will be the age.\n\n ZADD myindex 25 Manuel\n ZADD myindex 18 Anna\n ZADD myindex 35 Jon\n ZADD myindex 67 Helen\n\nIn order to retrieve all persons with an age between 20 and 40, the following\ncommand can be used:\n\n ZRANGE myindex 20 40 BYSCORE\n 1) \"Manuel\"\n 2) \"Jon\"\n\nBy using the **WITHSCORES** option of [`ZRANGE`](https://redis.io/docs/latest/commands/zrange) it is also possible\nto obtain the scores associated with the returned elements.\n\nThe [`ZCOUNT`](https://redis.io/docs/latest/commands/zcount) command can be used in order to retrieve the number of elements\nwithin a given range, without actually fetching the elements, which is also\nuseful, especially given the fact the operation is executed in logarithmic\ntime regardless of the size of the range.\n\nRanges can be inclusive or exclusive, please refer to the [`ZRANGE`](https://redis.io/docs/latest/commands/zrange)\ncommand documentation for more information.\n\n**Note**: Using the [`ZRANGE`](https://redis.io/docs/latest/commands/zrange) with the `BYSCORE` and `REV` arguments, it is possible to query a range in\nreversed order, which is often useful when data is indexed in a given\ndirection (ascending or descending) but we want to retrieve information\nthe other way around." }, { "id": "use-object-ids-as-associated-values", "title": "Use object IDs as associated values", "role": "content", "text": "In the above example we associated names to ages. However in general we\nmay want to index some field of an object which is stored elsewhere.\nInstead of using the sorted set value directly to store the data associated\nwith the indexed field, it is possible to store just the ID of the object.\n\nFor example I may have Redis hashes representing users. Each user is\nrepresented by a single key, directly accessible by ID:\n\n HMSET user:1 id 1 username antirez ctime 1444809424 age 38\n HMSET user:2 id 2 username maria ctime 1444808132 age 42\n HMSET user:3 id 3 username jballard ctime 1443246218 age 33\n\nIf I want to create an index in order to query users by their age, I\ncould do:\n\n ZADD user.age.index 38 1\n ZADD user.age.index 42 2\n ZADD user.age.index 33 3\n\nThis time the value associated with the score in the sorted set is the\nID of the object. So once I query the index with [`ZRANGE`](https://redis.io/docs/latest/commands/zrange) with the `BYSCORE` argument, I'll\nalso have to retrieve the information I need with [`HGETALL`](https://redis.io/docs/latest/commands/hgetall) or similar\ncommands. The obvious advantage is that objects can change without touching\nthe index, as long as we don't change the indexed field.\n\nIn the next examples we'll almost always use IDs as values associated with\nthe index, since this is usually the more sounding design, with a few\nexceptions." }, { "id": "update-simple-sorted-set-indexes", "title": "Update simple sorted set indexes", "role": "content", "text": "Often we index things which change over time. In the above\nexample, the age of the user changes every year. In such a case it would\nmake sense to use the birth date as index instead of the age itself,\nbut there are other cases where we simply want some field to change from\ntime to time, and the index to reflect this change.\n\nThe [`ZADD`](https://redis.io/docs/latest/commands/zadd) command makes updating simple indexes a very trivial operation\nsince re-adding back an element with a different score and the same value\nwill simply update the score and move the element at the right position,\nso if the user `antirez` turned 39 years old, in order to update the\ndata in the hash representing the user, and in the index as well, we need\nto execute the following two commands:\n\n HSET user:1 age 39\n ZADD user.age.index 39 1\n\nThe operation may be wrapped in a [`MULTI`](https://redis.io/docs/latest/commands/multi)/[`EXEC`](https://redis.io/docs/latest/commands/exec) transaction in order to\nmake sure both fields are updated or none." }, { "id": "turn-multi-dimensional-data-into-linear-data", "title": "Turn multi-dimensional data into linear data", "role": "content", "text": "Indexes created with sorted sets are able to index only a single numerical\nvalue. Because of this you may think it is impossible to index something\nwhich has multiple dimensions using this kind of indexes, but actually this\nis not always true. If you can efficiently represent something\nmulti-dimensional in a linear way, they it is often possible to use a simple\nsorted set for indexing.\n\nFor example the [Redis geo indexing API](https://redis.io/docs/latest/commands/geoadd) uses a sorted\nset to index places by latitude and longitude using a technique called\n[Geo hash](https://en.wikipedia.org/wiki/Geohash). The sorted set score\nrepresents alternating bits of longitude and latitude, so that we map the\nlinear score of a sorted set to many small *squares* in the earth surface.\nBy doing an 8+1 style center plus neighborhoods search it is possible to\nretrieve elements by radius." }, { "id": "limits-of-the-score", "title": "Limits of the score", "role": "limits", "text": "Sorted set elements scores are double precision floats. It means that\nthey can represent different decimal or integer values with different\nerrors, because they use an exponential representation internally.\nHowever what is interesting for indexing purposes is that the score is\nalways able to represent without any error numbers between -9007199254740992\nand 9007199254740992, which is `-/+ 2^53`.\n\nWhen representing much larger numbers, you need a different form of indexing\nthat is able to index numbers at any precision, called a lexicographical\nindex." }, { "id": "time-series-indexes", "title": "Time series indexes", "role": "content", "text": "When you create a new time series using the [`TS.CREATE`](https://redis.io/docs/latest/commands/ts.create) command, you can associate one or more `LABELS` with it. Each label is a name-value pair, where the both name and value are text. Labels serve as a secondary index that allows you to execute queries on groups of time series keys using various time series commands.\n\nSee the [time series quickstart guide](https://redis.io/docs/latest/develop/data-types/timeseries#create-a-time-series) for an example of creating a time series with a label.\n\nThe [`TS.MGET`](https://redis.io/docs/latest/commands/ts.mget), [`TS.MRANGE`](https://redis.io/docs/latest/commands/ts.mrange), and [`TS.MREVRANGE`](https://redis.io/docs/latest/commands/ts.mrevrange) commands operate on multiple time series based on specified labels or using a label-related filter expression. The [`TS.QUERYINDEX`](https://redis.io/docs/latest/commands/ts.queryindex) command returns all time series keys matching a given label-related filter expression." }, { "id": "lexicographical-indexes", "title": "Lexicographical indexes", "role": "content", "text": "Redis sorted sets have an interesting property. When elements are added\nwith the same score, they are sorted lexicographically, comparing the\nstrings as binary data with the `memcmp()` function.\n\nFor people that don't know the C language nor the `memcmp` function, what\nit means is that elements with the same score are sorted comparing the\nraw values of their bytes, byte after byte. If the first byte is the same,\nthe second is checked and so forth. If the common prefix of two strings is\nthe same then the longer string is considered the greater of the two,\nso \"foobar\" is greater than \"foo\".\n\nThere are commands such as [`ZRANGE`](https://redis.io/docs/latest/commands/zrange) and [`ZLEXCOUNT`](https://redis.io/docs/latest/commands/zlexcount) that\nare able to query and count ranges in a lexicographically fashion, assuming\nthey are used with sorted sets where all the elements have the same score.\n\nThis Redis feature is basically equivalent to a `b-tree` data structure which\nis often used in order to implement indexes with traditional databases.\nAs you can guess, because of this, it is possible to use this Redis data\nstructure in order to implement pretty fancy indexes.\n\nBefore we dive into using lexicographical indexes, let's check how\nsorted sets behave in this special mode of operation. Since we need to\nadd elements with the same score, we'll always use the special score of\nzero.\n\n ZADD myindex 0 baaa\n ZADD myindex 0 abbb\n ZADD myindex 0 aaaa\n ZADD myindex 0 bbbb\n\nFetching all the elements from the sorted set immediately reveals that they\nare ordered lexicographically.\n\n ZRANGE myindex 0 -1\n 1) \"aaaa\"\n 2) \"abbb\"\n 3) \"baaa\"\n 4) \"bbbb\"\n\nNow we can use [`ZRANGE`](https://redis.io/docs/latest/commands/zrange) with the `BYLEX` argument in order to perform range queries.\n\n ZRANGE myindex [a (b BYLEX\n 1) \"aaaa\"\n 2) \"abbb\"\n\nNote that in the range queries we prefixed the `min` and `max` elements\nidentifying the range with the special characters `[` and `(`.\nThis prefixes are mandatory, and they specify if the elements\nof the range are inclusive or exclusive. So the range `[a (b` means give me\nall the elements lexicographically between `a` inclusive and `b` exclusive,\nwhich are all the elements starting with `a`.\n\nThere are also two more special characters indicating the infinitely negative\nstring and the infinitely positive string, which are `-` and `+`.\n\n ZRANGE myindex [b + BYLEX\n 1) \"baaa\"\n 2) \"bbbb\"\n\nThat's it basically. Let's see how to use these features to build indexes." }, { "id": "a-first-example-completion", "title": "A first example: completion", "role": "content", "text": "An interesting application of indexing is completion. Completion is what\nhappens when you start typing your query into a search engine: the user\ninterface will anticipate what you are likely typing, providing common\nqueries that start with the same characters.\n\nA naive approach to completion is to just add every single query we\nget from the user into the index. For example if the user searches `banana`\nwe'll just do:\n\n ZADD myindex 0 banana\n\nAnd so forth for each search query ever encountered. Then when we want to\ncomplete the user input, we execute a range query using [`ZRANGE`](https://redis.io/docs/latest/commands/zrange) with the `BYLEX` argument.\nImagine the user is typing \"bit\" inside the search form, and we want to\noffer possible search keywords starting for \"bit\". We send Redis a command\nlike that:\n\n ZRANGE myindex \"[bit\" \"[bit\\xff\" BYLEX\n\nBasically we create a range using the string the user is typing right now\nas start, and the same string plus a trailing byte set to 255, which is `\\xff` in the example, as the end of the range. This way we get all the strings that start for the string the user is typing.\n\nNote that we don't want too many items returned, so we may use the **LIMIT** option in order to reduce the number of results." }, { "id": "add-frequency-into-the-mix", "title": "Add frequency into the mix", "role": "content", "text": "The above approach is a bit naive, because all the user searches are the same\nin this way. In a real system we want to complete strings according to their\nfrequency: very popular searches will be proposed with a higher probability\ncompared to search strings typed very rarely.\n\nIn order to implement something which depends on the frequency, and at the\nsame time automatically adapts to future inputs, by purging searches that\nare no longer popular, we can use a very simple *streaming algorithm*.\n\nTo start, we modify our index in order to store not just the search term,\nbut also the frequency the term is associated with. So instead of just adding\n`banana` we add `banana:1`, where 1 is the frequency.\n\n ZADD myindex 0 banana:1\n\nWe also need logic in order to increment the index if the search term\nalready exists in the index, so what we'll actually do is something like\nthat:\n\n ZRANGE myindex \"[banana:\" + BYLEX LIMIT 0 1\n 1) \"banana:1\"\n\nThis will return the single entry of `banana` if it exists. Then we\ncan increment the associated frequency and send the following two\ncommands:\n\n ZREM myindex 0 banana:1\n ZADD myindex 0 banana:2\n\nNote that because it is possible that there are concurrent updates, the\nabove three commands should be send via a [Lua script](https://redis.io/docs/latest/commands/eval)\ninstead, so that the Lua script will atomically get the old count and\nre-add the item with incremented score.\n\nSo the result will be that, every time a user searches for `banana` we'll\nget our entry updated.\n\nThere is more: our goal is to just have items searched very frequently.\nSo we need some form of purging. When we actually query the index\nin order to complete the user input, we may see something like that:\n\n ZRANGE myindex \"[banana:\" + BYLEX LIMIT 0 10\n 1) \"banana:123\"\n 2) \"banaooo:1\"\n 3) \"banned user:49\"\n 4) \"banning:89\"\n\nApparently nobody searches for \"banaooo\", for example, but the query was\nperformed a single time, so we end presenting it to the user.\n\nThis is what we can do. Out of the returned items, we pick a random one,\ndecrement its score by one, and re-add it with the new score.\nHowever if the score reaches 0, we simply remove the item from the list.\nYou can use much more advanced systems, but the idea is that the index in\nthe long run will contain top searches, and if top searches will change over\nthe time it will adapt automatically.\n\nA refinement to this algorithm is to pick entries in the list according to\ntheir weight: the higher the score, the less likely entries are picked\nin order to decrement its score, or evict them." }, { "id": "normalize-strings-for-case-and-accents", "title": "Normalize strings for case and accents", "role": "content", "text": "In the completion examples we always used lowercase strings. However\nreality is much more complex than that: languages have capitalized names,\naccents, and so forth.\n\nOne simple way do deal with this issues is to actually normalize the\nstring the user searches. Whatever the user searches for \"Banana\",\n\"BANANA\" or \"Ba'nana\" we may always turn it into \"banana\".\n\nHowever sometimes we may like to present the user with the original\nitem typed, even if we normalize the string for indexing. In order to\ndo this, what we do is to change the format of the index so that instead\nof just storing `term:frequency` we store `normalized:frequency:original`\nlike in the following example:\n\n ZADD myindex 0 banana:273:Banana\n\nBasically we add another field that we'll extract and use only for\nvisualization. Ranges will always be computed using the normalized strings\ninstead. This is a common trick which has multiple applications." }, { "id": "add-auxiliary-information-in-the-index", "title": "Add auxiliary information in the index", "role": "content", "text": "When using a sorted set in a direct way, we have two different attributes\nfor each object: the score, which we use as an index, and an associated\nvalue. When using lexicographical indexes instead, the score is always\nset to 0 and basically not used at all. We are left with a single string,\nwhich is the element itself.\n\nLike we did in the previous completion examples, we are still able to\nstore associated data using separators. For example we used the colon in\norder to add the frequency and the original word for completion.\n\nIn general we can add any kind of associated value to our indexing key.\nIn order to use a lexicographical index to implement a simple key-value store\nwe just store the entry as `key:value`:\n\n ZADD myindex 0 mykey:myvalue\n\nAnd search for the key with:\n\n ZRANGE myindex [mykey: + BYLEX LIMIT 0 1\n 1) \"mykey:myvalue\"\n\nThen we extract the part after the colon to retrieve the value.\nHowever a problem to solve in this case is collisions. The colon character\nmay be part of the key itself, so it must be chosen in order to never\ncollide with the key we add.\n\nSince lexicographical ranges in Redis are binary safe you can use any\nbyte or any sequence of bytes. However if you receive untrusted user\ninput, it is better to use some form of escaping in order to guarantee\nthat the separator will never happen to be part of the key.\n\nFor example if you use two null bytes as separator `\"\\0\\0\"`, you may\nwant to always escape null bytes into two bytes sequences in your strings." }, { "id": "numerical-padding", "title": "Numerical padding", "role": "content", "text": "Lexicographical indexes may look like good only when the problem at hand\nis to index strings. Actually it is very simple to use this kind of index\nin order to perform indexing of arbitrary precision numbers.\n\nIn the ASCII character set, digits appear in the order from 0 to 9, so\nif we left-pad numbers with leading zeroes, the result is that comparing\nthem as strings will order them by their numerical value.\n\n ZADD myindex 0 00324823481:foo\n ZADD myindex 0 12838349234:bar\n ZADD myindex 0 00000000111:zap\n\n ZRANGE myindex 0 -1\n 1) \"00000000111:zap\"\n 2) \"00324823481:foo\"\n 3) \"12838349234:bar\"\n\nWe effectively created an index using a numerical field which can be as\nbig as we want. This also works with floating point numbers of any precision\nby making sure we left pad the numerical part with leading zeroes and the\ndecimal part with trailing zeroes like in the following list of numbers:\n\n 01000000000000.11000000000000\n 01000000000000.02200000000000\n 00000002121241.34893482930000\n 00999999999999.00000000000000" }, { "id": "use-numbers-in-binary-form", "title": "Use numbers in binary form", "role": "content", "text": "Storing numbers in decimal may use too much memory. An alternative approach\nis just to store numbers, for example 128 bit integers, directly in their\nbinary form. However for this to work, you need to store the numbers in\n*big endian format*, so that the most significant bytes are stored before\nthe least significant bytes. This way when Redis compares the strings with\n`memcmp()`, it will effectively sort the numbers by their value.\n\nKeep in mind that data stored in binary format is less observable for\ndebugging, harder to parse and export. So it is definitely a trade off." }, { "id": "composite-indexes", "title": "Composite indexes", "role": "content", "text": "So far we explored ways to index single fields. However we all know that\nSQL stores are able to create indexes using multiple fields. For example\nI may index products in a very large store by room number and price.\n\nI need to run queries in order to retrieve all the products in a given\nroom having a given price range. What I can do is to index each product\nin the following way:\n\n ZADD myindex 0 0056:0028.44:90\n ZADD myindex 0 0034:0011.00:832\n\nHere the fields are `room:price:product_id`. I used just four digits padding\nin the example for simplicity. The auxiliary data (the product ID) does not\nneed any padding.\n\nWith an index like that, to get all the products in room 56 having a price\nbetween 10 and 30 dollars is very easy. We can just run the following\ncommand:\n\n ZRANGE myindex [0056:0010.00 [0056:0030.00 BYLEX\n\nThe above is called a composed index. Its effectiveness depends on the\norder of the fields and the queries I want to run. For example the above\nindex cannot be used efficiently in order to get all the products having\na specific price range regardless of the room number. However I can use\nthe primary key in order to run queries regardless of the price, like\n*give me all the products in room 44*.\n\nComposite indexes are very powerful, and are used in traditional stores\nin order to optimize complex queries. In Redis they could be useful both\nto implement a very fast in-memory Redis index of something stored into\na traditional data store, or in order to directly index Redis data." }, { "id": "update-lexicographical-indexes", "title": "Update lexicographical indexes", "role": "content", "text": "The value of the index in a lexicographical index can get pretty fancy\nand hard or slow to rebuild from what we store about the object. So one\napproach to simplify the handling of the index, at the cost of using more\nmemory, is to also take alongside to the sorted set representing the index\na hash mapping the object ID to the current index value.\n\nSo for example, when we index we also add to a hash:\n\n MULTI\n ZADD myindex 0 0056:0028.44:90\n HSET index.content 90 0056:0028.44:90\n EXEC\n\nThis is not always needed, but simplifies the operations of updating\nthe index. In order to remove the old information we indexed for the object\nID 90, regardless of the *current* fields values of the object, we just\nhave to retrieve the hash value by object ID and [`ZREM`](https://redis.io/docs/latest/commands/zrem) it in the sorted\nset view." }, { "id": "represent-and-query-graphs-using-a-hexastore", "title": "Represent and query graphs using a hexastore", "role": "content", "text": "One cool thing about composite indexes is that they are handy in order\nto represent graphs, using a data structure which is called\n[Hexastore](http://www.vldb.org/pvldb/vol1/1453965.pdf).\n\nThe hexastore provides a representation for relations between objects,\nformed by a *subject*, a *predicate* and an *object*.\nA simple relation between objects could be:\n\n antirez is-friend-of matteocollina\n\nIn order to represent this relation I can store the following element\nin my lexicographical index:\n\n ZADD myindex 0 spo:antirez:is-friend-of:matteocollina\n\nNote that I prefixed my item with the string **spo**. It means that\nthe item represents a subject,predicate,object relation.\n\nIn can add 5 more entries for the same relation, but in a different order:\n\n ZADD myindex 0 sop:antirez:matteocollina:is-friend-of\n ZADD myindex 0 ops:matteocollina:is-friend-of:antirez\n ZADD myindex 0 osp:matteocollina:antirez:is-friend-of\n ZADD myindex 0 pso:is-friend-of:antirez:matteocollina\n ZADD myindex 0 pos:is-friend-of:matteocollina:antirez\n\nNow things start to be interesting, and I can query the graph in many\ndifferent ways. For example, who are all the people `antirez`\n*is friend of*?\n\n ZRANGE myindex \"[spo:antirez:is-friend-of:\" \"[spo:antirez:is-friend-of:\\xff\" BYLEX\n 1) \"spo:antirez:is-friend-of:matteocollina\"\n 2) \"spo:antirez:is-friend-of:wonderwoman\"\n 3) \"spo:antirez:is-friend-of:spiderman\"\n\nOr, what are all the relationships `antirez` and `matteocollina` have where\nthe first is the subject and the second is the object?\n\n ZRANGE myindex \"[sop:antirez:matteocollina:\" \"[sop:antirez:matteocollina:\\xff\" BYLEX\n 1) \"sop:antirez:matteocollina:is-friend-of\"\n 2) \"sop:antirez:matteocollina:was-at-conference-with\"\n 3) \"sop:antirez:matteocollina:talked-with\"\n\nBy combining different queries, I can ask fancy questions. For example:\n*Who are all my friends that, like beer, live in Barcelona, and matteocollina consider friends as well?*\nTo get this information I start with an `spo` query to find all the people\nI'm friend with. Then for each result I get I perform an `spo` query\nto check if they like beer, removing the ones for which I can't find\nthis relation. I do it again to filter by city. Finally I perform an `ops`\nquery to find, of the list I obtained, who is considered friend by\nmatteocollina.\n\nMake sure to check [Matteo Collina's slides about Levelgraph](http://nodejsconfit.levelgraph.io/) in order to better understand these ideas." }, { "id": "multi-dimensional-indexes", "title": "Multi-dimensional indexes", "role": "content", "text": "A more complex type of index is an index that allows you to perform queries\nwhere two or more variables are queried at the same time for specific\nranges. For example I may have a data set representing persons age and\nsalary, and I want to retrieve all the people between 50 and 55 years old\nhaving a salary between 70000 and 85000.\n\nThis query may be performed with a multi column index, but this requires\nus to select the first variable and then scan the second, which means we\nmay do a lot more work than needed. It is possible to perform these kinds of\nqueries involving multiple variables using different data structures.\nFor example, multi-dimensional trees such as *k-d trees* or *r-trees* are\nsometimes used. Here we'll describe a different way to index data into\nmultiple dimensions, using a representation trick that allows us to perform\nthe query in a very efficient way using Redis lexicographical ranges.\n\nLet's say we have points in the space, which represent our data samples, where `x` and `y` are our coordinates. The max value of both variables is 400.\n\nIn the next figure, the blue box represents our query. We want all the points where `x` is between 50 and 100, and where `y` is between 100 and 300.\n\n![Points in the space](2idx_0.png)\n\nIn order to represent data that makes these kinds of queries fast to perform,\nwe start by padding our numbers with 0. So for example imagine we want to\nadd the point 10,25 (x,y) to our index. Given that the maximum range in the\nexample is 400 we can just pad to three digits, so we obtain:\n\n x = 010\n y = 025\n\nNow what we do is to interleave the digits, taking the leftmost digit\nin x, and the leftmost digit in y, and so forth, in order to create a single\nnumber:\n\n 001205\n\nThis is our index, however in order to more easily reconstruct the original\nrepresentation, if we want (at the cost of space), we may also add the\noriginal values as additional columns:\n\n 001205:10:25\n\nNow, let's reason about this representation and why it is useful in the\ncontext of range queries. For example let's take the center of our blue\nbox, which is at `x=75` and `y=200`. We can encode this number as we did\nearlier by interleaving the digits, obtaining:\n\n 027050\n\nWhat happens if we substitute the last two digits respectively with 00 and 99?\nWe obtain a range which is lexicographically continuous:\n\n 027000 to 027099\n\nWhat this maps to is to a square representing all values where the `x`\nvariable is between 70 and 79, and the `y` variable is between 200 and 209.\nTo identify this specific area, we can write random points in that interval.\n\n![Small area](2idx_1.png)\n\nSo the above lexicographic query allows us to easily query for points in\na specific square in the picture. However the square may be too small for\nthe box we are searching, so that too many queries are needed.\nSo we can do the same but instead of replacing the last two digits with 00\nand 99, we can do it for the last four digits, obtaining the following\nrange:\n\n 020000 029999\n\nThis time the range represents all the points where `x` is between 0 and 99\nand `y` is between 200 and 299. Drawing random points in this interval\nshows us this larger area.\n\n![Large area](2idx_2.png)\n\nSo now our area is too big for our query, and still our search box is\nnot completely included. We need more granularity, but we can easily obtain\nit by representing our numbers in binary form. This time, when we replace\ndigits instead of getting squares which are ten times bigger, we get squares\nwhich are just two times bigger.\n\nOur numbers in binary form, assuming we need just 9 bits for each variable\n(in order to represent numbers up to 400 in value) would be:\n\n x = 75 -> 001001011\n y = 200 -> 011001000\n\nSo by interleaving digits, our representation in the index would be:\n\n 000111000011001010:75:200\n\nLet's see what are our ranges as we substitute the last 2, 4, 6, 8, ...\nbits with 0s ad 1s in the interleaved representation:\n\n 2 bits: x between 74 and 75, y between 200 and 201 (range=2)\n 4 bits: x between 72 and 75, y between 200 and 203 (range=4)\n 6 bits: x between 72 and 79, y between 200 and 207 (range=8)\n 8 bits: x between 64 and 79, y between 192 and 207 (range=16)\n\nAnd so forth. Now we have definitely better granularity!\nAs you can see substituting N bits from the index gives us\nsearch boxes of side `2^(N/2)`.\n\nSo what we do is check the dimension where our search box is smaller,\nand check the nearest power of two to this number. Our search box\nwas 50,100 to 100,300, so it has a width of 50 and a height of 200.\nWe take the smaller of the two, 50, and check the nearest power of two\nwhich is 64. 64 is 2^6, so we would work with indexes obtained replacing\nthe latest 12 bits from the interleaved representation (so that we end\nreplacing just 6 bits of each variable).\n\nHowever single squares may not cover all our search, so we may need more.\nWhat we do is to start with the left bottom corner of our search box,\nwhich is 50,100, and find the first range by substituting the last 6 bits\nin each number with 0. Then we do the same with the right top corner.\n\nWith two trivial nested for loops where we increment only the significant\nbits, we can find all the squares between these two. For each square we\nconvert the two numbers into our interleaved representation, and create\nthe range using the converted representation as our start, and the same\nrepresentation but with the latest 12 bits turned on as end range.\n\nFor each square found we perform our query and get the elements inside,\nremoving the elements which are outside our search box.\n\nTurning this into code is simple. Here is a Ruby example:\n\n def spacequery(x0,y0,x1,y1,exp)\n bits=exp*2\n x_start = x0/(2**exp)\n x_end = x1/(2**exp)\n y_start = y0/(2**exp)\n y_end = y1/(2**exp)\n (x_start..x_end).each{|x|\n (y_start..y_end).each{|y|\n x_range_start = x*(2**exp)\n x_range_end = x_range_start | ((2**exp)-1)\n y_range_start = y*(2**exp)\n y_range_end = y_range_start | ((2**exp)-1)\n puts \"#{x},#{y} x from #{x_range_start} to #{x_range_end}, y from #{y_range_start} to #{y_range_end}\"\n\n # Turn it into interleaved form for ZRANGE query.\n # We assume we need 9 bits for each integer, so the final\n # interleaved representation will be 18 bits.\n xbin = x_range_start.to_s(2).rjust(9,'0')\n ybin = y_range_start.to_s(2).rjust(9,'0')\n s = xbin.split(\"\").zip(ybin.split(\"\")).flatten.compact.join(\"\")\n # Now that we have the start of the range, calculate the end\n # by replacing the specified number of bits from 0 to 1.\n e = s[0..-(bits+1)]+(\"1\"*bits)\n puts \"ZRANGE myindex [#{s} [#{e} BYLEX\"\n }\n }\n end\n\n spacequery(50,100,100,300,6)\n\nWhile non immediately trivial this is a very useful indexing strategy that\nin the future may be implemented in Redis in a native way.\nFor now, the good thing is that the complexity may be easily encapsulated\ninside a library that can be used in order to perform indexing and queries.\nOne example of such library is [Redimension](https://github.com/antirez/redimension), a proof of concept Ruby library which indexes N-dimensional data inside Redis using the technique described here." }, { "id": "multi-dimensional-indexes-with-negative-or-floating-point-numbers", "title": "Multi-dimensional indexes with negative or floating point numbers", "role": "content", "text": "The simplest way to represent negative values is just to work with unsigned\nintegers and represent them using an offset, so that when you index, before\ntranslating numbers in the indexed representation, you add the absolute value\nof your smaller negative integer.\n\nFor floating point numbers, the simplest approach is probably to convert them\nto integers by multiplying the integer for a power of ten proportional to the\nnumber of digits after the dot you want to retain." }, { "id": "non-range-indexes", "title": "Non-range indexes", "role": "content", "text": "So far we checked indexes which are useful to query by range or by single\nitem. However other Redis data structures such as Sets or Lists can be used\nin order to build other kind of indexes. They are very commonly used but\nmaybe we don't always realize they are actually a form of indexing.\n\nFor instance I can index object IDs into a Set data type in order to use\nthe *get random elements* operation via [`SRANDMEMBER`](https://redis.io/docs/latest/commands/srandmember) in order to retrieve\na set of random objects. Sets can also be used to check for existence when\nall I need is to test if a given item exists or not or has a single boolean\nproperty or not.\n\nSimilarly lists can be used in order to index items into a fixed order.\nI can add all my items into a Redis list and rotate the list with\n[`RPOPLPUSH`](https://redis.io/docs/latest/commands/rpoplpush) using the same key name as source and destination. This is useful\nwhen I want to process a given set of items again and again forever in the\nsame order. Think of an RSS feed system that needs to refresh the local copy\nperiodically.\n\nAnother popular index often used with Redis is a **capped list**, where items\nare added with [`LPUSH`](https://redis.io/docs/latest/commands/lpush) and trimmed with [`LTRIM`](https://redis.io/docs/latest/commands/ltrim), in order to create a view\nwith just the latest N items encountered, in the same order they were\nseen." }, { "id": "index-inconsistency", "title": "Index inconsistency", "role": "content", "text": "Keeping the index updated may be challenging, in the course of months\nor years it is possible that inconsistencies are added because of software\nbugs, network partitions or other events.\n\nDifferent strategies could be used. If the index data is outside Redis\n*read repair* can be a solution, where data is fixed in a lazy way when\nit is requested. When we index data which is stored in Redis itself\nthe [`SCAN`](https://redis.io/docs/latest/commands/scan) family of commands can be used in order to verify, update or\nrebuild the index from scratch, incrementally." } ], "examples": [] }