diff --git a/docs/command-reference/stream/xack.md b/docs/command-reference/stream/xack.md index acf0719..68864fe 100644 --- a/docs/command-reference/stream/xack.md +++ b/docs/command-reference/stream/xack.md @@ -1,5 +1,5 @@ --- -description: Learn how to use Redis XACK to acknowledge the processing of a message from a stream by a consumer. +description: Learn how to use Redis XACK to acknowledge the processing of a message from a stream by a consumer. --- import PageTitle from '@site/src/components/PageTitle'; @@ -8,26 +8,87 @@ import PageTitle from '@site/src/components/PageTitle'; -## Syntax +## Introduction + +In Dragonfly, as well as in Redis and Valkey, the `XACK` command is used to acknowledge one or more messages in a stream that have been successfully processed. +This is particularly useful in stream processing systems where consumers must signal that a message has been handled, helping to manage message delivery and retention more efficiently within a consumer group. - XACK key group id [id ... ] +## Syntax -**Time complexity:** O(1) for each message ID processed. +```shell +XACK key group id [id ...] +``` -**ACL categories:** @write, @stream, @fast +## Parameter Explanations -**XACK** command acknowledges one or more messages by removing the messages from the pending entries list (PEL) of the specified consumer stream group. A message is pending, and as such stored inside the PEL, when it was delivered to some consumer, normally as a side effect of calling XREADGROUP, or when a consumer took ownership of a message calling XCLAIM. The pending message was delivered to some consumer but the server is yet not sure it was processed at least once. So new calls to XREADGROUP to grab the messages history for a consumer (for instance using an ID of 0), will return such message. Similarly the pending message will be listed by the XPENDING command, that inspects the PEL. Once a consumer successfully processes a message, it should call **XACK** so that such message does not get processed again, and as a side effect, the PEL entry about this message is also purged, releasing memory from the Dragonfly server. +- `key`: The name of the stream. +- `group`: The name of the consumer group. +- `id`: The ID of the message to acknowledge. Multiple IDs can be specified. +## Return Values -## Return +The command returns an integer indicating the number of messages that were successfully acknowledged. -[Integer reply](https://redis.io/docs/reference/protocol-spec/#integers), specifically: +## Code Examples -The command returns the number of messages successfully acknowledged. Certain message IDs may no longer be part of the PEL (for example because they have already been acknowledged), and XACK will not count them as successfully acknowledged. +### Basic Acknowledgement Example -## Examples +Acknowledge a single message that has been processed: ```shell -dragonfly> XACK mystream mygroup 1526569495631-0 +dragonfly$> XADD mystream * name Alice age 30 +"1609097574170-0" +dragonfly$> XGROUP CREATE mystream mygroup $ MKSTREAM +OK +dragonfly$> XREADGROUP GROUP mygroup Alice COUNT 1 STREAMS mystream > +1) 1) "mystream" + 2) 1) 1) "1609097574170-0" + 2) 1) "name" + 2) "Alice" + 3) "age" + 4) "30" +dragonfly$> XACK mystream mygroup 1609097574170-0 (integer) 1 ``` + +### Acknowledging Multiple Messages + +Acknowledge multiple messages after processing: + +```shell +dragonfly$> XADD mystream * name Bob age 25 +"1609097574171-0" +dragonfly$> XADD mystream * name Charlie age 40 +"1609097574172-0" +dragonfly$> XACK mystream mygroup 1609097574171-0 1609097574172-0 +(integer) 2 +``` + +### Handling Non-existent IDs + +Acknowledge attempt on a non-existent message ID: + +```shell +dragonfly$> XACK mystream mygroup 1609097574173-0 +(integer) 0 # No acknowledgment since the ID does not exist. +``` + +## Best Practices + +- Use `XACK` to signal message processing completion to avoid reprocessing. +- Implement proper error handling to manage message IDs that may no longer exist or are not yet acknowledged. + +## Common Mistakes + +- Not creating a consumer group before attempting to acknowledge messages; `XACK` requires a valid consumer group. +- Attempting to acknowledge messages using IDs that were never read by the group. + +## FAQs + +### What happens if the message ID is not found? + +If the message ID does not exist in the pending entries list of the consumer group, `XACK` returns `0`. + +### Can `XACK` be used for automatic acknowledgement? + +No, `XACK` requires explicit calls to acknowledge messages after successful processing by consumers. diff --git a/docs/command-reference/stream/xinfo-groups.md b/docs/command-reference/stream/xinfo-groups.md index 010b10c..49aaced 100644 --- a/docs/command-reference/stream/xinfo-groups.md +++ b/docs/command-reference/stream/xinfo-groups.md @@ -1,5 +1,5 @@ --- -description: Learn how to use Redis XINFO GROUPS to get information about consumer groups of a stream. +description: Learn how to use Redis XINFO GROUPS to get information about consumer groups of a stream. --- import PageTitle from '@site/src/components/PageTitle'; @@ -8,47 +8,90 @@ import PageTitle from '@site/src/components/PageTitle'; +## Introduction + +In Dragonfly, as well as in Redis and Valkey, the `XINFO GROUPS` command provides information about consumer groups associated with a specific stream. +This command is essential for monitoring and managing stream processing tasks, providing insights into consumer group configurations and activity. + ## Syntax - XINFO GROUPS key +```shell +XINFO GROUPS key +``` -**ACL categories:** @read, @stream, @slow +## Parameter Explanations -**XINFO GROUPS** command returns details of every consumer group -that belong to the specified stream ****. +- `key`: The key of the stream for which consumer groups information is requested. -The command returns the following details for each group: +## Return Values - * **name:** The consumer group's name - * **consumers:** The number of consumers in the group - * **pending:** The length of the group's pending entries list (PEL), - which are messages that were delivered but are yet to be acknowledged. - * **last-delivered-id:** The ID of the last entry delivered to the - group's consumers. +The command returns a list of dictionaries, each dictionary representing a consumer group and its details. +The details include fields such as `name`, `consumers`, `pending`, `last-delivered-id`, etc. -## Return +## Code Examples -[Array reply](https://redis.io/docs/reference/protocol-spec/#arrays): -a list of consumer groups. +### Basic Example -## Example +Get information about consumer groups for a stream: ```shell -dragonfly> XINFO GROUPS mystream +dragonfly$> XGROUP CREATE mystream mygroup $ +OK +dragonfly$> XINFO GROUPS mystream 1) 1) "name" 2) "mygroup" 3) "consumers" - 4) (integer) 2 + 4) (integer) 0 + 5) "pending" + 6) (integer) 0 + 7) "last-delivered-id" + 8) "0-0" +``` + +### Monitor Multiple Consumer Groups + +Creating and monitoring multiple consumer groups: + +```shell +dragonfly$> XGROUP CREATE mystream group1 $ +OK +dragonfly$> XGROUP CREATE mystream group2 $ +OK +dragonfly$> XINFO GROUPS mystream +1) 1) "name" + 2) "group1" + 3) "consumers" + 4) (integer) 0 5) "pending" - 6) (integer) 10 + 6) (integer) 0 7) "last-delivered-id" - 8) "1623910467320-1" + 8) "0-0" 2) 1) "name" - 2) "another-group" + 2) "group2" 3) "consumers" - 4) (integer) 1 + 4) (integer) 0 5) "pending" - 6) (integer) 1 + 6) (integer) 0 7) "last-delivered-id" - 8) "1623910847311-1" + 8) "0-0" ``` + +## Best Practices + +- Regularly use `XINFO GROUPS` to check on the health and status of consumer groups in your stream processing architecture. +- Ensure consumer groups are correctly created with unique and descriptive names to avoid confusion and manage them effectively. + +## Common Mistakes + +- Omitting the `key` parameter, which is necessary to identify the stream for which group information is needed. +- Assuming that `XINFO GROUPS` modifies consumer groups; it only retrieves information. + +## FAQs + +### What happens if the stream key does not exist? + +If the stream key does not exist, `XINFO GROUPS` returns an empty list as there are no consumer groups associated with a nonexistent stream. + +### Can `XINFO GROUPS` be used on keys that are not streams? + +No, `XINFO GROUPS` is specific to streams, and attempting to use it on a non-stream key will result in an error. diff --git a/docs/command-reference/stream/xinfo-stream.md b/docs/command-reference/stream/xinfo-stream.md index 9459404..c80be7a 100644 --- a/docs/command-reference/stream/xinfo-stream.md +++ b/docs/command-reference/stream/xinfo-stream.md @@ -8,130 +8,75 @@ import PageTitle from '@site/src/components/PageTitle'; -## Syntax +## Introduction - XINFO STREAM key [FULL [COUNT count]] +In Dragonfly, as well as in Redis and Valkey, the `XINFO STREAM` command provides information about a specific stream. +This is particularly useful for monitoring and debugging streams, as it helps you understand the structure and state of a stream. -**ACL categories:** @read, @stream, @slow +## Syntax -**XINFO STREAM** command returns information about the stream stored at ****. +```shell +XINFO STREAM key +``` -The informative details provided by this command are: +## Parameter Explanations - * **length:** the number of entries in the stream (see **XLEN**) - * **radix-tree-keys:** the number of keys in the underlying radix data structure - * **radix-tree-nodes:** the number of nodes in the underlying radix data structure - * **groups:** the number of consumer groups defined for the stream - * **last-generated-id:** the ID of the least-recently entry that was added to the stream - * **max-deleted-entry-id:** the maximal entry ID that was deleted from the stream - * **entries-added:** the count of all entries added to the stream during its lifetime - * **first-entry:** the ID and field-value tuples of the first entry in the stream - * **last-entry:** the ID and field-value tuples of the last entry in the stream +- `key`: The key of the stream for which information is to be retrieved. -The optional **FULL** modifier provides a more verbose reply. When provided, the **FULL** reply includes an **entries** array that consists of the stream entries (ID and field-value tuples) in ascending order. Furthermore, **groups** is also an array, and for each of the consumer groups it consists of the information reported by **XINFO GROUPS** and **XINFO CONSUMERS**. +## Return Values -The **COUNT** option can be used to limit the number of stream and PEL entries that are returned (The first **** entries are returned). The default **COUNT** is **10** and a **COUNT** of **0** means that all entries will be returned (execution time may be long if the stream has a lot of entries). +The command returns a list of key-value pairs providing information about the specified stream's state and elements. -## Return +## Code Examples -[Array reply](https://redis.io/docs/reference/protocol-spec/#arrays): -a list of informational bits. +### Retrieve Stream Information -## Example +Get information about a stream: ```shell -dragonfly> XINFO STREAM mystream +dragonfly$> XADD mystream * sensor-id 1234 temperature 19.8 +"1632494980015-0" +dragonfly$> XINFO STREAM mystream 1) "length" - 2) (integer) 2 + 2) (integer) 1 3) "radix-tree-keys" 4) (integer) 1 5) "radix-tree-nodes" 6) (integer) 2 - 7) "last-generated-id" - 8) "1638125141232-0" - 9) "max-deleted-entry-id" -10) "0-0" -11) "entries-added" -12) (integer) 2 -13) "groups" -14) (integer) 1 -15) "first-entry" -16) 1) "1638125133432-0" - 2) 1) "message" - 2) "apple" -17) "last-entry" -18) 1) "1638125141232-0" - 2) 1) "message" - 2) "banana" + 7) "groups" + 8) (integer) 0 + 9) "last-generated-id" +10) "1632494980015-0" +11) "first-entry" +12) 1) "1632494980015-0" + 2) 1) "sensor-id" + 2) "1234" + 3) "temperature" + 4) "19.8" +13) "last-entry" +14) 1) "1632494980015-0" + 2) 1) "sensor-id" + 2) "1234" + 3) "temperature" + 4) "19.8" ``` -Full reply: +## Best Practices -```shell -dragonfly> XADD mystream * foo bar -"1638125133432-0" +- Regularly use `XINFO STREAM` to monitor the health and performance of streams. +- Analyze the output to optimize memory and understand stream usage patterns. -dragonfly> XADD mystream * foo bar2 -"1638125141232-0" +## Common Mistakes -dragonfly> XGROUP CREATE mystream mygroup 0-0 -OK +- Not grasping the meaning of each output field; ensure you understand terms like "radix-tree-keys" and "last-generated-id". +- Forgetting that `XINFO STREAM` only queries data and does not modify the stream content. -dragonfly> XREADGROUP GROUP mygroup Alice COUNT 1 STREAMS mystream > -1) 1) "mystream" - 2) 1) 1) "1638125133432-0" - 2) 1) "foo" - 2) "bar" +## FAQs -dragonfly> XINFO STREAM mystream FULL - 1) "length" - 2) (integer) 2 - 3) "radix-tree-keys" - 4) (integer) 1 - 5) "radix-tree-nodes" - 6) (integer) 2 - 7) "last-generated-id" - 8) "1638125141232-0" - 9) "max-deleted-entry-id" -10) "0-0" -11) "entries-added" -12) (integer) 2 -13) "entries" -14) 1) 1) "1638125133432-0" - 2) 1) "foo" - 2) "bar" - 2) 1) "1638125141232-0" - 2) 1) "foo" - 2) "bar2" -15) "groups" -16) 1) 1) "name" - 2) "mygroup" - 3) "last-delivered-id" - 4) "1638125133432-0" - 5) "entries-read" - 6) (integer) 1 - 7) "lag" - 8) (integer) 1 - 9) "pel-count" - 10) (integer) 1 - 11) "pending" - 12) 1) 1) "1638125133432-0" - 2) "Alice" - 3) (integer) 1638125153423 - 4) (integer) 1 - 13) "consumers" - 14) 1) 1) "name" - 2) "Alice" - 3) "seen-time" - 4) (integer) 1638125133422 - 5) "active-time" - 6) (integer) 1638125133432 - 7) "pel-count" - 8) (integer) 1 - 9) "pending" - 10) 1) 1) "1638125133432-0" - 2) (integer) 1638125133432 - 3) (integer) 1 -``` +### What does "radix-tree-keys" signify? + +"radix-tree-keys" indicates the number of entries in the underlying radix tree data structure, which can help assess memory use. +### How do I interpret "first-entry" and "last-entry"? +"first-entry" and "last-entry" give you the stream's start and end records, which are important for identifying the current pattern of data. \ No newline at end of file diff --git a/docs/command-reference/stream/xlen.md b/docs/command-reference/stream/xlen.md index f089d0e..9b46811 100644 --- a/docs/command-reference/stream/xlen.md +++ b/docs/command-reference/stream/xlen.md @@ -8,36 +8,85 @@ import PageTitle from '@site/src/components/PageTitle'; +## Introduction + +In Dragonfly, as well as in Redis and Valkey, the `XLEN` command is used to determine the number of entries in a stream. +It is particularly useful for monitoring and managing data streams, providing quick insights into the size and health of your data pipeline. + ## Syntax - XLEN key +```shell +XLEN key +``` + +## Parameter Explanations -**Time Complexity:** O(1) +- `key`: The key of the stream for which you wish to know the number of entries. -**ACL categories:** @read, @stream, @fast +## Return Values -Returns the number of entries inside a stream. If the specified -key does not exist the command returns zero, as if the stream was -empty. Note that a stream can be empty and hence **XLEN** is not -a good option to check if a stream exists. +The command returns the number of entries (as an integer) currently present in the specified stream. -Streams are not auto-deleted once they have no entries inside -(for instance after an **XDEL** call), because the stream may have -consumer groups associated with it. +## Code Examples -## Return -[Integer reply](https://redis.io/docs/reference/protocol-spec/#integers): -the number of entries of the stream at key. +### Basic Example -## Example +Determine the number of entries in a stream: + ```shell -dragonfly> XADD mystream * name John -"1623910120014-0" -dragonfly> XADD mystream * name Bob -"1623910194423-0" -dragonfly> XADD mystream * name Alice -"1623910226955-0" -dragonfly> XLEN mystream +dragonfly$> XADD mystream * sensor-temperature 23.1 +"1678819562090-0" +dragonfly$> XADD mystream * sensor-humidity 60 +"1678819562091-0" +dragonfly$> XADD mystream * sensor-temperature 22.8 +"1678819562092-0" +dragonfly$> XLEN mystream (integer) 3 ``` + +### Handling a Non-existent Stream + +Check the length of a non-existent stream: + + +```shell +dragonfly$> XLEN nonexistent_stream +(integer) 0 +``` + +### Using `XLEN` in Monitoring + +Consider using `XLEN` as part of a monitoring solution to track the number of events in a stream at any point in time: + + +```shell +dragonfly$> XADD events * user-login user123 +"1678819562093-0" +dragonfly$> XADD events * user-logout user123 +"1678819562094-0" +dragonfly$> XLEN events +(integer) 2 +# Use this value to monitor activity levels or detect anomalies in the number of events. +``` + +## Best Practices + +- Integrate `XLEN` into your monitoring dashboards to provide real-time feedback on stream sizes and help detect bottlenecks or unusual activity patterns. +- Regularly check stream lengths as part of a comprehensive data management strategy. + +## Common Mistakes + +- Using `XLEN` on non-stream data types will result in an error, so ensure that the key provided is indeed a stream. +- Confusing `XLEN` with list length commands like `LLEN`, as they apply to different data structures. + +## FAQs + +### What happens if the key does not exist? + +If the key does not exist or is not of stream type, `XLEN` returns `0`. + +### Can `XLEN` be used to determine entries in other data types like lists or sets? + +No, `XLEN` is specifically designed for streams. +To determine the length of other data types, use the appropriate commands such as `LLEN` for lists or `SCARD` for sets. \ No newline at end of file diff --git a/docs/command-reference/stream/xpending.md b/docs/command-reference/stream/xpending.md index b3f596d..82c89c2 100644 --- a/docs/command-reference/stream/xpending.md +++ b/docs/command-reference/stream/xpending.md @@ -8,139 +8,95 @@ import PageTitle from '@site/src/components/PageTitle'; -## Syntax +## Introduction - XPENDING key group [[IDLE min-idle-time] start end count [consumer]] +In Dragonfly, as well as in Redis and Valkey, the `XPENDING` command is used to retrieve information about pending messages in a stream consumer group. +This command is particularly useful for monitoring and managing consumer group message delivery and ensuring robust processing in streaming applications. -**Time Complexity:** O(N) with N being the number of elements returned, so asking for a small fixed number of entries per call is O(1). -O(M), where M is the total number of entries scanned when used with the IDLE filter. -When the command returns just the summary and the list of consumers is small, it runs in O(1) time; otherwise, an additional O(N) time for iterating every consumer. +## Syntax -**ACL categories:** @read, @stream, @slow +```shell +XPENDING key group [start end count] [consumer] +``` -Fetching data from a stream via a consumer group, and not acknowledging such data, has the effect of creating pending entries. -This is well explained in the `XREADGROUP` command. -You can also learn more about Streams [here](https://redis.io/docs/data-types/streams/). +## Parameter Explanations -The `XACK` command will immediately remove the pending entry from the Pending Entries List (PEL) since once a message is successfully processed, -there is no longer need for the consumer group to track it and to remember the current owner of the message. +- `key`: The name of the stream. +- `group`: The name of the consumer group to inspect. +- `start` (optional): Start ID for the range of messages to retrieve. +- `end` (optional): End ID for the range of messages. +- `count` (optional): Maximum number of messages to return. +- `consumer` (optional): When specified, the command restricts the results to messages pending for this particular consumer. -The `XPENDING` command is the interface to inspect the list of pending messages, -and is as thus a very important command in order to observe and understand what is happening with a streams consumer groups: -what clients are active, what messages are pending to be consumed, or to see if there are idle messages. -Moreover, this command, together with [`XCLAIM`](./xclaim.md) is used in order to implement recovering of consumers that are failing for a long time, -and as a result certain messages are not processed: a different consumer can claim the message and continue. +## Return Values -## Summary Form +The `XPENDING` command returns the number of pending messages, followed by a summary with detailed information about specified pending messages. +It includes message IDs, consumer IDs, and the time since each message was claimed. -When `XPENDING` is called with just a key name and a consumer group name, it just returns output in the **summary form** about the pending messages in a given consumer group. -In the following example, we create a consumer group and immediately create a pending message by reading from the group with `XREADGROUP`. +## Code Examples -```shell -dragonfly> XADD mystream * name Alice surname Adams -"1695755830453-0" +### Basic Example -dragonfly> XADD mystream * name John surname Doe -"1695755847112-0" +Get basic pending message information: -dragonfly> XGROUP CREATE mystream mygroup 0-0 +```shell +dragonfly$> XADD mystream * name John +"1657659051233-0" +dragonfly$> XGROUP CREATE mystream mygroup 0 OK - -dragonfly> XREADGROUP GROUP mygroup consumer-123 COUNT 1 STREAMS mystream > +dragonfly$> XREADGROUP GROUP mygroup myconsumer COUNT 1 STREAMS mystream > 1) 1) "mystream" - 2) 1) 1) "1695755830453-0" + 2) 1) 1) "1657659051233-0" 2) 1) "name" - 2) "Alice" - 3) "surname" - 4) "Adams" -``` - -We expect the pending entries list for the consumer group `mygroup` to have a message right now: consumer named `consumer-123` fetched the message without acknowledging its processing. -The simple summary form of `XPENDING` will give us this information: - -```shell -dragonfly> XPENDING mystream mygroup + 2) "John" +dragonfly$> XPENDING mystream mygroup 1) (integer) 1 -2) "1695755830453-0" -3) "1695755830453-0" -4) 1) 1) "consumer-123" - 2) (integer) 1 +2) "1657659051233-0" +3) "myconsumer" +4) (integer) 1231 ``` -In this form, the command outputs the total number of pending messages for this consumer group, which is one, followed by the smallest and greatest ID among the pending messages, -and then list every consumer in the consumer group with at least one pending message, and the number of pending messages it has. +### Get Pending Messages for a Specific Consumer -## Extended Form - -The summary form provides a good overview, but sometimes we are interested in the details. -In order to see all the pending messages, in the **extended form**, with more associated information we need to also pass a range of IDs, in a similar way we do it with [`XRANGE`](./xrange.md) , -and a non-optional count argument, to limit the number of messages returned per call: +Retrieve pending messages for a specific consumer: ```shell -dragonfly> XPENDING mystream mygroup - + 10 -1) 1) "1695755830453-0" - 2) "consumer-123" - 3) (integer) 1257837 +dragonfly$> XPENDING mystream mygroup - + 10 myconsumer +1) 1) "1657659051233-0" + 2) "myconsumer" + 3) (integer) 1231 4) (integer) 1 ``` -In the extended form we no longer see the summary information, instead there is detailed information for each message in the pending entries list. -For each message four attributes are returned: - -1. The ID of the message. -2. The current owner of the message (i.e., the consumer that fetched the message and has still to acknowledge it). -3. The number of milliseconds that elapsed since the last time this message was delivered to this consumer. -4. The number of times this message was delivered. - -The deliveries counter, that is the fourth element in the array, is incremented when some other consumer claims the message with [`XCLAIM`](./xclaim.md) -or when the message is delivered again via `XREADGROUP`, when accessing the history of a consumer in a consumer group. +### Filter by Message ID Range -It is possible to pass an additional argument to the command, in order to see the messages having a specific owner: +Get pending messages within a specific ID range: ```shell -dragonfly> XPENDING mystream mygroup - + 10 consumer-123 +dragonfly$> XPENDING mystream mygroup "1657650000000-0" "1657659999999-0" 10 +1) 1) "1657659051233-0" + 2) "myconsumer" + 3) (integer) 1231 + 4) (integer) 1 ``` -But in the above case the output would be the same, since we have pending messages only for a single consumer `consumer-123`. -However, what is important to keep in mind is that this operation, filtering by a specific consumer, is not inefficient even when there are many pending messages from many consumers: -we have a pending entries list data structure both globally, and for every consumer, so we can very efficiently show just messages pending for a single consumer. - -## Idle Time Filter +## Best Practices -It is also possible to filter pending stream entries by their idle-time, given in milliseconds (useful for [`XCLAIM`](./xclaim.md)ing entries that have not been processed for some time): +- Regularly monitor pending messages to manage unacknowledged messages effectively. +- Use `XPENDING` in coordination with other stream commands like `XCLAIM` to address stuck messages. +- Limit the number of returned messages with `count` to avoid excessive data retrieval. -```shell -dragonfly> XPENDING mystream mygroup IDLE 9000 - + 10 -dragonfly> XPENDING mystream mygroup IDLE 9000 - + 10 consumer-123 -``` +## Common Mistakes -The first case will return the first 10 (or less) PEL entries of the entire group that are idle for over 9 seconds, whereas in the second case only those of `consumer-123`. +- Not specifying the `group` parameter, as it is mandatory to identify the consumer group. +- Misunderstanding `start` and `end` as time ranges instead of message ID ranges. -## Pending Entries List & Exclusive Ranges +## FAQs -The `XPENDING` command allows iterating over the Pending Entries List (PEL) just like [`XRANGE`](./xrange.md) and [`XREVRANGE`](./xrevrange.md) allow for the stream's entries. -You can do this by prefixing the ID of the last-read pending entry with the `(` character that denotes an open (exclusive) range, and proving it to the subsequent call to the command. - -By default, the command returns entries including specified IDs: - -```shell -dragonfly> XPENDING mystream mygroup 1695755830453-0 + 10 -1) 1) "1695755830453-0" - 2) "consumer-123" - 3) (integer) 1577015 - 4) (integer) 1 -``` - -If the ID is prefixed with `(`, the range is exclusive: - -```shell -dragonfly> XPENDING mystream mygroup (1695755830453-0 + 10 -(empty array) -``` +### What happens if the stream or consumer group does not exist? -## Return +If the stream or consumer group does not exist, the `XPENDING` command will return an error. -[Array reply](https://redis.io/docs/reference/protocol-spec/#arrays), specifically: +### Can `start` and `end` be specified as `-` and `+`? -- The command returns data in different format depending on the way it is called, as previously explained in this page. -- However, the reply is always an array of items. +Yes, the special identifiers `-` and `+` can be used to indicate the smallest and largest message IDs in the stream, respectively. \ No newline at end of file diff --git a/docs/command-reference/stream/xrange.md b/docs/command-reference/stream/xrange.md index 832e6f8..8dfe171 100644 --- a/docs/command-reference/stream/xrange.md +++ b/docs/command-reference/stream/xrange.md @@ -8,98 +8,90 @@ import PageTitle from '@site/src/components/PageTitle'; +## Introduction + +In Dragonfly, as well as in Redis and Valkey, the `XRANGE` command is used to return a range of elements in a stream. +Streams are powerful data types for handling ordered logs and time-series data, making `XRANGE` especially useful for retrieving entries within specified time frames or other ranges. + ## Syntax - XRANGE key start end [COUNT count] +```shell +XRANGE key start end [COUNT count] +``` -**Time complexity:** O(N) with N being the number of elements -being returned. If N is constant (e.g. always asking for the -first 10 elements with COUNT), you can consider it O(1). +## Parameter Explanations -**ACL categories:** @read, @stream, @slow +- `key`: The key of the stream from which entries are fetched. +- `start`: The minimum ID of the range, inclusive. +- `end`: The maximum ID of the range, inclusive. +- `COUNT count` (optional): Limits the number of entries returned to 'count'. -The **XRANGE** command returns stream entries matching the -given range of IDs. The range is specified by a minimum and -maximum ID. All the entries having an ID between the two -specified or exactly one of the two IDs specified (closed -interval) are returned. +## Return Values -**XRANGE** allows both **** and **** IDs to be -the same. In that case, the command only returns the specified -entry. +The command returns a list of stream entries that correspond to the specified range. +Each entry is represented by a two-element array with an ID and a subsequent map of field-value pairs. -```shell -dragonfly> xrange mystream 1687926126874-0 1687926126874-0 -1) 1) "1687926126874-0" - 2) 1) "k" - 2) "v" -``` +## Code Examples -## Special IDs +### Basic Example -Sometimes, it is logical to get all the entries of a stream. It -is tedious to mention the minimum and maximum IDs explicitly. -To tackle this issue, **XRANGE** accepts two special characters -to denote the minimum ID and maximum ID of a stream. These are -"**-**" and "**+**" respectively. The following command returns -all entries from the stream *mystream*. +Retrieve all entries within a specified range: ```shell -dragonfly> XRANGE mystream - + -1) 1) "1687926126874-0" - 2) 1) "k" - 2) "v" -2) 1) "1687926132506-0" - 2) 1) "l" - 2) "x" -3) 1) "1687926136634-0" - 2) 1) "n" - 2) "t" -4) 1) "1687926140032-0" - 2) 1) "q" - 2) "r" +# Adding entries to the stream +dragonfly$> XADD mystream * foo "1" bar "data1" +"1609459200000-0" +dragonfly$> XADD mystream * foo "2" bar "data2" +"1609459200010-0" + +# Retrieving entries from '1609459200000-0' to '1609459200010-0' +dragonfly$> XRANGE mystream 1609459200000-0 1609459200010-0 +1) 1) "1609459200000-0" + 2) 1) "foo" + 2) "1" + 3) "bar" + 4) "data1" +2) 1) "1609459200010-0" + 2) 1) "foo" + 2) "2" + 3) "bar" + 4) "data2" ``` -## Incomplete IDs +### Using `COUNT` Option -It is possible to use incomplete IDs in **XRANGE** command. User -can specify just the first part of ID, the millisecond time: +Retrieve a limited number of entries starting from a specific ID: ```shell -dragonfly> XRANGE mystream 1687926126874 1687926140032 +# Adding another entry +dragonfly$> XADD mystream * foo "3" bar "data3" +"1609459200020-0" + +# Retrieve only one entry starting from '1609459200000-0' +dragonfly$> XRANGE mystream 1609459200000-0 1609459200020-0 COUNT 1 +1) 1) "1609459200000-0" + 2) 1) "foo" + 2) "1" + 3) "bar" + 4) "data1" ``` -In this case, XRANGE will auto-complete the start interval with -**-0** and end interval with **-18446744073709551615**, in order -to return all the entries that were generated between a given -millisecond and the end of the other specified millisecond. This -also means that repeating the same millisecond two times, will get -all the entries within such millisecond, because the sequence number -range will be from zero to the maximum. +## Best Practices -## Exclusive Ranges +- When processing large streams, using the `COUNT` parameter can help limit the amount of data transferred and improve performance. +- Use precise `start` and `end` IDs to avoid retrieving unnecessary entries and to maintain efficient operations. -By default, the **XRANGE** command returns entries including specified -IDs. If you want the behaviour to be exclusive, prefix the ID with -**(** character. +## Common Mistakes -## COUNT option +- Providing `start` or `end` in incorrect formats can lead to errors; ensure the IDs follow the correct stream ID format (e.g., `-`). +- Not specifying an `end` range if the intent is to read bounded data. Omitting this parameter leads to fetching entries up to the maximum sequence for the start's timestamp. -**XRANGE** accepts a **COUNT** option to limit the number of entries -returned. It takes integer value. +## FAQs -```shell -dragonfly> XRANGE mystream - + COUNT 1 -1) 1) "1687926126874-0" - 2) 1) "k" - 2) "v" -``` +### What happens if the stream key does not exist? -## Return +If the stream key does not exist, `XRANGE` returns an empty list. -[Array reply](https://redis.io/docs/reference/protocol-spec/#arrays): +### How are stream IDs formatted? -The returned entries are complete, that means that the ID and all the -fields they are composed are returned. Moreover, the entries are -returned with their fields and values in the exact same order as -XADD added them. +Stream IDs are formatted as `-`, where the sequence number starts at zero and increases with each entry at the same timestamp. \ No newline at end of file diff --git a/docs/command-reference/stream/xread.md b/docs/command-reference/stream/xread.md index 9ce34e2..0cb37f9 100644 --- a/docs/command-reference/stream/xread.md +++ b/docs/command-reference/stream/xread.md @@ -8,132 +8,107 @@ import PageTitle from '@site/src/components/PageTitle'; +## Introduction + +In Dragonfly, as well as in Redis and Valkey, the `XREAD` command is used to read data from one or more streams. +It is a blocking command by nature and is commonly utilized for implementing message queues, real-time data processing, and event sourcing patterns. + ## Syntax - XREAD [COUNT count] [BLOCK Milliseconds] STREAMS key [key ...] id [id ...] +```shell +XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] ID [ID ...] +``` + +## Parameter Explanations + +- `COUNT count`: The maximum number of entries to return per stream. Optional, defaults to return all available entries. +- `BLOCK milliseconds`: The maximum number of milliseconds the command will block if no messages are available. Optional, defaults to non-blocking behavior. +- `STREAMS key [key ...]`: One or more stream keys to read from. +- `ID [ID ...]`: One or more specific entry IDs to start reading from, or use `$` to start from the latest message. +## Return Values -**ACL categories:** @read, @stream, @slow, @blocking +The command returns a list of streams and the corresponding entries that were read from them, or an empty list if the `BLOCK` option is used and no messages are available within the specified time. -Read entries from one or multiple streams. In case of multiple -streams, the order of stream entries is maintained. Note that -XREAD returns entries with IDs **greater** than the specified -ones. This is useful in cases where user need to know the new -unread entries from specified streams. +## Code Examples -The number of keys and ids should be balanced. User in no way -can provide keys with fewer ids or vice-versa. The i'th id -corresponds to the i'th key. Below is an example - +### Basic Example + +Read from a single stream starting from the first message: ```shell -dragonfly> XREAD STREAMS mystream other-stream 1687926136634-0 1687921762755-0 +dragonfly$> XADD mystream * name Alice age 30 +"1609945144079-0" +dragonfly$> XADD mystream * name Bob age 25 +"1609945144079-1" + +dragonfly$> XREAD STREAMS mystream 0 1) 1) "mystream" - 2) 1) 1) "1687926140032-0" - 2) 1) "k" - 2) "v" -2) 1) "other-stream" - 2) 1) 1) "1687924609465-0" - 2) 1) "k2" - 2) "v2" + 2) 1) 1) "1609945144079-0" + 2) 1) "name" + 2) "Alice" + 3) "age" + 4) "30" + 2) 1) "1609945144079-1" + 2) 1) "name" + 2) "Bob" + 3) "age" + 4) "25" ``` +### Using `XREAD` with `COUNT` -### COUNT - -User can limit the number of received entries by specifying the -**COUNT** limit. It takes an integer value and returns atmost -that number of entries. +Read a limited number of entries from a stream: ```shell -dragonfly> XREAD COUNT 2 STREAMS mystream 0 +dragonfly$> XREAD COUNT 1 STREAMS mystream 0 1) 1) "mystream" - 2) 1) 1) "1686831019899-0" - 2) 1) "a" - 2) "1" - 2) 1) "1686831037806-0" - 2) 1) "b" - 2) "2" + 2) 1) 1) "1609945144079-0" + 2) 1) "name" + 2) "Alice" + 3) "age" + 4) "30" ``` -### BLOCK +### Using `XREAD` with Block Option -Sometimes, the specified stream doesn't have any new entries to -consume. New entries may be added in some interval or time range. -In that case, **BLOCK** option comes in handy. **BLOCK** takes -a value denoting the *milliseconds* the command will block for. +Block the command until new data arrives in the stream: -Once **BLOCK** is used in the command, the command waits for the -specified time interval. The command returns immediately if one of -the requested stream receive new entries within the specified time -range. Else it simply returns a Null reply. - -If new entries are already present in the stream before executing -command, it immediately returns those entries even if **BLOCK** -option is specified. +```shell +# Start a new terminal to add data after blocking +dragonfly$> XREAD BLOCK 2000 STREAMS mystream $ +(null) -Note that, all clients who are waiting for the same range of IDs -get same new entries. +# In another terminal, add a new entry +dragonfly$> XADD mystream * name Charlie age 40 +"1609945245092-0" -### STREAMS +# Return to previous terminal and observe `XREAD` output +1) 1) "mystream" + 2) 1) 1) "1609945245092-0" + 2) 1) "name" + 2) "Charlie" + 3) "age" + 4) "40" +``` -**STREAMS** is a required option and it must be the last specified -option. It takes lists of keys and ids. The format is as follows - +## Best Practices -```shell -STREAMS key1 key2 key3 ... id1 id2 id3 ... -``` +- Consider using the `BLOCK` option to efficiently poll streams for new entries, reducing the need for constant querying. +- Use `COUNT` to prevent overwhelming your application with massive amounts of data if not necessary. -The number of keys and ids should be exactly same. Else a *syntax -error* will be triggered. For a given list of keys and ids, the -command searches for entries greater than i'th id of the i'th stream -(like in the above example, XREAD searches for entries greater than -*id1* for the stream *key1* and so on). Keys should be valid streams. +## Common Mistakes -User can specify IDs in three ways - - * **Incomplete IDs:** Client only specifies the timestamp. Here the sequence - part is interpreted as 0. -```shell -dragonfly> XREAD STREAMS key1 key2 1 0 -``` - The above command is equivalent to the below command: -```shell -dragonfly> XREAD STREAMS key1 key2 1-0 0-0 -``` - * **Complete IDs:** Client specifies the full ID. - * **Special "$" ID:** In some cases, client wants only the recent entries - which are not previously read. Using hard-coded IDs is not an option - here. The special ID **$** is what we need to use to accomplish that. - When specified, the command returns the recent unread entries. It is - equivalent to specifying the last read stream entry ID. -```shell -dragonfly> XREAD STREAMS key $ -``` +- Overlooking the starting ID for the stream, which may result in missing entries if set incorrectly. +- Using blocking operations without a reasonable timeout can lead to application hangs if streams are quiet. -## Return +## FAQs -[Array Reply](https://redis.io/docs/reference/protocol-spec/#arrays). -In some cases, the command returns a null reply. For example, when **BLOCK** -is used. +### What happens if a stream doesn't exist? -## Example +If a specified stream key does not exist, `XREAD` will not return entries from it and will move on to other specified streams. -```shell -dragonfly> XADD mystream * k v -"1687927570399-0" -dragonfly> XADD mystream * k2 v2 -"1687927576419-0" -dragonfly> XADD mystream * k3 v3 -"1687927580654-0" -dragonfly> xread streams mystream 1687927570398 -1) 1) "mystream" - 2) 1) 1) "1687927570399-0" - 2) 1) "k" - 2) "v" - 2) 1) "1687927576419-0" - 2) 1) "k2" - 2) "v2" - 3) 1) "1687927580654-0" - 2) 1) "k3" - 2) "v3" -``` +### Can `XREAD` be used for multiple streams? +Yes, you can specify multiple streams to read simultaneously. +The command will return data from them in the specified order. \ No newline at end of file diff --git a/docs/command-reference/stream/xreadgroup.md b/docs/command-reference/stream/xreadgroup.md index 0992a07..db79e9d 100644 --- a/docs/command-reference/stream/xreadgroup.md +++ b/docs/command-reference/stream/xreadgroup.md @@ -8,140 +8,96 @@ import PageTitle from '@site/src/components/PageTitle'; -## Syntax - - XREADGROUP GROUP group consumer [COUNT count] [BLOCK milliseconds] - [NOACK] STREAMS key [key ...] id [id ...] - -**Time Complexity:** For each stream mentioned: O(M) with M being the number of elements returned. -If M is constant (e.g. always asking for the first 10 elements with COUNT), you can consider it O(1). -On the other side when `XREADGROUP` blocks, [`XADD`](./xadd.md) will pay the O(N) time in order to serve the N clients blocked on the stream getting new data. - -**ACL categories:** @write, @stream, @slow, @blocking - -The `XREADGROUP` command is a special version of the [`XREAD`](./xread.md) command with support for consumer groups. -It is recommended to understand the [`XREAD`](./xread.md) command before reading this page. - -You can learn more about Streams [here](https://redis.io/docs/data-types/streams/). +## Introduction -## Differences Between `XREAD` & `XREADGROUP` +In Dragonfly, as well as in Redis and Valkey, the `XREADGROUP` command is used to read messages from a stream, primarily used within the context of consumer groups. +Consumer groups in Redis allow you to implement message feed processing, where different consumers can read distinct messages avoiding reading duplicates, thus enhancing efficient data processing. -From the point of view of the syntax, the commands are almost the same, however `XREADGROUP` requires a special and mandatory option: +## Syntax -```text -GROUP +```shell +XREADGROUP GROUP groupname consumer [COUNT count] [BLOCK milliseconds] + [NOACK] STREAMS key [key ...] ID [ID ...] ``` -The `group-name` argument is the name of a consumer group associated to the stream. -The group is created using the [`XGROUP CREATE`](./xgroup-create.md) command. -The consumer name is the string that is used by the client to identify itself inside the group. -The consumer is auto created inside the consumer group the first time it is seen. -Different clients should select a different consumer name. - -When you read with `XREADGROUP`, the server will remember that a given message was delivered to you: -the message will be stored inside the consumer group in what is called a **Pending Entries List (PEL)**, -that is a list of message IDs delivered but not yet acknowledged. - -The client will have to acknowledge the message processing using [`XACK`](./xack.md) in order for the pending entry to be removed from the PEL. -The PEL can be inspected using the [`XPENDING`](./xpending.md) command. - -The `NOACK` subcommand can be used to avoid adding the message to the PEL in cases where reliability is not a requirement and the occasional message loss is acceptable. -This is equivalent to acknowledging the message when it is read. - -The ID(s) to specify in the `STREAMS` option when using `XREADGROUP` can be one of the following two: +## Parameter Explanations -- The special `>` ID, which means that the consumer want to receive only messages that were never delivered to any other consumer. - It just means, give me new messages. -- Any other ID, that is, `0` or any other valid ID or incomplete ID (just the millisecond time part), - will have the effect of returning entries that are pending for the consumer sending the command with IDs greater than the one provided. - So basically if the ID is not `>`, then the command will just let the client access its pending entries: messages delivered to it, but not yet acknowledged. - Note that in this case, both `BLOCK` and `NOACK` options are ignored. +- `groupname`: The name of the consumer group. +- `consumer`: The name of the consumer within the group. +- `COUNT count` (optional): Limits the number of returned entries. Default is all available entries. +- `BLOCK milliseconds` (optional): Blocks the command and waits for specified milliseconds if no entries are available. +- `NOACK` (optional): Specifies the server should not maintain the acknowledged status of the messages. +- `key`: One or more stream keys to read from. +- `ID`: One or more IDs, typically `>`, to indicate reading new messages. -Like [`XREAD`](./xread.md), the `XREADGROUP` command can be used in a blocking way. -There are no differences in this regard. +## Return Values -## Message Delivery +The command returns entries from the stream, grouped by stream name. +If no entries are available and the `BLOCK` option was not used, it returns an empty array. -When a message is delivered to a consumer (i.e., read by using `XREADGROUP`), two things happen: +## Code Examples -1. If the message was never delivered to anyone (i.e., a new message) then a PEL (Pending Entries List) is created. -2. If instead the message was already delivered to this consumer, and it is just re-fetching the same message again, - then the last delivery counter is updated to the current time, and the number of deliveries is incremented by one. - You can access those message properties using the [`XPENDING`](./xpending.md) command. +### Basic Example -## Message Deletion +Create a consumer group and read messages: -Entries may be deleted from the stream due to trimming or explicit [`XDEL`](./xdel.md) at any time. -Dragonfly doesn't prevent the deletion of entries that are present in the stream's PELs. -When this happens, the PELs retain the deleted entries' IDs, but the actual entry payload is no longer available. -Therefore, when reading such PEL entries, Dragonfly will return a null value in place of their respective data. -See the [Read Deleted Messages](#read-deleted-messages) section below for more information. - -## Return - -[Array reply](https://redis.io/docs/reference/protocol-spec/#arrays), specifically: +```shell +dragonfly$> XADD mystream * field1 value1 +"16082358984-0" +dragonfly$> XGROUP CREATE mystream mygroup 0 +OK +dragonfly$> XREADGROUP GROUP mygroup consumer1 STREAMS mystream > +1) "mystream" +2) 1) 1) "16082358984-0" + 2) 1) "field1" + 2) "value1" +``` -The command returns an array of results: each element of the returned array is an array composed of a two element containing the key name and the entries reported for that key. -The entries reported are full stream entries, having IDs and the list of all the fields and values. -Field and values are guaranteed to be reported in the same order they were added by [`XADD`](./xadd.md). +### Reading Specific Number of Messages -When `BLOCK` is used, a `null` reply is returned upon timeout. +Read up to two messages from a stream: -## Examples +```shell +dragonfly$> XADD mystream * field2 value2 +"16082358985-0" +dragonfly$> XADD mystream * field3 value3 +"16082358986-0" +dragonfly$> XREADGROUP GROUP mygroup consumer1 COUNT 2 STREAMS mystream > +1) "mystream" +2) 1) 1) "16082358985-0" + 2) 1) "field2" + 2) "value2" + 2) 1) "16082358986-0" + 2) 1) "field3" + 2) "value3" +``` -### Basic Usage +### Using `XREADGROUP` with `BLOCK` -Normally you can use the `XREADGROUP` command to get new messages and process them. In pseudocode: +Wait for new messages for up to 2000 milliseconds: ```shell -WHILE true - entries = XREADGROUP GROUP $GroupName $ConsumerName BLOCK 2000 COUNT 10 STREAMS mystream > - if entries == nil - puts "Timeout... try again" - CONTINUE - end - - FOREACH entries AS stream_entries - FOREACH stream_entries as message - process_message(message.id,message.fields) - - # ACK the message as processed - XACK mystream $GroupName message.id - END - END -END +dragonfly$> XREADGROUP GROUP mygroup consumer2 BLOCK 2000 STREAMS mystream > ``` -In this way the example consumer code will fetch only new messages, process them, and acknowledge them via [`XACK`](./xack.md). -However, the pseudocode above does not handle recovering after a crash. -If the consumer code crashes in the middle of processing messages, messages will remain in the PEL. -We can access historical messages by giving `XREADGROUP` initially an ID of `0`, and performing the same loop. -Once providing an ID of `0` the reply is an empty set of messages, we know that we processed and acknowledged all the pending messages. -From there, we can start to use `>` as ID, in order to get the new messages and rejoin the consumers that are processing new messages. +## Best Practices -### Read Deleted Messages +- Use consumer groups to parallelize the processing of a stream across multiple workers. +- Consider using the `ACK` command frequently to acknowledge processed messages for memory efficiency. +- Selectively use the `BLOCK` option to avoid unnecessary database polling, especially in high-throughput systems. -As mentioned earlier, when a message is deleted from a stream, it is not removed from the consumer group's PEL. -When reading such PEL entries, Dragonfly will return a null value in place of their respective data. +## Common Mistakes -```shell -dragonfly> XADD mystream 1 myfield mydata -"1-0" +- Forgetting to create a consumer group using `XGROUP CREATE` before calling `XREADGROUP`. +- Misunderstanding blocking behavior when using the `BLOCK` option without a timeout, which can lead to hanging processes. +- Ignoring the need for message acknowledgment unless `NOACK` is specified. -dragonfly> XGROUP CREATE mystream mygroup 0 -OK +## FAQs -dragonfly> XREADGROUP GROUP mygroup myconsumer STREAMS mystream > -1) 1) "mystream" - 2) 1) 1) "1-0" - 2) 1) "myfield" - 2) "mydata" +### What happens if a stream key doesn't exist? -dragonfly> XDEL mystream 1-0 -(integer) 1 +The `XREADGROUP` command will return an empty array if a stream key does not exist or if there are no messages to read. -dragonfly> XREADGROUP GROUP mygroup myconsumer STREAMS mystream 0 -1) 1) "mystream" - 2) 1) 1) "1-0" - 2) (nil) -``` +### How can I include already read messages? + +To re-read messages, specify the appropriate entry IDs instead of using just the `>` character in the `ID` argument. \ No newline at end of file diff --git a/docs/command-reference/stream/xrevrange.md b/docs/command-reference/stream/xrevrange.md index 0defb60..12b9ee3 100644 --- a/docs/command-reference/stream/xrevrange.md +++ b/docs/command-reference/stream/xrevrange.md @@ -8,50 +8,113 @@ import PageTitle from '@site/src/components/PageTitle'; +## Introduction + +In Dragonfly, as well as in Redis and Valkey, the `XREVRANGE` command is used to return a range of elements from a stream in reverse order. +The directionality of this command makes it useful for scenarios where you need to process or display recent data entries before older ones, such as in messaging apps, event logging, or any application requiring reverse chronological order retrieval. + ## Syntax - XREVRANGE key end start [COUNT count] +```shell +XREVRANGE key end start [COUNT count] +``` + +## Parameter Explanations + +- `key`: The key of the stream from which elements are retrieved. +- `end`: The maximum ID of the stream entry to include in the output. +- `start`: The minimum ID of the stream entry to include in the output. +- `COUNT count` (optional): Limits the number of entries returned, fetching the first `count` elements in reverse order. -**Time complexity:** O(N) with N being the number of elements -being returned. If N is constant (e.g. always asking for the -first 10 elements with COUNT), you can consider it O(1). +## Return Values -**ACL categories:** @read, @stream, @slow +The command returns an array of entries, where each entry is itself an array consisting of an ID and a field-value pair list. +The entries are ordered from the highest ID to the lowest ID, within the specified range. -**XREVRANGE** is almost same as **XRANGE**. The only difference -is it returns the entries in reverse order, and also takes the -start-end range in reverse order (as you can notice in the -syntax). Reverse order means the range starts with higher IDs -and ends with lower IDs. +## Code Examples + +### Basic Example + +Retrieve entries from a stream within a specified ID range in reverse order: -So for instance, the following command returns all the -elements from the higher ID to the lower ID: ```shell -XREVRANGE mystream + - +# Adding entries to a stream. +dragonfly$> XADD mystream * field1 value1 +"1617825600000-0" +dragonfly$> XADD mystream * field1 value2 +"1617826800000-0" +dragonfly$> XADD mystream * field1 value3 +"1617828000000-0" + +# Retrieve entries in reverse order. +dragonfly$> XREVRANGE mystream 1617828000000-0 1617825600000-0 +1) 1) "1617828000000-0" + 2) 1) "field1" + 2) "value3" +2) 1) "1617826800000-0" + 2) 1) "field1" + 2) "value2" +3) 1) "1617825600000-0" + 2) 1) "field1" + 2) "value1" ``` -**COUNT** option limits the number of entries returned by -the command. +### Using `COUNT` Option + +Retrieve a limited number of entries in reverse order using the `COUNT` option: + ```shell -XREVRANGE mystream + - COUNT 2 -``` +# Adding more entries to the stream. +dragonfly$> XADD mystream * field1 value4 +"1617829200000-0" -## Return -[Array reply](https://redis.io/docs/reference/protocol-spec/#arrays) +# Retrieve only the two most recent entries. +dragonfly$> XREVRANGE mystream + - COUNT 2 +1) 1) "1617829200000-0" + 2) 1) "field1" + 2) "value4" +2) 1) "1617828000000-0" + 2) 1) "field1" + 2) "value3" +``` -The return format is exactly same as **XRANGE** command. +### Applications in Event Logging -## Example +Consider a log stream where you want to fetch the most recent events up to a specific moment: ```shell -dragonfly> XREVRANGE mystream + - -1) 1) "1687927770804-0" - 2) 1) "k3" - 2) "v3" -2) 1) "1687927767585-0" - 2) 1) "k2" - 2) "v2" -3) 1) "1687927765116-0" - 2) 1) "k" - 2) "v" +# Adding log entries. +dragonfly$> XADD logs * event login +"1617830000000-0" +dragonfly$> XADD logs * event logout +"1617831000000-0" + +# Retrieve events in reverse order up to a specific ID. +dragonfly$> XREVRANGE logs 1617831000000-0 1617830000000-0 +1) 1) "1617831000000-0" + 2) 1) "event" + 2) "logout" +2) 1) "1617830000000-0" + 2) 1) "event" + 2) "login" ``` + +## Best Practices + +- Utilize `COUNT` to limit the number of entries returned, optimizing performance and reducing memory usage when processing large streams. +- Ensure that your specified `end` and `start` IDs correctly encapsulate the desired time or event window to avoid unexpected results. + +## Common Mistakes + +- Misinterpreting the order of `start` and `end`: since `XREVRANGE` operates in reverse, `end` should be greater than `start`. +- Overlooking the fact that stream IDs are ordered lexicographically, which can affect how ranges are defined and queried. + +## FAQs + +### What happens if the stream key does not exist? + +If the stream key does not exist, `XREVRANGE` returns an empty array. + +### Can negative indexes be used for `end` and `start` parameters? + +Stream IDs are lexicographic and do not support negative indexes. Instead, ranges define specific or approximate position and order within the stream based on IDs. \ No newline at end of file diff --git a/docs/command-reference/stream/xsetid.md b/docs/command-reference/stream/xsetid.md index c1f3d4f..5ef812b 100644 --- a/docs/command-reference/stream/xsetid.md +++ b/docs/command-reference/stream/xsetid.md @@ -1,5 +1,5 @@ --- -description: Learn how to use Redis XSETID to set the last delivered ID for streams. +description: Learn how to use Redis XSETID to set the last delivered ID for streams. --- import PageTitle from '@site/src/components/PageTitle'; @@ -8,19 +8,84 @@ import PageTitle from '@site/src/components/PageTitle'; +## Introduction + +In Dragonfly, as well as in Redis and Valkey, the `XSETID` command is used to set the ID of an existing consumer group in a stream. +This command allows you to manually manage the last delivered ID of a consumer group, which can be particularly useful in scenarios where you need precise control over message processing. + ## Syntax - XSETID key last-id +```shell +XSETID key groupname id +``` + +## Parameter Explanations + +- `key`: The key of the stream for which you want to set the consumer group ID. +- `groupname`: The name of the consumer group whose last delivered ID you want to set. +- `id`: The new ID you want to assign as the last delivered ID of the consumer group. This should be a valid stream ID. + +## Return Values + +The command returns `OK` if the ID for the consumer group is successfully set. + +## Code Examples + +### Basic Example + +Set the last delivered ID for a consumer group: + +```shell +dragonfly$> XGROUP CREATE mystream mygroup 0 +OK +# Assuming some entries have been added to the stream... +dragonfly$> XSETID mystream mygroup 1526569495631-0 +OK +``` + +### Changing the Last Delivered ID + +Imagine you have read up to a certain message ID, and you want to mark this as the new last delivered ID: + +```shell +# Add some entries to the stream +dragonfly$> XADD mystream * message "Hello World" +1526569495631-0 +dragonfly$> XADD mystream * message "Another Message" +1526569495632-0 + +# Now change the last delivered ID for `mygroup` +dragonfly$> XSETID mystream mygroup 1526569495632-0 +OK +``` + +This ensures that the consumer group acknowledges up to the specified message ID. + +### Error Handling + +Attempt to set a last delivered ID that doesn't exist: + +```shell +dragonfly$> XSETID mystream mygroup 9999999999999-0 +(error) ERR The specified ID is greater than the maximal ID in the stream +``` + +## Best Practices + +- Ensure the provided `id` exists in the stream; otherwise, the command will return an error. +- Use `XSETID` carefully, as setting the wrong ID can lead to message delivery inconsistencies. + +## Common Mistakes + +- Setting an ID that does not exist in the stream. +- Overwriting the last delivered ID in a way that skips some messages unintentionally. -**Time Complexity:** O(1) +## FAQs -**ACL categories:** @write, @stream, @fast +### What happens if the consumer group does not exist? -**XSETID** sets the last id of the specified stream. **** -must exists before executing the command. The **** can't -be smaller than target stream top entry. +If the consumer group does not exist, `XSETID` will return an error because there is no group to set the ID for. -## Return +### Can I use wildcard or relative IDs? -[Simple string reply](https://redis.io/docs/reference/protocol-spec/#simple-strings): -**OK** on success. +No, you must specify an exact stream ID that exists in the stream. diff --git a/docs/command-reference/stream/xtrim.md b/docs/command-reference/stream/xtrim.md index 29fb4ac..39ed77a 100644 --- a/docs/command-reference/stream/xtrim.md +++ b/docs/command-reference/stream/xtrim.md @@ -8,72 +8,98 @@ import PageTitle from '@site/src/components/PageTitle'; +## Introduction + +In Dragonfly, as well as in Redis and Valkey, the `XTRIM` command is used to manage the length of a stream by trimming entries. +It helps in controlling memory usage by removing old entries from the stream. +`XTRIM` can be useful for applications like logging, where recent data is more critical than historical data. + ## Syntax - XTRIM key [= | ~] threshold [LIMIT count] +```shell +XTRIM key [MAXLEN | MINID] [~ | =] threshold +``` + +## Parameter Explanations -**Time Complexity:** O(N), with N being the number of evicted entries. -Constant times are very small however, since entries are organized in -macro nodes containing multiple entries that can be released with a -single deallocation. +- `key`: The key of the stream that you want to trim. +- `MAXLEN`: Trims the stream to ensure it does not exceed a specified number of entries. +- `MINID`: Trims entries with IDs less than a specified threshold. +- `~`: Approximates the trimming operation to improve performance. +- `=`: Ensures the trimming is precisely executed to the specified threshold. +- `threshold`: The maximum number of entries or the entry ID threshold. -**ACL categories:** @write, @stream, @slow +## Return Values -**XTRIM** works same like how trimming works in **XADD** (when option -specified). But instead of adding a new entry, **XTRIM** focuses -completely on trimming entries. **** is the stream name of -which the entries need to be trimmed. +The command returns the number of entries removed from the stream. -With **XTRIM**, you can either specify **MAXLEN** or **MINID** to -control the stratgy to trim. +## Code Examples -**MAXLEN** ensures that the number of entries in a stream -doesn't exceed a certain limit. **MINID** on the other hand -ensures that entries with IDs less than the specified **MINID** -get deleted. These two options take a *threshold* denoting the -length (in case of **MAXLEN**) or ID (in case of **MINID**). +### Basic MAXLEN Usage -Dragonfly gives two options to control the trimming nature. -"**=**" argument tells the command to do the exact trimming of -entries. Whereas **~** argument tells the command to do -the approximate trimming. That is, it is upto the command to -decide how many entries need to be deleted. So a stream may -have **few more** entries than the given *threshold* (due to -performance reasons). It is more efficient than exact trimming. -By default, exact trimming is used when no options are specified. +Trim a stream to a maximum of three entries: ```shell -dragonfly> XTRIM mystream MAXLEN ~ 100 -(integer) 98 +dragonfly$> XADD mystream * field1 value1 +"1609459200000-0" +dragonfly$> XADD mystream * field2 value2 +"1609459200001-0" +dragonfly$> XADD mystream * field3 value3 +"1609459200002-0" +dragonfly$> XADD mystream * field4 value4 +"1609459200003-0" +dragonfly$> XTRIM mystream MAXLEN 3 +(integer) 1 ``` -**LIMIT** is useful when you want to limit the number of delete -operations used for **MAXLEN** or **MINID** (in case of approximate -trimming). When **LIMIT** isn't specified, the default value of -*100 \* the number of entries* in a macro node will be implicitly -used as the count. Specifying the value 0 as count disables the -limiting mechanism entirely. +### Using MINID to Trim + +Remove all entries with IDs less than a specific timestamp: ```shell -dragonfly> XTRIM mystream MAXLEN ~ 100 LIMIT 2 -(integer) 2 +dragonfly$> XADD mystream * field1 value1 +"1609459200000-0" +dragonfly$> XADD mystream * field2 value2 +"1609459200001-0" +dragonfly$> XADD mystream * field3 value3 +"1609459200002-0" +dragonfly$> XTRIM mystream MINID 1609459200001-0 +(integer) 1 ``` -## Return -[Integer reply](https://redis.io/docs/reference/protocol-spec/#integers): -The number of entries deleted from the stream. +### Approximative Trimming with `~` -## Example +Improve performance by approximating the trim operation: ```shell -dragonfly> XADD mystream * name John -"1687921762755-0" -dragonfly> XADD mystream * name Alice -"1687924580856-0" -dragonfly> XADD mystream * name Bob -"1687924609465-0" -dragonfly> XLEN mystream -(integer) 3 -dragonfly> XTRIM mystream MAXLEN = 2 +dragonfly$> XADD mystream * field1 value1 +"1609459200000-0" +dragonfly$> XADD mystream * field2 value2 +"1609459200001-0" +dragonfly$> XADD mystream * field3 value3 +"1609459200002-0" +dragonfly$> XADD mystream * field4 value4 +"1609459200003-0" +dragonfly$> XTRIM mystream MAXLEN ~ 2 (integer) 2 ``` + +## Best Practices + +- Use approximative trimming with `~` for better performance on large streams, when precision is not critical. +- Regularly trim streams if your application generates a significant amount of data, to manage memory usage efficiently. + +## Common Mistakes + +- Confusing `MAXLEN` and `MINID` — `MAXLEN` specifies a count of entries, whereas `MINID` specifies an entry ID threshold. +- Not considering the trade-off between approximation (`~`) and precision (`=`) in terms of performance and accuracy. + +## FAQs + +### What happens if the stream is empty or the key does not exist? + +If the stream is empty or the key does not exist, `XTRIM` returns `0` since no entries are removed. + +### Can I use negative indexes for `threshold`? + +No, `threshold` values for both `MAXLEN` and `MINID` must be positive integers. Negative values are not applicable. \ No newline at end of file