diff --git a/Makefile b/Makefile index 11decf20a..bd101788d 100644 --- a/Makefile +++ b/Makefile @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -REDIS_VERSION:=4.0.11 +REDIS_VERSION:=5.0-rc4 SPRING_PROFILE?=ci ####### diff --git a/src/main/asciidoc/new-features.adoc b/src/main/asciidoc/new-features.adoc index d349f4717..ff638d910 100644 --- a/src/main/asciidoc/new-features.adoc +++ b/src/main/asciidoc/new-features.adoc @@ -3,6 +3,11 @@ This section briefly covers items that are new and noteworthy in the latest releases. +[[new-in-2.2.0]] +== New in Spring Data Redis 2.2 + +* <> + [[new-in-2.1.0]] == New in Spring Data Redis 2.1 diff --git a/src/main/asciidoc/reference/redis-streams.adoc b/src/main/asciidoc/reference/redis-streams.adoc new file mode 100644 index 000000000..b6c627bb2 --- /dev/null +++ b/src/main/asciidoc/reference/redis-streams.adoc @@ -0,0 +1,190 @@ +[[redis.streams]] += Redis Streams + +Redis Streams model a log data structure in an abstract approach. Typically, logs are append-only data structures and are consumed from the beginning on, at a random position, or by streaming new messages. + +NOTE: Learn more about Redis Streams in the https://redis.io/topics/streams-intro[Redis reference documentation]. + +Redis Streams can be roughly divided into two areas of functionality: + +* Appending or sending messages +* Consuming or receiving messages + +Although this pattern has similarities to <>, the main difference lies in the persistence of messages and how they are consumed. + +While Pub/Sub relies on the broadcasting of transient messages (i.e. if you don't listen, you miss a message), Redis Stream use a persistent, append-only data type that retains messages until the stream is trimmed. Another difference in consumption is that Pub/Sub registers a server-side subscription. Redis pushes arriving messages to the client while Redis Streams require active polling. + +The `org.springframework.data.redis.connection` and `org.springframework.data.redis.stream` packages provide the core functionality for Redis Streams. + + +[[redis.streams.publish]] +== Appending (Sending Messages) + +To publish a message, you can use, as with the other operations, either the low-level `RedisConnection` or the high-level `StreamOperations`. Both entities offer the `add` (`xAdd`) method, which accepts the message and the destination stream as arguments. While `RedisConnection` requires raw data (array of bytes), the `StreamOperations` lets arbitrary objects be passed in as messages, as shown in the following example: + +[source,java] +---- +// append message through connection +RedisConnection con = … +byte[] stream = … +Map msg = … +con.xAdd(stream, msg); + +// append message through RedisTemplate +RedisTemplate template = … +template.streamOps().add("my-stream", Collections.singletonMap("Hello", "World!")); +---- + +Stream messages carry a `Map`, key-value tuples, as their payload. Appending a message to a stream returns the message Id that can be used as further reference. + +[[redis.streams.receive]] +== Consuming (Receiving Messages) + +On the consuming side, one can consume one or multiple streams. Redis Streams provide read commands that allow consumption of the stream from an arbitrary position (random access) within the known stream content and beyond the stream end to consume new stream messages. + +At the low-level, `RedisConnection` offers the `xRead` and `xReadGroup` methods that map the Redis commands for reading and reading within a consumer group, respectively. Note that multiple streams can be used as arguments. + +NOTE: Subscription commands in Redis can be blocking. That is, calling `xRead` on a connection causes the current thread to block as it starts waiting for messages. The thread is released only if the read command times out or receives a message. + +To consume stream messages, one can either poll for messages in application code, or use one of the two <>, the imperative or the reactive one. Each time a new message arrives, the container notifies the application code. + +[[redis.streams.receive.synchronous]] +=== Synchronous reception + +While stream consumption is typically associated with asynchronous processing, it is possible to consume messages synchronously. The overloaded `StreamOperations.read(…)` methods provide this functionality. During a synchronous receive, the calling thread potentially blocks until a message becomes available. The property `StreamReadOptions.block` specifies how long the receiver should wait before giving up waiting for a message. + +[source,java] +---- +// Read message through RedisTemplate +RedisTemplate template = … + +List> messages = template.streamOps().read(StreamReadOptions.empty().count(2), + StreamOffset.create("my-stream", ReadOffset.latest())); + +List> messages = template.streamOps().read(Consumer.from("my-group", "my-consumer"), + StreamReadOptions.empty().count(2), + StreamOffset.create("my-stream", ReadOffset.lastConsumed())) +---- + +[[redis.streams.receive.containers]] +=== Asynchronous reception through Message Listener Containers + +Due to its blocking nature, low-level polling is not attractive, as it requires connection and thread management for every single consumer. To alleviate this problem, Spring Data offers message listeners, which do all the heavy lifting. If you are familiar with EJB and JMS, you should find the concepts familiar, as it is designed to be as close as possible to the support in Spring Framework and its message-driven POJOs (MDPs). + +Spring Data ships with two implementations tailored to the used programming model: + +* `StreamMessageListenerContainer` acts as message listener container for imperative programming models. It is used to consume messages from a Redis Stream and drive the `StreamListener` instances that are injected into it. +* `StreamReceiver` provides a reactive variant of a message listener. It is used to consume messages from a Redis Stream as potentially infinite stream and emit stream messages through a `Flux`. + +`StreamMessageListenerContainer` and `StreamReceiver` are responsible for all threading of message reception and dispatch into the listener for processing. A message listener container/receiver is the intermediary between an MDP and a messaging provider and takes care of registering to receive messages, resource acquisition and release, exception conversion, and the like. This lets you as an application developer write the (possibly complex) business logic associated with receiving a message (and reacting to it) and delegates boilerplate Redis infrastructure concerns to the framework. + +Both containers allow runtime configuration changes so that you can add or remove subscriptions while an application is running without the need for a restart. Additionally, the container uses a lazy subscription approach, using a `RedisConnection` only when needed. If all the listeners are unsubscribed, it automatically performs a cleanup, and the thread is released. + + +==== Imperative `StreamMessageListenerContainer` + +In a fashion similar to a Message-Driven Bean (MDB) in the EJB world, the Stream-Driven POJO (SDP) acts as a receiver for Stream messages. The one restriction on an SDP is that it must implement the `org.springframework.data.redis.stream.StreamListener` interface. Please also be aware that in the case where your POJO receives messages on multiple threads, it is important to ensure that your implementation is thread-safe. + +[source,java] +---- +class ExampleStreamListener implements StreamListener { + + @Override + public void onMessage(StreamMessage message) { + + System.out.println("MessageId: " + message.getId()); + System.out.println("Stream: " + message.getStream()); + System.out.println("Body: " + message.getBody()); + } +} +---- + +`StreamListener` represents a functional interface so implementations can be rewritten using their Lambda form: + +[source,java] +---- +message -> { + + System.out.println("MessageId: " + message.getId()); + System.out.println("Stream: " + message.getStream()); + System.out.println("Body: " + message.getBody()); +}; +---- + +Once you’ve implemented your `StreamListener`, it’s time to create a message listener container and register a subscription: + +[source,java] +---- +RedisConnectionFactory connectionFactory = … +StreamListener streamListener = … + +StreamMessageListenerContainerOptions containerOptions = StreamMessageListenerContainerOptions + .builder().pollTimeout(Duration.ofMillis(100)).build(); + +StreamMessageListenerContainer container = StreamMessageListenerContainer.create(connectionFactory, + containerOptions); + +Subscription subscription = container.receive(StreamOffset.create("my-stream", ReadOffset.from("0-0")), streamListener); +---- + +Please refer to the Javadoc of the various message listener containers for a full description of the features supported by each implementation. + +==== Reactive `StreamReceiver` + +Reactive consumption of streaming data sources typically happens through a `Flux` of events or messages. The reactive receiver implementation is provided with `StreamReceiver` and its overloaded `receive(…)` messages. The reactive approach requires fewer infrastructure resources such as threads in comparison to `StreamMessageListenerContainer` as it is leveraging threading resources provided by the driver. The receiving stream is a demand-driven publisher of ``StreamMessage``: + +[source,java] +---- +Flux> messages = … + +return messages.doOnNext(it -> { + System.out.println("MessageId: " + message.getId()); + System.out.println("Stream: " + message.getStream()); + System.out.println("Body: " + message.getBody()); +}); +---- + +Now we need to create the `StreamReceiver` and register a subscription to consume stream messages: + +[source,java] +---- +ReactiveRedisConnectionFactory connectionFactory = … + +StreamReceiverOptions options = StreamReceiverOptions.builder().pollTimeout(Duration.ofMillis(100)) + .build(); +StreamReceiver receiver = StreamReceiver.create(connectionFactory, options); + +Flux> messages = receiver.receive(StreamOffset.create("my-stream", ReadOffset.from("0-0"))); +---- + +Please refer to the Javadoc of the various message listener containers for a full description of the features supported by each implementation. + +NOTE: Demand-driven consumption uses backpressure signals to activate and deactivate polling. `StreamReceiver` subscriptions pause polling if the demand is satisfied until subscribers signal further demand. Depending on the `ReadOffset` strategy, this can cause messages to be skipped. + +[[redis.streams.receive.readoffset]] +=== `ReadOffset` strategies + +Stream read operations accept a read offset specification to consume messages from the given offset on. `ReadOffset` represents the read offset specification. Redis supports 3 variants of offsets, depending on whether you consume the stream standalone or within a consumer group: + +* `ReadOffset.latest()` – Read the latest message. +* `ReadOffset.from(…)` – Read after a specific message Id. +* `ReadOffset.lastConsumed()` – Read after the last consumed message Id (consumer-group only). + +In the context of a message container-based consumption, we need to advance (or increment) the read offset when consuming a message. Advancing depends on the requested `ReadOffset` and consumption mode (with/without consumer groups). The following matrix explains how containers advance `ReadOffset`: + +.ReadOffset Advancing +[options="header,footer,autowidth"] +|=== +| Read offset | Standalone | Consumer Group +| Latest | Read latest message | Read latest message +| Specific Message Id | Use last seen message as the next MessageId | Use last seen message as the next MessageId +| Last Consumed | Use last seen message as the next MessageId | Last consumed message as per consumer group +|=== + +Reading from a specific message id and the last consumed message can be considered safe operations that ensure consumption of all messages that were appended to the stream. +Using the latest message for read can skip messages that were added to the stream while the poll operation was in the state of dead time. Polling introduces a dead time in which messages can arrive between individual polling commands. Stream consumption is not a linear contiguous read but split into repeating `XREAD` calls. + +[[redis.streams.receive.serialization]] +=== Serialization + +TBD. diff --git a/src/main/asciidoc/reference/redis.adoc b/src/main/asciidoc/reference/redis.adoc index 19c159e5b..92acb6eff 100644 --- a/src/main/asciidoc/reference/redis.adoc +++ b/src/main/asciidoc/reference/redis.adoc @@ -482,6 +482,8 @@ NOTE: Flattening requires all property names to not interfere with the JSON path :leveloffset: 2 include::{referenceDir}/redis-messaging.adoc[] +include::{referenceDir}/redis-streams.adoc[] + include::{referenceDir}/redis-transactions.adoc[] include::{referenceDir}/pipelining.adoc[] diff --git a/src/main/java/org/springframework/data/redis/connection/DataType.java b/src/main/java/org/springframework/data/redis/connection/DataType.java index 29ad5cd0c..6ccabe390 100644 --- a/src/main/java/org/springframework/data/redis/connection/DataType.java +++ b/src/main/java/org/springframework/data/redis/connection/DataType.java @@ -13,7 +13,6 @@ * See the License for the specific language governing permissions and * limitations under the License. */ - package org.springframework.data.redis.connection; import java.util.EnumSet; @@ -24,12 +23,17 @@ import java.util.concurrent.ConcurrentHashMap; * Enumeration of the Redis data types. * * @author Costin Leau + * @author Mark Paluch */ public enum DataType { - NONE("none"), STRING("string"), LIST("list"), SET("set"), ZSET("zset"), HASH("hash"); + NONE("none"), STRING("string"), LIST("list"), SET("set"), ZSET("zset"), HASH("hash"), + /** + * @since 2.2 + */ + STREAM("stream"); - private static final Map codeLookup = new ConcurrentHashMap<>(6); + private static final Map codeLookup = new ConcurrentHashMap<>(7); static { for (DataType type : EnumSet.allOf(DataType.class)) diff --git a/src/main/java/org/springframework/data/redis/connection/DefaultStringRedisConnection.java b/src/main/java/org/springframework/data/redis/connection/DefaultStringRedisConnection.java index 24fdc173a..6e787f7b0 100644 --- a/src/main/java/org/springframework/data/redis/connection/DefaultStringRedisConnection.java +++ b/src/main/java/org/springframework/data/redis/connection/DefaultStringRedisConnection.java @@ -16,19 +16,10 @@ package org.springframework.data.redis.connection; import java.time.Duration; -import java.util.ArrayList; -import java.util.Collection; -import java.util.HashMap; -import java.util.LinkedHashMap; -import java.util.LinkedList; -import java.util.List; -import java.util.Map; -import java.util.Map.Entry; -import java.util.Properties; -import java.util.Queue; -import java.util.Set; +import java.util.*; import java.util.Map.Entry; import java.util.concurrent.TimeUnit; +import java.util.function.IntFunction; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; @@ -78,6 +69,8 @@ public class DefaultStringRedisConnection implements StringRedisConnection, Deco private MapConverter byteMapToStringMap = new MapConverter<>(bytesToString); private SetConverter byteSetToStringSet = new SetConverter<>(bytesToString); private Converter>, GeoResults>> byteGeoResultsToStringGeoResults; + private Converter, StreamMessage> byteStreamMessageToStringStreamMessageConverter; + private ListConverter, StreamMessage> byteStreamMessageListToStringStreamMessageConverter; @SuppressWarnings("rawtypes") private Queue pipelineConverters = new LinkedList<>(); @SuppressWarnings("rawtypes") private Queue txConverters = new LinkedList<>(); @@ -145,6 +138,11 @@ public class DefaultStringRedisConnection implements StringRedisConnection, Deco this.delegate = connection; this.serializer = serializer; this.byteGeoResultsToStringGeoResults = Converters.deserializingGeoResultsConverter(serializer); + + this.byteStreamMessageToStringStreamMessageConverter = Converters + .deserializingStreamMessageConverter(serializer::deserialize); + this.byteStreamMessageListToStringStreamMessageConverter = new ListConverter<>( + byteStreamMessageToStringStreamMessageConverter); } /* @@ -1205,7 +1203,7 @@ public class DefaultStringRedisConnection implements StringRedisConnection, Deco @Nullable @Override public Long bitPos(byte[] key, boolean bit, org.springframework.data.domain.Range range) { - return convertAndReturn(delegate.bitPos(key, bit, range), identityConverter); + return convertAndReturn(delegate.bitPos(key, bit, range), identityConverter); } /* @@ -1746,6 +1744,13 @@ public class DefaultStringRedisConnection implements StringRedisConnection, Deco return serializer.serialize(data); } + @SuppressWarnings("unchecked") + private StreamOffset[] serialize(StreamOffset[] offsets) { + + return Arrays.stream(offsets).map(it -> StreamOffset.create(serialize(it.getKey()), it.getOffset())) + .toArray((IntFunction[]>) StreamOffset[]::new); + } + private byte[][] serializeMulti(String... keys) { if (keys == null) { @@ -3603,6 +3608,236 @@ public class DefaultStringRedisConnection implements StringRedisConnection, Deco delegate.migrate(key, target, dbIndex, option, timeout); } + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.StringRedisConnection#xAck(java.lang.String, java.lang.String, java.lang.String[]) + */ + @Override + public Long xAck(String key, String group, String... messageIds) { + return convertAndReturn(delegate.xAck(this.serialize(key), group, messageIds), identityConverter); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.StringRedisConnection#xAdd(java.lang.String, java.util.Map) + */ + @Override + public String xAdd(String key, Map body) { + return convertAndReturn(delegate.xAdd(serialize(key), serialize(body)), identityConverter); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.StringRedisConnection#xDel(java.lang.String, java.lang.String[]) + */ + @Override + public Long xDel(String key, String... messageIds) { + return convertAndReturn(delegate.xDel(serialize(key), messageIds), identityConverter); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.StringRedisConnection#xGroupCreate(java.lang.String, org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset, java.lang.String) + */ + @Override + public String xGroupCreate(String key, ReadOffset readOffset, String group) { + return convertAndReturn(delegate.xGroupCreate(serialize(key), readOffset, group), identityConverter); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.StringRedisConnection#xGroupDelConsumer(java.lang.String, org.springframework.data.redis.connection.RedisStreamCommands.Consumer) + */ + @Override + public Boolean xGroupDelConsumer(String key, Consumer consumer) { + return convertAndReturn(delegate.xGroupDelConsumer(serialize(key), consumer), identityConverter); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.StringRedisConnection#xGroupDestroy(java.lang.String, java.lang.String) + */ + @Override + public Boolean xGroupDestroy(String key, String group) { + return convertAndReturn(delegate.xGroupDestroy(serialize(key), group), identityConverter); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.StringRedisConnection#xLen(java.lang.String) + */ + @Override + public Long xLen(String key) { + return convertAndReturn(delegate.xLen(serialize(key)), identityConverter); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.StringRedisConnection#xRange(java.lang.String, org.springframework.data.domain.Range, org.springframework.data.redis.connection.RedisZSetCommands.Limit) + */ + @Override + public List> xRange(String key, org.springframework.data.domain.Range range, + Limit limit) { + + return convertAndReturn(delegate.xRange(serialize(key), range, limit), + byteStreamMessageListToStringStreamMessageConverter); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.StringRedisConnection#xReadAsString(org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions, org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset[]) + */ + @Override + public List> xReadAsString(StreamReadOptions readOptions, + StreamOffset... streams) { + return convertAndReturn(delegate.xRead(readOptions, serialize(streams)), + byteStreamMessageListToStringStreamMessageConverter); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.StringRedisConnection#xReadGroupAsString(org.springframework.data.redis.connection.RedisStreamCommands.Consumer, org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions, org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset[]) + */ + @Override + public List> xReadGroupAsString(Consumer consumer, StreamReadOptions readOptions, + StreamOffset... streams) { + + return convertAndReturn(delegate.xReadGroup(consumer, readOptions, serialize(streams)), + byteStreamMessageListToStringStreamMessageConverter); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.StringRedisConnection#xRevRange(java.lang.String, org.springframework.data.domain.Range, org.springframework.data.redis.connection.RedisZSetCommands.Limit) + */ + @Override + public List> xRevRange(String key, org.springframework.data.domain.Range range, + Limit limit) { + + return convertAndReturn(delegate.xRevRange(serialize(key), range, limit), + byteStreamMessageListToStringStreamMessageConverter); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.StringRedisConnection#xTrim(java.lang.String, long) + */ + @Override + public Long xTrim(String key, long count) { + return convertAndReturn(delegate.xTrim(serialize(key), count), identityConverter); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xAck(byte[], java.lang.String, java.lang.String[]) + */ + @Override + public Long xAck(byte[] key, String group, String... messageIds) { + return delegate.xAck(key, group, messageIds); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xAdd(byte[], java.util.Map) + */ + @Override + public String xAdd(byte[] key, Map body) { + return delegate.xAdd(key, body); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xDel(byte[], java.lang.String[]) + */ + @Override + public Long xDel(byte[] key, String... messageIds) { + return delegate.xDel(key, messageIds); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xGroupCreate(byte[], org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset, java.lang.String) + */ + @Override + public String xGroupCreate(byte[] key, ReadOffset readOffset, String group) { + return delegate.xGroupCreate(key, readOffset, group); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xGroupDelConsumer(byte[], org.springframework.data.redis.connection.RedisStreamCommands.Consumer) + */ + @Override + public Boolean xGroupDelConsumer(byte[] key, Consumer consumer) { + return delegate.xGroupDelConsumer(key, consumer); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xGroupDestroy(byte[], java.lang.String) + */ + @Override + public Boolean xGroupDestroy(byte[] key, String group) { + return delegate.xGroupDestroy(key, group); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xLen(byte[]) + */ + @Override + public Long xLen(byte[] key) { + return delegate.xLen(key); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xRange(byte[], org.springframework.data.domain.Range, org.springframework.data.redis.connection.RedisZSetCommands.Limit) + */ + @Override + public List> xRange(byte[] key, org.springframework.data.domain.Range range, + Limit limit) { + return delegate.xRange(key, range, limit); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xRead(org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions, org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset[]) + */ + @Override + public List> xRead(StreamReadOptions readOptions, StreamOffset... streams) { + return delegate.xRead(readOptions, streams); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xReadGroup(org.springframework.data.redis.connection.RedisStreamCommands.Consumer, org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions, org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset[]) + */ + @Override + public List> xReadGroup(Consumer consumer, StreamReadOptions readOptions, + StreamOffset... streams) { + return delegate.xReadGroup(consumer, readOptions, streams); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xRevRange(byte[], org.springframework.data.domain.Range, org.springframework.data.redis.connection.RedisZSetCommands.Limit) + */ + @Override + public List> xRevRange(byte[] key, org.springframework.data.domain.Range range, + Limit limit) { + return delegate.xRevRange(key, range, limit); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xTrim(byte[], long) + */ + @Override + public Long xTrim(byte[] key, long count) { + return delegate.xTrim(key, count); + } + /** * Specifies if pipelined and tx results should be deserialized to Strings. If false, results of * {@link #closePipeline()} and {@link #exec()} will be of the type returned by the underlying connection diff --git a/src/main/java/org/springframework/data/redis/connection/DefaultedRedisConnection.java b/src/main/java/org/springframework/data/redis/connection/DefaultedRedisConnection.java index 9913529d0..a3822f744 100644 --- a/src/main/java/org/springframework/data/redis/connection/DefaultedRedisConnection.java +++ b/src/main/java/org/springframework/data/redis/connection/DefaultedRedisConnection.java @@ -428,6 +428,124 @@ public interface DefaultedRedisConnection extends RedisConnection { return stringCommands().strLen(key); } + // STREAM COMMANDS + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default Long xAck(byte[] key, String group, String... messageIds) { + return streamCommands().xAck(key, group, messageIds); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default String xAdd(byte[] key, Map body) { + return streamCommands().xAdd(key, body); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default Long xDel(byte[] key, String... messageIds) { + return streamCommands().xDel(key, messageIds); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default String xGroupCreate(byte[] key, ReadOffset readOffset, String group) { + return streamCommands().xGroupCreate(key, readOffset, group); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default Boolean xGroupDelConsumer(byte[] key, Consumer consumer) { + return streamCommands().xGroupDelConsumer(key, consumer); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default Boolean xGroupDestroy(byte[] key, String group) { + return streamCommands().xGroupDestroy(key, group); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default Long xLen(byte[] key) { + return streamCommands().xLen(key); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default List> xRange(byte[] key, org.springframework.data.domain.Range range) { + return streamCommands().xRange(key, range); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default List> xRange(byte[] key, org.springframework.data.domain.Range range, + Limit limit) { + return streamCommands().xRange(key, range, limit); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default List> xRead(StreamOffset... streams) { + return streamCommands().xRead(streams); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default List> xRead(StreamReadOptions readOptions, StreamOffset... streams) { + return streamCommands().xRead(readOptions, streams); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default List> xReadGroup(Consumer consumer, StreamOffset... streams) { + return streamCommands().xReadGroup(consumer, streams); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default List> xReadGroup(Consumer consumer, StreamReadOptions readOptions, + StreamOffset... streams) { + return streamCommands().xReadGroup(consumer, readOptions, streams); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default List> xRevRange(byte[] key, + org.springframework.data.domain.Range range) { + return streamCommands().xRevRange(key, range); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default List> xRevRange(byte[] key, org.springframework.data.domain.Range range, + Limit limit) { + return streamCommands().xRevRange(key, range, limit); + } + + /** @deprecated in favor of {@link RedisConnection#streamCommands()}}. */ + @Override + @Deprecated + default Long xTrim(byte[] key, long count) { + return streamCommands().xTrim(key, count); + } + // LIST COMMANDS /** @deprecated in favor of {@link RedisConnection#listCommands()}}. */ diff --git a/src/main/java/org/springframework/data/redis/connection/ReactiveClusterStreamCommands.java b/src/main/java/org/springframework/data/redis/connection/ReactiveClusterStreamCommands.java new file mode 100644 index 000000000..90aa1b70e --- /dev/null +++ b/src/main/java/org/springframework/data/redis/connection/ReactiveClusterStreamCommands.java @@ -0,0 +1,23 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.connection; + +/** + * @author Mark Paluch + * @since 2.2 + */ +public interface ReactiveClusterStreamCommands extends ReactiveStreamCommands { +} diff --git a/src/main/java/org/springframework/data/redis/connection/ReactiveRedisClusterConnection.java b/src/main/java/org/springframework/data/redis/connection/ReactiveRedisClusterConnection.java index b8690247f..a98dc887d 100644 --- a/src/main/java/org/springframework/data/redis/connection/ReactiveRedisClusterConnection.java +++ b/src/main/java/org/springframework/data/redis/connection/ReactiveRedisClusterConnection.java @@ -54,6 +54,9 @@ public interface ReactiveRedisClusterConnection extends ReactiveRedisConnection @Override ReactiveClusterServerCommands serverCommands(); + @Override + ReactiveClusterStreamCommands streamCommands(); + /** * Test the connection to a specific Redis cluster node. * diff --git a/src/main/java/org/springframework/data/redis/connection/ReactiveRedisConnection.java b/src/main/java/org/springframework/data/redis/connection/ReactiveRedisConnection.java index 5c6e28efc..09edcf190 100644 --- a/src/main/java/org/springframework/data/redis/connection/ReactiveRedisConnection.java +++ b/src/main/java/org/springframework/data/redis/connection/ReactiveRedisConnection.java @@ -152,6 +152,14 @@ public interface ReactiveRedisConnection extends Closeable { */ ReactiveServerCommands serverCommands(); + /** + * Get {@link ReactiveStreamCommands}. + * + * @return never {@literal null}. + * @since 2.2 + */ + ReactiveStreamCommands streamCommands(); + /** * Test connection. * diff --git a/src/main/java/org/springframework/data/redis/connection/ReactiveStreamCommands.java b/src/main/java/org/springframework/data/redis/connection/ReactiveStreamCommands.java new file mode 100644 index 000000000..1b1f35ead --- /dev/null +++ b/src/main/java/org/springframework/data/redis/connection/ReactiveStreamCommands.java @@ -0,0 +1,775 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.connection; + +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +import java.nio.ByteBuffer; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; + +import org.reactivestreams.Publisher; +import org.springframework.data.domain.Range; +import org.springframework.data.redis.connection.ReactiveRedisConnection.CommandResponse; +import org.springframework.data.redis.connection.ReactiveRedisConnection.KeyCommand; +import org.springframework.data.redis.connection.ReactiveRedisConnection.NumericResponse; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.connection.RedisZSetCommands.Limit; +import org.springframework.lang.Nullable; +import org.springframework.util.Assert; + +/** + * Stream-specific Redis commands executed using reactive infrastructure. + * + * @author Mark Paluch + * @since 2.2 + */ +public interface ReactiveStreamCommands { + + /** + * {@code XACK} command parameters. + * + * @see Redis Documentation: XACK + */ + class AcknowledgeCommand extends KeyCommand { + + private final @Nullable String group; + private final List messageIds; + + private AcknowledgeCommand(@Nullable ByteBuffer key, @Nullable String group, List messageIds) { + + super(key); + this.group = group; + this.messageIds = messageIds; + } + + /** + * Creates a new {@link AcknowledgeCommand} given a {@link ByteBuffer key}. + * + * @param key must not be {@literal null}. + * @return a new {@link AcknowledgeCommand} for {@link ByteBuffer key}. + */ + public static AcknowledgeCommand stream(ByteBuffer key) { + + Assert.notNull(key, "Key must not be null!"); + + return new AcknowledgeCommand(key, null, Collections.emptyList()); + } + + /** + * Applies the {@literal messageIds}. Constructs a new command instance with all previously configured properties. + * + * @param messageIds must not be {@literal null}. + * @return a new {@link AcknowledgeCommand} with {@literal messageIds} applied. + */ + public AcknowledgeCommand forMessage(String... messageIds) { + + Assert.notNull(messageIds, "MessageIds must not be null!"); + + List newMessageIds = new ArrayList<>(getMessageIds().size() + messageIds.length); + newMessageIds.addAll(getMessageIds()); + newMessageIds.addAll(Arrays.asList(messageIds)); + + return new AcknowledgeCommand(getKey(), getGroup(), newMessageIds); + } + + /** + * Applies the {@literal group}. Constructs a new command instance with all previously configured properties. + * + * @param messageIds must not be {@literal null}. + * @return a new {@link AcknowledgeCommand} with {@literal group} applied. + */ + public AcknowledgeCommand inGroup(String group) { + + Assert.notNull(group, "Group must not be null!"); + + return new AcknowledgeCommand(getKey(), group, getMessageIds()); + } + + @Nullable + public String getGroup() { + return group; + } + + public List getMessageIds() { + return messageIds; + } + } + + /** + * Acknowledge one or more messages as processed. + * + * @param key the stream key. + * @param group name of the consumer group. + * @param messageIds message Id's to acknowledge. + * @return + * @see Redis Documentation: XADD + */ + default Mono xAck(ByteBuffer key, String group, String... messageIds) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(messageIds, "MessageIds must not be null!"); + + return xAck(Mono.just(AcknowledgeCommand.stream(key).inGroup(group).forMessage(messageIds))).next() + .map(NumericResponse::getOutput); + } + + /** + * Acknowledge one or more messages as processed. + * + * @param commands must not be {@literal null}. + * @return + * @see Redis Documentation: XACK + */ + Flux> xAck(Publisher commands); + + /** + * {@code XADD} command parameters. + * + * @see Redis Documentation: XADD + */ + class AddStreamMessage extends KeyCommand { + + private final Map body; + + private AddStreamMessage(@Nullable ByteBuffer key, Map body) { + + super(key); + + this.body = body; + } + + /** + * Creates a new {@link AddStreamMessage} given {@link Map body}. + * + * @param body must not be {@literal null}. + * @return a new {@link AddStreamMessage} for {@link Map}. + */ + public static AddStreamMessage body(Map body) { + + Assert.notNull(body, "GeoLocation must not be null!"); + + return new AddStreamMessage(null, body); + } + + /** + * Applies the Geo set {@literal key}. Constructs a new command instance with all previously configured properties. + * + * @param key must not be {@literal null}. + * @return a new {@link ReactiveGeoCommands.GeoAddCommand} with {@literal key} applied. + */ + public AddStreamMessage to(ByteBuffer key) { + return new AddStreamMessage(key, body); + } + + /** + * @return + */ + public Map getBody() { + return body; + } + } + + /** + * Add stream message with given {@literal body} to {@literal key}. + * + * @param key must not be {@literal null}. + * @param body must not be {@literal null}. + * @return + * @see Redis Documentation: XADD + */ + default Mono xAdd(ByteBuffer key, Map body) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(body, "Body must not be null!"); + + return xAdd(Mono.just(AddStreamMessage.body(body).to(key))).next().map(CommandResponse::getOutput); + } + + /** + * Add stream message with given {@literal body} to {@literal key}. + * + * @param commands must not be {@literal null}. + * @return + * @see Redis Documentation: XADD + */ + Flux> xAdd(Publisher commands); + + /** + * {@code XDEL} command parameters. + * + * @see Redis Documentation: XDEL + */ + class DeleteCommand extends KeyCommand { + + private final List messageIds; + + private DeleteCommand(@Nullable ByteBuffer key, List messageIds) { + + super(key); + this.messageIds = messageIds; + } + + /** + * Creates a new {@link DeleteCommand} given a {@link ByteBuffer key}. + * + * @param key must not be {@literal null}. + * @return a new {@link DeleteCommand} for {@link ByteBuffer key}. + */ + public static DeleteCommand stream(ByteBuffer key) { + + Assert.notNull(key, "Key must not be null!"); + + return new DeleteCommand(key, Collections.emptyList()); + } + + /** + * Applies the {@literal messageIds}. Constructs a new command instance with all previously configured properties. + * + * @param messageIds must not be {@literal null}. + * @return a new {@link DeleteCommand} with {@literal messageIds} applied. + */ + public DeleteCommand messages(String... messageIds) { + + Assert.notNull(messageIds, "MessageIds must not be null!"); + + List newMessageIds = new ArrayList<>(getMessageIds().size() + messageIds.length); + newMessageIds.addAll(getMessageIds()); + newMessageIds.addAll(Arrays.asList(messageIds)); + + return new DeleteCommand(getKey(), newMessageIds); + } + + public List getMessageIds() { + return messageIds; + } + } + + /** + * Removes the specified entries from the stream. Returns the number of items deleted, that may be different from the + * number of IDs passed in case certain IDs do not exist. + * + * @param key the stream key. + * @param messageIds stream message Id's. + * @return number of removed entries. + * @see Redis Documentation: XDEL + */ + default Mono xDel(ByteBuffer key, String... messageIds) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(messageIds, "Body must not be null!"); + + return xDel(Mono.just(DeleteCommand.stream(key).messages(messageIds))).next().map(CommandResponse::getOutput); + } + + /** + * Removes the specified entries from the stream. Returns the number of items deleted, that may be different from the + * number of IDs passed in case certain IDs do not exist. + * + * @param commands must not be {@literal null}. + * @return + * @see Redis Documentation: XDEL + */ + Flux> xDel(Publisher commands); + + /** + * Get the size of the stream stored at {@literal key}. + * + * @param key must not be {@literal null}. + * @return length of the stream. + * @see Redis Documentation: XLEN + */ + default Mono xLen(ByteBuffer key) { + + Assert.notNull(key, "Key must not be null!"); + + return xLen(Mono.just(new KeyCommand(key))).next().map(NumericResponse::getOutput); + } + + /** + * Get the size of the stream stored at {@link KeyCommand#getKey()} + * + * @param commands must not be {@literal null}. + * @return + * @see Redis Documentation: XLEN + */ + Flux> xLen(Publisher commands); + + /** + * {@code XRANGE}/{@code XREVRANGE} command parameters. + * + * @see Redis Documentation: XRANGE + * @see Redis Documentation: XREVRANGE + */ + class RangeCommand extends KeyCommand { + + private final Range range; + private final Limit limit; + + /** + * Creates a new {@link RangeCommand} given a {@code key}, {@link Range}, and {@link Limit}. + * + * @param key must not be {@literal null}. + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + */ + private RangeCommand(ByteBuffer key, Range range, Limit limit) { + + super(key); + this.range = range; + this.limit = limit; + } + + /** + * Creates a new {@link RangeCommand} given a {@code key}. + * + * @param key must not be {@literal null}. + * @return a new {@link RangeCommand} for {@code key}. + */ + public static RangeCommand stream(ByteBuffer key) { + return new RangeCommand(key, Range.unbounded(), Limit.unlimited()); + } + + /** + * Applies a {@link Range}. Constructs a new command instance with all previously configured properties. + * + * @param range must not be {@literal null}. + * @return a new {@link RangeCommand} with {@link Range} applied. + */ + public RangeCommand within(Range range) { + + Assert.notNull(range, "Range must not be null!"); + + return new RangeCommand(getKey(), range, getLimit()); + } + + /** + * Applies a {@code Limit}. Constructs a new command instance with all previously configured properties. + * + * @param count + * @return a new {@link RangeCommand} with {@code limit} applied. + */ + public RangeCommand limit(int count) { + return new RangeCommand(getKey(), range, Limit.unlimited().count(count)); + } + + /** + * Applies a {@code Limit}. Constructs a new command instance with all previously configured properties. + * + * @param limit must not be {@literal null}. + * @return a new {@link RangeCommand} with {@code limit} applied. + */ + public RangeCommand limit(Limit limit) { + + Assert.notNull(limit, "Limit must not be null!"); + + return new RangeCommand(getKey(), range, limit); + } + + /** + * @return the {@link Range}. + */ + public Range getRange() { + return range; + } + + /** + * @return the {@link Limit}. + */ + public Limit getLimit() { + return limit; + } + } + + /** + * Read messages from a stream within a specific {@link Range}. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @return list with members of the resulting stream. + * @see Redis Documentation: XRANGE + */ + default Flux> xRange(ByteBuffer key, Range range) { + return xRange(key, range, Limit.unlimited()); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit}. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + * @return list with members of the resulting stream. + * @see Redis Documentation: XRANGE + */ + default Flux> xRange(ByteBuffer key, Range range, Limit limit) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(range, "Range must not be null!"); + Assert.notNull(limit, "Limit must not be null!"); + + return xRange(Mono.just(RangeCommand.stream(key).within(range).limit(limit))).next() + .flatMapMany(CommandResponse::getOutput); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit}. + * + * @param commands must not be {@literal null}. + * @return + * @see Redis Documentation: XRANGE + */ + Flux>>> xRange( + Publisher commands); + + /** + * {@code XRANGE}/{@code XREVRANGE} command parameters. + * + * @see Redis Documentation: XRANGE + * @see Redis Documentation: XREVRANGE + */ + class ReadCommand { + + private final List> streamOffsets; + private final @Nullable StreamReadOptions readOptions; + private final @Nullable Consumer consumer; + + /** + * @param streamOffsets must not be {@literal null}. + * @param readArgs + * @param consumer + */ + public ReadCommand(List> streamOffsets, @Nullable StreamReadOptions readOptions, + @Nullable Consumer consumer) { + + this.readOptions = readOptions; + this.consumer = consumer; + this.streamOffsets = streamOffsets; + } + + /** + * Creates a new {@link ReadCommand} given a {@link StreamOffset}. + * + * @param streamOffset must not be {@literal null}. + * @return a new {@link ReadCommand} for {@link StreamOffset}. + */ + public static ReadCommand from(StreamOffset streamOffset) { + + Assert.notNull(streamOffset, "StreamOffset must not be null!"); + + return new ReadCommand(Collections.singletonList(streamOffset), StreamReadOptions.empty(), null); + } + + /** + * Creates a new {@link ReadCommand} given a {@link StreamOffset}s. + * + * @param streamOffsets must not be {@literal null}. + * @return a new {@link ReadCommand} for {@link StreamOffset}s. + */ + public static ReadCommand from(StreamOffset... streamOffsets) { + + Assert.notNull(streamOffsets, "StreamOffsets must not be null!"); + + return new ReadCommand(Arrays.asList(streamOffsets), StreamReadOptions.empty(), null); + } + + /** + * Applies a {@link Consumer}. Constructs a new command instance with all previously configured properties. + * + * @param consumer must not be {@literal null}. + * @return a new {@link ReadCommand} with {@link Consumer} applied. + */ + public ReadCommand as(Consumer consumer) { + + Assert.notNull(consumer, "Consumer must not be null!"); + + return new ReadCommand(getStreamOffsets(), getReadOptions(), consumer); + } + + /** + * Applies a {@link Consumer}. Constructs a new command instance with all previously configured properties. + * + * @param consumer must not be {@literal null}. + * @return a new {@link ReadCommand} with {@link Consumer} applied. + */ + public ReadCommand withOptions(StreamReadOptions options) { + + Assert.notNull(options, "StreamReadOptions must not be null!"); + + return new ReadCommand(getStreamOffsets(), options, getConsumer()); + } + + public List> getStreamOffsets() { + return streamOffsets; + } + + @Nullable + public StreamReadOptions getReadOptions() { + return readOptions; + } + + @Nullable + public Consumer getConsumer() { + return consumer; + } + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param stream the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREAD + */ + default Flux> xRead(StreamOffset stream) { + return xRead(StreamReadOptions.empty(), new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param streams the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREAD + */ + default Flux> xRead(StreamOffset... streams) { + return xRead(StreamReadOptions.empty(), streams); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param readOptions read arguments. + * @param stream the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREAD + */ + default Flux> xRead(StreamReadOptions readOptions, + StreamOffset stream) { + + Assert.notNull(readOptions, "StreamReadOptions must not be null!"); + Assert.notNull(stream, "StreamOffset must not be null!"); + + return xRead(readOptions, new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param readOptions read arguments. + * @param streams the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREAD + */ + default Flux> xRead(StreamReadOptions readOptions, + StreamOffset... streams) { + + Assert.notNull(readOptions, "StreamReadOptions must not be null!"); + Assert.notNull(streams, "StreamOffsets must not be null!"); + + return read(Mono.just(ReadCommand.from(streams).withOptions(readOptions))).next() + .flatMapMany(CommandResponse::getOutput); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param commands must not be {@literal null}. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREAD + * @see Redis Documentation: XREADGROUP + */ + Flux>>> read(Publisher commands); + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param stream the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREADGROUP + */ + default Flux> xReadGroup(Consumer consumer, StreamOffset stream) { + return xReadGroup(consumer, StreamReadOptions.empty(), new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param streams the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREADGROUP + */ + default Flux> xReadGroup(Consumer consumer, + StreamOffset... streams) { + return xReadGroup(consumer, StreamReadOptions.empty(), streams); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param readOptions read arguments. + * @param stream the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREADGROUP + */ + default Flux> xReadGroup(Consumer consumer, StreamReadOptions readOptions, + StreamOffset stream) { + return xReadGroup(consumer, readOptions, new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param readOptions read arguments. + * @param streams the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREADGROUP + */ + default Flux> xReadGroup(Consumer consumer, StreamReadOptions readOptions, + StreamOffset... streams) { + + Assert.notNull(consumer, "Consumer must not be null!"); + Assert.notNull(streams, "StreamOffsets must not be null!"); + Assert.notNull(streams, "StreamOffsets must not be null!"); + + return read(Mono.just(ReadCommand.from(streams).withOptions(readOptions).as(consumer))).next() + .flatMapMany(CommandResponse::getOutput); + } + + /** + * Read messages from a stream within a specific {@link Range} in reverse order. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREVRANGE + */ + default Flux> xRevRange(ByteBuffer key, Range range) { + return xRevRange(key, range, Limit.unlimited()); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit} in reverse order. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREVRANGE + */ + default Flux> xRevRange(ByteBuffer key, Range range, Limit limit) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(range, "Range must not be null!"); + Assert.notNull(limit, "Limit must not be null!"); + + return xRevRange(Mono.just(RangeCommand.stream(key).within(range).limit(limit))).next() + .flatMapMany(CommandResponse::getOutput); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit} in reverse order. + * + * @param commands must not be {@literal null}. + * @return + * @see Redis Documentation: XREVRANGE + */ + Flux>>> xRevRange( + Publisher commands); + + /** + * {@code XTRIM} command parameters. + * + * @see Redis Documentation: XTRIM + */ + class TrimCommand extends KeyCommand { + + private @Nullable Long count; + + private TrimCommand(ByteBuffer key, @Nullable Long count) { + + super(key); + this.count = count; + } + + /** + * Creates a new {@link TrimCommand} given a {@link ByteBuffer key}. + * + * @param key must not be {@literal null}. + * @return a new {@link TrimCommand} for {@link ByteBuffer key}. + */ + public static TrimCommand stream(ByteBuffer key) { + + Assert.notNull(key, "Key must not be null!"); + + return new TrimCommand(key, null); + } + + /** + * Applies the numeric {@literal count}. Constructs a new command instance with all previously configured + * properties. + * + * @param count + * @return a new {@link TrimCommand} with {@literal count} applied. + */ + public TrimCommand to(long count) { + return new TrimCommand(getKey(), count); + } + + /** + * @return can be {@literal null}. + */ + @Nullable + public Long getCount() { + return count; + } + } + + /** + * Trims the stream to {@code count} elements. + * + * @param key the stream key. + * @param count length of the stream. + * @return number of removed entries. + * @see Redis Documentation: XTRIM + */ + default Mono xTrim(ByteBuffer key, long count) { + + Assert.notNull(key, "Key must not be null!"); + + return xTrim(Mono.just(TrimCommand.stream(key).to(count))).next().map(NumericResponse::getOutput); + } + + /** + * Trims the stream to {@code count} elements. + * + * @param commands must not be {@literal null}. + * @return + * @see Redis Documentation: XTRIM + */ + Flux> xTrim(Publisher commands); +} diff --git a/src/main/java/org/springframework/data/redis/connection/RedisCommands.java b/src/main/java/org/springframework/data/redis/connection/RedisCommands.java index daf970a8d..e59448ffe 100644 --- a/src/main/java/org/springframework/data/redis/connection/RedisCommands.java +++ b/src/main/java/org/springframework/data/redis/connection/RedisCommands.java @@ -13,7 +13,6 @@ * See the License for the specific language governing permissions and * limitations under the License. */ - package org.springframework.data.redis.connection; import org.springframework.lang.Nullable; @@ -23,10 +22,11 @@ import org.springframework.lang.Nullable; * * @author Costin Leau * @author Christoph Strobl + * @author Mark Paluch */ public interface RedisCommands extends RedisKeyCommands, RedisStringCommands, RedisListCommands, RedisSetCommands, RedisZSetCommands, RedisHashCommands, RedisTxCommands, RedisPubSubCommands, RedisConnectionCommands, - RedisServerCommands, RedisScriptingCommands, RedisGeoCommands, RedisHyperLogLogCommands { + RedisServerCommands, RedisStreamCommands, RedisScriptingCommands, RedisGeoCommands, RedisHyperLogLogCommands { /** * 'Native' or 'raw' execution of the given command along-side the given arguments. The command is executed as is, diff --git a/src/main/java/org/springframework/data/redis/connection/RedisConnection.java b/src/main/java/org/springframework/data/redis/connection/RedisConnection.java index c30cf003d..0ccddec62 100644 --- a/src/main/java/org/springframework/data/redis/connection/RedisConnection.java +++ b/src/main/java/org/springframework/data/redis/connection/RedisConnection.java @@ -110,6 +110,16 @@ public interface RedisConnection extends RedisCommands { return this; } + /** + * Get {@link RedisStreamCommands}. + * + * @return never {@literal null}. + * @since 2.2 + */ + default RedisStreamCommands streamCommands() { + return this; + } + /** * Get {@link RedisStringCommands}. * diff --git a/src/main/java/org/springframework/data/redis/connection/RedisStreamCommands.java b/src/main/java/org/springframework/data/redis/connection/RedisStreamCommands.java new file mode 100644 index 000000000..8af514ae5 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/connection/RedisStreamCommands.java @@ -0,0 +1,486 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.connection; + +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.ToString; + +import java.time.Duration; +import java.util.List; +import java.util.Map; + +import org.springframework.data.domain.Range; +import org.springframework.data.redis.connection.RedisZSetCommands.Limit; +import org.springframework.lang.Nullable; +import org.springframework.util.Assert; + +/** + * Stream-specific Redis commands. + * + * @author Mark Paluch + * @since 2.2 + */ +public interface RedisStreamCommands { + + /** + * Acknowledge one or more messages as processed. + * + * @param key the stream key. + * @param group name of the consumer group. + * @param messageIds message Id's to acknowledge. + * @return length of acknowledged messages. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XACK + */ + @Nullable + Long xAck(byte[] key, String group, String... messageIds); + + /** + * Append a message to the stream {@code key}. + * + * @param key the stream key. + * @param body message body. + * @return the message Id. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XADD + */ + @Nullable + String xAdd(byte[] key, Map body); + + /** + * Removes the specified entries from the stream. Returns the number of items deleted, that may be different from the + * number of IDs passed in case certain IDs do not exist. + * + * @param key the stream key. + * @param messageIds stream message Id's. + * @return number of removed entries. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XDEL + */ + @Nullable + Long xDel(byte[] key, String... messageIds); + + /** + * Create a consumer group. + * + * @param key the stream key. + * @param readOffset the offset to start with. + * @param group name of the consumer group. + * @return {@literal true} if successful. {@literal null} when used in pipeline / transaction. + */ + @Nullable + String xGroupCreate(byte[] key, ReadOffset readOffset, String group); + + /** + * Delete a consumer from a consumer group. + * + * @param key the stream key. + * @param consumer consumer identified by group name and consumer key. + * @return {@literal true} if successful. {@literal null} when used in pipeline / transaction. + */ + @Nullable + Boolean xGroupDelConsumer(byte[] key, Consumer consumer); + + /** + * Destroy a consumer group. + * + * @param key the stream key. + * @param group name of the consumer group. + * @return {@literal true} if successful. {@literal null} when used in pipeline / transaction. + */ + @Nullable + Boolean xGroupDestroy(byte[] key, String group); + + /** + * Get the length of a stream. + * + * @param key the stream key. + * @return length of the stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XLEN + */ + @Nullable + Long xLen(byte[] key); + + /** + * Read messages from a stream within a specific {@link Range}. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XRANGE + */ + @Nullable + default List> xRange(byte[] key, Range range) { + return xRange(key, range, Limit.unlimited()); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit}. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XRANGE + */ + @Nullable + List> xRange(byte[] key, Range range, Limit limit); + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param stream the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREAD + */ + @Nullable + default List> xRead(StreamOffset stream) { + return xRead(StreamReadOptions.empty(), new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param streams the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREAD + */ + @Nullable + default List> xRead(StreamOffset... streams) { + return xRead(StreamReadOptions.empty(), streams); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param readOptions read arguments. + * @param stream the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREAD + */ + @Nullable + default List> xRead(StreamReadOptions readOptions, StreamOffset stream) { + return xRead(readOptions, new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param readOptions read arguments. + * @param streams the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREAD + */ + @Nullable + List> xRead(StreamReadOptions readOptions, StreamOffset... streams); + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param stream the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREADGROUP + */ + @Nullable + default List> xReadGroup(Consumer consumer, StreamOffset stream) { + return xReadGroup(consumer, StreamReadOptions.empty(), new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param streams the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREADGROUP + */ + @Nullable + default List> xReadGroup(Consumer consumer, StreamOffset... streams) { + return xReadGroup(consumer, StreamReadOptions.empty(), streams); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param readOptions read arguments. + * @param stream the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREADGROUP + */ + @Nullable + default List> xReadGroup(Consumer consumer, StreamReadOptions readOptions, + StreamOffset stream) { + return xReadGroup(consumer, readOptions, new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param readOptions read arguments. + * @param streams the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREADGROUP + */ + @Nullable + List> xReadGroup(Consumer consumer, StreamReadOptions readOptions, + StreamOffset... streams); + + /** + * Read messages from a stream within a specific {@link Range} in reverse order. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREVRANGE + */ + @Nullable + default List> xRevRange(byte[] key, Range range) { + return xRevRange(key, range, Limit.unlimited()); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit} in reverse order. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREVRANGE + */ + @Nullable + List> xRevRange(byte[] key, Range range, Limit limit); + + /** + * Trims the stream to {@code count} elements. + * + * @param key the stream key. + * @param count length of the stream. + * @return number of removed entries. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XTRIM + */ + @Nullable + Long xTrim(byte[] key, long count); + + /** + * A stream message and its id. + * + * @author Mark Paluch + */ + @EqualsAndHashCode + @Getter + class StreamMessage { + + private final K stream; + private final String id; + private final Map body; + + /** + * Create a new {@link io.lettuce.core.StreamMessage}. + * + * @param stream the stream. + * @param id the message id. + * @param body map containing the message body. + */ + public StreamMessage(K stream, String id, Map body) { + + this.stream = stream; + this.id = id; + this.body = body; + } + + @Override + public String toString() { + return String.format("StreamMessage[%s:%s]%s", stream, id, body); + } + } + + /** + * Value object representing read offset for a Stream. + */ + @EqualsAndHashCode + @ToString + @Getter + class ReadOffset { + + private final String offset; + + private ReadOffset(String offset) { + this.offset = offset; + } + + /** + * Read from the latest offset. + * + * @return + */ + public static ReadOffset latest() { + return new ReadOffset("$"); + } + + /** + * Read all new arriving elements with ids greater than the last one consumed by the consumer group. + * + * @return the {@link ReadOffset} object without a specific offset. + */ + public static ReadOffset lastConsumed() { + return new ReadOffset(">"); + } + + /** + * Read all arriving elements from the stream starting at {@code offset}. + * + * @param offset the stream offset. + * @return the {@link StreamOffset} object without a specific offset. + */ + public static ReadOffset from(String offset) { + + Assert.hasText(offset, "Offset must not be empty"); + + return new ReadOffset(offset); + } + } + + /** + * Value object representing a Stream Id with its offset. + */ + @EqualsAndHashCode + @ToString + @Getter + class StreamOffset { + + private final K key; + private final ReadOffset offset; + + private StreamOffset(K key, ReadOffset offset) { + this.key = key; + this.offset = offset; + } + + /** + * Create a {@link StreamOffset} given {@code key} and {@link ReadOffset}. + * + * @return + */ + public static StreamOffset create(K key, ReadOffset readOffset) { + return new StreamOffset<>(key, readOffset); + } + } + + /** + * Options for reading messages from a Redis Stream. + */ + @EqualsAndHashCode + @ToString + @Getter + class StreamReadOptions { + + private static final StreamReadOptions EMPTY = new StreamReadOptions(null, null, false); + + private final @Nullable Long block; + private final @Nullable Long count; + private final boolean noack; + + private StreamReadOptions(@Nullable Long block, @Nullable Long count, boolean noack) { + this.block = block; + this.count = count; + this.noack = noack; + } + + /** + * Creates an empty {@link StreamReadOptions} instance. + * + * @return an empty {@link StreamReadOptions} instance. + */ + public static StreamReadOptions empty() { + return EMPTY; + } + + /** + * Disable auto-acknowledgement when reading in the context of a consumer group. + * + * @return {@link StreamReadOptions} with {@code noack} applied. + */ + public StreamReadOptions noack() { + return new StreamReadOptions(block, count, true); + } + + /** + * Use a blocking read and supply the {@link Duration timeout} after which the call will terminate if no message was + * read. + * + * @param timeout the timeout for the blocking read, must not be {@literal null} or negative. + * @return {@link StreamReadOptions} with {@code block} applied. + */ + public StreamReadOptions block(Duration timeout) { + + Assert.notNull(timeout, "Block timeout must not be null!"); + Assert.isTrue(!timeout.isNegative(), "Block timeout must not be negative!"); + + return new StreamReadOptions(timeout.toMillis(), count, noack); + } + + /** + * Limit the number of messages returned per stream. + * + * @param count the maximum number of messages to read. + * @return {@link StreamReadOptions} with {@code count} applied. + */ + public StreamReadOptions count(long count) { + + Assert.isTrue(count > 0, "Count must be greater or equal to zero!"); + + return new StreamReadOptions(block, count, noack); + } + } + + /** + * Value object representing a Stream consumer within a consumer group. Group name and consumer name are encoded as + * keys. + */ + @EqualsAndHashCode + @Getter + class Consumer { + + private final String group; + private final String name; + + private Consumer(String group, String name) { + this.group = group; + this.name = name; + } + + /** + * Create a new consumer. + * + * @param group name of the consumer group, must not be {@literal null} or empty. + * @param name name of the consumer, must not be {@literal null} or empty. + * @return the consumer {@link io.lettuce.core.Consumer} object. + */ + public static Consumer from(String group, String name) { + + Assert.hasText(group, "Group must not be null"); + Assert.hasText(name, "Name must not be null"); + + return new Consumer(group, name); + } + + @Override + public String toString() { + return String.format("%s:%s", group, name); + } + } +} diff --git a/src/main/java/org/springframework/data/redis/connection/StringRedisConnection.java b/src/main/java/org/springframework/data/redis/connection/StringRedisConnection.java index a19131910..d82a22400 100644 --- a/src/main/java/org/springframework/data/redis/connection/StringRedisConnection.java +++ b/src/main/java/org/springframework/data/redis/connection/StringRedisConnection.java @@ -1945,7 +1945,6 @@ public interface StringRedisConnection extends RedisConnection { */ List getClientList(); - /** * Get / Manipulate specific integer fields of varying bit widths and arbitrary non (necessary) aligned offset stored * at a given {@code key}. @@ -1955,4 +1954,271 @@ public interface StringRedisConnection extends RedisConnection { * @return */ List bitfield(String key, BitFieldSubCommands command); + + // ------------------------------------------------------------------------- + // Methods dealing with Redis Streams + // ------------------------------------------------------------------------- + + /** + * Acknowledge one or more messages as processed. + * + * @param key the stream key. + * @param group name of the consumer group. + * @param messageIds message Id's to acknowledge. + * @return length of acknowledged messages. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XACK + */ + @Nullable + Long xAck(String key, String group, String... messageIds); + + /** + * Append a message to the stream {@code key}. + * + * @param key the stream key. + * @param body message body. + * @return the message Id. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XADD + */ + @Nullable + String xAdd(String key, Map body); + + /** + * Removes the specified entries from the stream. Returns the number of items deleted, that may be different from the + * number of IDs passed in case certain IDs do not exist. + * + * @param key the stream key. + * @param messageIds stream message Id's. + * @return number of removed entries. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XDEL + */ + @Nullable + Long xDel(String key, String... messageIds); + + /** + * Create a consumer group. + * + * @param key + * @param readOffset + * @param group name of the consumer group. + * @since 2.2 + * @return {@literal true} if successful. {@literal null} when used in pipeline / transaction. + */ + @Nullable + String xGroupCreate(String key, ReadOffset readOffset, String group); + + /** + * Delete a consumer from a consumer group. + * + * @param key the stream key. + * @param consumer consumer identified by group name and consumer key. + * @since 2.2 + * @return {@literal true} if successful. {@literal null} when used in pipeline / transaction. + */ + @Nullable + Boolean xGroupDelConsumer(String key, Consumer consumer); + + /** + * Destroy a consumer group. + * + * @param key the stream key. + * @param group name of the consumer group. + * @return {@literal true} if successful. {@literal null} when used in pipeline / transaction. + * @since 2.2 + */ + @Nullable + Boolean xGroupDestroy(String key, String group); + + /** + * Get the length of a stream. + * + * @param key the stream key. + * @return length of the stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XLEN + */ + @Nullable + Long xLen(String key); + + /** + * Read messages from a stream within a specific {@link Range}. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XRANGE + */ + @Nullable + default List> xRange(String key, org.springframework.data.domain.Range range) { + return xRange(key, range, Limit.unlimited()); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit}. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XRANGE + */ + @Nullable + List> xRange(String key, org.springframework.data.domain.Range range, + Limit limit); + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param streams the streams to read from. + * @return list ith members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XREAD + */ + @Nullable + default List> xReadAsString(StreamOffset stream) { + return xReadAsString(StreamReadOptions.empty(), new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param streams the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XREAD + */ + @Nullable + default List> xReadAsString(StreamOffset... streams) { + return xReadAsString(StreamReadOptions.empty(), streams); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param readOptions read arguments. + * @param stream the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XREAD + */ + @Nullable + default List> xReadAsString(StreamReadOptions readOptions, + StreamOffset stream) { + return xReadAsString(readOptions, new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param readOptions read arguments. + * @param streams the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XREAD + */ + @Nullable + List> xReadAsString(StreamReadOptions readOptions, StreamOffset... streams); + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param stream the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XREADGROUP + */ + @Nullable + default List> xReadGroupAsString(Consumer consumer, StreamOffset stream) { + return xReadGroupAsString(consumer, StreamReadOptions.empty(), new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param streams the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XREADGROUP + */ + @Nullable + default List> xReadGroupAsString(Consumer consumer, StreamOffset... streams) { + return xReadGroupAsString(consumer, StreamReadOptions.empty(), streams); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param readOptions read arguments. + * @param stream the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XREADGROUP + */ + @Nullable + default List> xReadGroupAsString(Consumer consumer, StreamReadOptions readOptions, + StreamOffset stream) { + return xReadGroupAsString(consumer, readOptions, new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param readOptions read arguments. + * @param streams the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XREADGROUP + */ + @Nullable + List> xReadGroupAsString(Consumer consumer, StreamReadOptions readOptions, + StreamOffset... streams); + + /** + * Read messages from a stream within a specific {@link Range} in reverse order. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XREVRANGE + */ + @Nullable + default List> xRevRange(String key, + org.springframework.data.domain.Range range) { + return xRevRange(key, range, Limit.unlimited()); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit} in reverse order. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XREVRANGE + */ + @Nullable + List> xRevRange(String key, org.springframework.data.domain.Range range, + Limit limit); + + /** + * Trims the stream to {@code count} elements. + * + * @param key the stream key. + * @param count length of the stream. + * @return number of removed entries. {@literal null} when used in pipeline / transaction. + * @since 2.2 + * @see Redis Documentation: XTRIM + */ + @Nullable + Long xTrim(String key, long count); } diff --git a/src/main/java/org/springframework/data/redis/connection/convert/Converters.java b/src/main/java/org/springframework/data/redis/connection/convert/Converters.java index 25b138f04..011975c3f 100644 --- a/src/main/java/org/springframework/data/redis/connection/convert/Converters.java +++ b/src/main/java/org/springframework/data/redis/connection/convert/Converters.java @@ -18,16 +18,7 @@ package org.springframework.data.redis.connection.convert; import lombok.RequiredArgsConstructor; import java.time.Duration; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collection; -import java.util.Collections; -import java.util.LinkedHashMap; -import java.util.LinkedHashSet; -import java.util.List; -import java.util.Map; -import java.util.Properties; -import java.util.Set; +import java.util.*; import java.util.concurrent.TimeUnit; import org.springframework.core.convert.converter.Converter; @@ -45,6 +36,7 @@ import org.springframework.data.redis.connection.RedisClusterNode.SlotRange; import org.springframework.data.redis.connection.RedisGeoCommands.DistanceUnit; import org.springframework.data.redis.connection.RedisGeoCommands.GeoLocation; import org.springframework.data.redis.connection.RedisNode.NodeType; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; import org.springframework.data.redis.connection.RedisZSetCommands.Tuple; import org.springframework.data.redis.serializer.RedisSerializer; import org.springframework.lang.Nullable; @@ -380,6 +372,24 @@ abstract public class Converters { return new DeserializingGeoResultsConverter<>(serializer); } + /** + * {@link Converter} capable of deserializing {@link StreamMessage}. + * + * @param serializer + * @return + * @since 2.2 + */ + public static Converter, StreamMessage> deserializingStreamMessageConverter( + Converter serializer) { + + MapConverter mapConverter = new MapConverter<>(serializer); + return message -> { + + return new StreamMessage<>(serializer.convert(message.getStream()), message.getId(), + mapConverter.convert(message.getBody())); + }; + } + /** * {@link Converter} capable of converting Double into {@link Distance} using given {@link Metric}. * diff --git a/src/main/java/org/springframework/data/redis/connection/jedis/JedisConnection.java b/src/main/java/org/springframework/data/redis/connection/jedis/JedisConnection.java index 8461c927a..fdd8f3a57 100644 --- a/src/main/java/org/springframework/data/redis/connection/jedis/JedisConnection.java +++ b/src/main/java/org/springframework/data/redis/connection/jedis/JedisConnection.java @@ -156,6 +156,15 @@ public class JedisConnection extends AbstractRedisConnection { return new JedisKeyCommands(this); } + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisConnection#streamCommands() + */ + @Override + public RedisStreamCommands streamCommands() { + throw new UnsupportedOperationException("Streams not supported using Jedis!"); + } + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisConnection#stringCommands() diff --git a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceConnection.java b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceConnection.java index 9bd87dec3..973c7fde5 100644 --- a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceConnection.java +++ b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceConnection.java @@ -331,6 +331,15 @@ public class LettuceConnection extends AbstractRedisConnection { return new LettuceScriptingCommands(this); } + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisConnection#streamCommands() + */ + @Override + public RedisStreamCommands streamCommands() { + return new LettuceStreamCommands(this); + } + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisConnection#stringCommands() @@ -424,7 +433,7 @@ public class LettuceConnection extends AbstractRedisConnection { } } - /* + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.AbstractRedisConnection#close() */ @@ -457,7 +466,7 @@ public class LettuceConnection extends AbstractRedisConnection { this.dbIndex = defaultDbIndex; } - /* + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisConnection#isClosed() */ @@ -466,7 +475,7 @@ public class LettuceConnection extends AbstractRedisConnection { return isClosed && !isSubscribed(); } - /* + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisConnection#getNativeConnection() */ @@ -477,7 +486,7 @@ public class LettuceConnection extends AbstractRedisConnection { return (subscription != null ? subscription.getNativeConnection().async() : getAsyncConnection()); } - /* + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisConnection#isQueueing() */ @@ -486,7 +495,7 @@ public class LettuceConnection extends AbstractRedisConnection { return isMulti; } - /* + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisConnection#isPipelined() */ @@ -495,7 +504,7 @@ public class LettuceConnection extends AbstractRedisConnection { return isPipelined; } - /* + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisConnection#openPipeline() */ @@ -507,7 +516,7 @@ public class LettuceConnection extends AbstractRedisConnection { } } - /* + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisConnection#closePipeline() */ @@ -743,7 +752,7 @@ public class LettuceConnection extends AbstractRedisConnection { // Pub/Sub functionality // - /* + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisPubSubCommands#publish(byte[], byte[]) */ @@ -768,7 +777,7 @@ public class LettuceConnection extends AbstractRedisConnection { } } - /* + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisPubSubCommands#getSubscription() */ @@ -777,7 +786,7 @@ public class LettuceConnection extends AbstractRedisConnection { return subscription; } - /* + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisPubSubCommands#isSubscribed() */ @@ -786,7 +795,7 @@ public class LettuceConnection extends AbstractRedisConnection { return (subscription != null && subscription.isAlive()); } - /* + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisPubSubCommands#pSubscribe(org.springframework.data.redis.connection.MessageListener, byte[][]) */ @@ -807,7 +816,7 @@ public class LettuceConnection extends AbstractRedisConnection { } } - /* + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.RedisPubSubCommands#subscribe(org.springframework.data.redis.connection.MessageListener, byte[][]) */ diff --git a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceConverters.java b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceConverters.java index 0354d0762..2c69eeb92 100644 --- a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceConverters.java +++ b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceConverters.java @@ -531,7 +531,20 @@ abstract public class LettuceConverters extends Converters { * @since 2.0 */ public static Range toRange(org.springframework.data.redis.connection.RedisZSetCommands.Range range) { - return Range.from(lowerBoundaryOf(range), upperBoundaryOf(range)); + return Range.from(lowerBoundaryOf(range, false), upperBoundaryOf(range, false)); + } + + /** + * Convert a {@link org.springframework.data.redis.connection.RedisZSetCommands.Range} to a lettuce {@link Range}. + * + * @param range + * @param convertNumberToBytes + * @return + * @since 2.2 + */ + public static Range toRange(org.springframework.data.redis.connection.RedisZSetCommands.Range range, + boolean convertNumberToBytes) { + return Range.from(lowerBoundaryOf(range, convertNumberToBytes), upperBoundaryOf(range, convertNumberToBytes)); } /** @@ -543,23 +556,23 @@ abstract public class LettuceConverters extends Converters { * @since 2.0 */ public static Range toRevRange(org.springframework.data.redis.connection.RedisZSetCommands.Range range) { - return Range.from(upperBoundaryOf(range), lowerBoundaryOf(range)); + return Range.from(upperBoundaryOf(range, false), lowerBoundaryOf(range, false)); } @SuppressWarnings("unchecked") private static Range.Boundary lowerBoundaryOf( - org.springframework.data.redis.connection.RedisZSetCommands.Range range) { - return (Range.Boundary) rangeToBoundaryArgumentConverter(false).convert(range); + org.springframework.data.redis.connection.RedisZSetCommands.Range range, boolean convertNumberToBytes) { + return (Range.Boundary) rangeToBoundaryArgumentConverter(false, convertNumberToBytes).convert(range); } @SuppressWarnings("unchecked") private static Range.Boundary upperBoundaryOf( - org.springframework.data.redis.connection.RedisZSetCommands.Range range) { - return (Range.Boundary) rangeToBoundaryArgumentConverter(true).convert(range); + org.springframework.data.redis.connection.RedisZSetCommands.Range range, boolean convertNumberToBytes) { + return (Range.Boundary) rangeToBoundaryArgumentConverter(true, convertNumberToBytes).convert(range); } private static Converter> rangeToBoundaryArgumentConverter( - boolean upper) { + boolean upper, boolean convertNumberToBytes) { return (source) -> { @@ -572,7 +585,12 @@ abstract public class LettuceConverters extends Converters { Object value = sourceBoundary.getValue(); if (value instanceof Number) { - return inclusive ? Range.Boundary.including((Number) value) : Range.Boundary.excluding((Number) value); + + if (convertNumberToBytes) { + value = value.toString(); + } else { + return inclusive ? Range.Boundary.including((Number) value) : Range.Boundary.excluding((Number) value); + } } if (value instanceof String) { diff --git a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveClusterStreamCommands.java b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveClusterStreamCommands.java new file mode 100644 index 000000000..0103f850b --- /dev/null +++ b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveClusterStreamCommands.java @@ -0,0 +1,35 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.connection.lettuce; + +import org.springframework.data.redis.connection.ReactiveClusterStreamCommands; + +/** + * @author Mark Paluch + * @since 2.2 + */ +class LettuceReactiveClusterStreamCommands extends LettuceReactiveStreamCommands + implements ReactiveClusterStreamCommands { + + /** + * Create new {@link LettuceReactiveClusterStreamCommands}. + * + * @param connection must not be {@literal null}. + */ + LettuceReactiveClusterStreamCommands(LettuceReactiveRedisConnection connection) { + super(connection); + } +} diff --git a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveRedisClusterConnection.java b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveRedisClusterConnection.java index d694f2973..44873ddbb 100644 --- a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveRedisClusterConnection.java +++ b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveRedisClusterConnection.java @@ -178,6 +178,15 @@ class LettuceReactiveRedisClusterConnection extends LettuceReactiveRedisConnecti return new LettuceReactiveClusterServerCommands(this, topologyProvider); } + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.lettuce.LettuceReactiveRedisConnection#streamCommands() + */ + @Override + public LettuceReactiveClusterStreamCommands streamCommands() { + return new LettuceReactiveClusterStreamCommands(this); + } + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.ReactiveRedisClusterConnection#ping(org.springframework.data.redis.connection.RedisClusterNode) diff --git a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveRedisConnection.java b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveRedisConnection.java index 6a11a8bf6..8dac9f250 100644 --- a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveRedisConnection.java +++ b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveRedisConnection.java @@ -197,6 +197,15 @@ class LettuceReactiveRedisConnection implements ReactiveRedisConnection { return new LettuceReactiveServerCommands(this); } + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.ReactiveRedisConnection#streamCommands() + */ + @Override + public ReactiveStreamCommands streamCommands() { + return new LettuceReactiveStreamCommands(this); + } + /* * (non-Javadoc) * @see org.springframework.data.redis.connection.ReactiveRedisConnection#ping() diff --git a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveStreamCommands.java b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveStreamCommands.java new file mode 100644 index 000000000..9738dd9ea --- /dev/null +++ b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveStreamCommands.java @@ -0,0 +1,237 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.connection.lettuce; + +import io.lettuce.core.XReadArgs; +import io.lettuce.core.XReadArgs.StreamOffset; +import io.lettuce.core.cluster.api.reactive.RedisClusterReactiveCommands; +import reactor.core.publisher.Flux; + +import java.nio.ByteBuffer; +import java.util.Collection; +import java.util.function.Function; + +import org.reactivestreams.Publisher; +import org.springframework.data.redis.connection.ReactiveRedisConnection.CommandResponse; +import org.springframework.data.redis.connection.ReactiveRedisConnection.KeyCommand; +import org.springframework.data.redis.connection.ReactiveRedisConnection.NumericResponse; +import org.springframework.data.redis.connection.ReactiveStreamCommands; +import org.springframework.data.redis.connection.RedisStreamCommands; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.util.ByteUtils; +import org.springframework.util.Assert; + +/** + * {@link ReactiveStreamCommands} implementation for {@literal Lettuce}. + * + * @author Mark Paluch + * @since 2.2 + */ +class LettuceReactiveStreamCommands implements ReactiveStreamCommands { + + private final LettuceReactiveRedisConnection connection; + + /** + * Create new {@link LettuceReactiveStreamCommands}. + * + * @param connection must not be {@literal null}. + */ + LettuceReactiveStreamCommands(LettuceReactiveRedisConnection connection) { + + Assert.notNull(connection, "Connection must not be null!"); + this.connection = connection; + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.ReactiveStreamCommands#xAck(org.reactivestreams.Publisher) + */ + @Override + public Flux> xAck(Publisher commands) { + + return connection.execute(cmd -> Flux.from(commands).concatMap(command -> { + + Assert.notNull(command.getKey(), "Key must not be null!"); + Assert.notNull(command.getGroup(), "Group must not be null!"); + Assert.notNull(command.getMessageIds(), "MessageIds must not be null!"); + + return cmd.xack(command.getKey(), ByteUtils.getByteBuffer(command.getGroup()), + command.getMessageIds().toArray(new String[0])).map(value -> new NumericResponse<>(command, value)); + })); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.ReactiveStreamCommands#xAdd(org.reactivestreams.Publisher) + */ + @Override + public Flux> xAdd(Publisher commands) { + + return connection.execute(cmd -> Flux.from(commands).concatMap(command -> { + + Assert.notNull(command.getKey(), "Key must not be null!"); + Assert.notNull(command.getBody(), "Body must not be null!"); + + return cmd.xadd(command.getKey(), command.getBody()).map(value -> new CommandResponse<>(command, value)); + })); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.ReactiveStreamCommands#xDel(org.reactivestreams.Publisher) + */ + @Override + public Flux> xDel(Publisher commands) { + + return connection.execute(cmd -> Flux.from(commands).concatMap(command -> { + + Assert.notNull(command.getKey(), "Key must not be null!"); + Assert.notNull(command.getMessageIds(), "MessageIds must not be null!"); + + return cmd.xdel(command.getKey(), command.getMessageIds().toArray(new String[0])) + .map(value -> new NumericResponse<>(command, value)); + })); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.ReactiveStreamCommands#xLen(org.reactivestreams.Publisher) + */ + @Override + public Flux> xLen(Publisher commands) { + + return connection.execute(cmd -> Flux.from(commands).concatMap(command -> { + + Assert.notNull(command.getKey(), "Key must not be null!"); + + return cmd.xlen(command.getKey()).map(value -> new NumericResponse<>(command, value)); + })); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.ReactiveStreamCommands#xRange(org.reactivestreams.Publisher) + */ + @Override + public Flux>>> xRange( + Publisher commands) { + + return connection.execute(cmd -> Flux.from(commands).map(command -> { + + Assert.notNull(command.getKey(), "Key must not be null!"); + Assert.notNull(command.getRange(), "Range must not be null!"); + Assert.notNull(command.getLimit(), "Limit must not be null!"); + + io.lettuce.core.Range lettuceRange = RangeConverter.toRange(command.getRange(), Function.identity()); + io.lettuce.core.Limit lettuceLimit = LettuceConverters.toLimit(command.getLimit()); + + return new CommandResponse<>(command, cmd.xrange(command.getKey(), lettuceRange, lettuceLimit) + .map(StreamConverters::toStreamMessage)); + })); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.ReactiveStreamCommands#read(org.reactivestreams.Publisher) + */ + @Override + public Flux>>> read( + Publisher commands) { + + return Flux.from(commands).map(command -> { + + Assert.notNull(command.getStreamOffsets(), "StreamOffsets must not be null!"); + Assert.notNull(command.getReadOptions(), "ReadOptions must not be null!"); + + StreamReadOptions readOptions = command.getReadOptions(); + + if (readOptions.getBlock() != null && readOptions.getBlock() > 0) { + return new CommandResponse<>(command, connection.executeDedicated(cmd -> doRead(command, readOptions, cmd))); + } + + return new CommandResponse<>(command, connection.execute(cmd -> doRead(command, readOptions, cmd))); + }); + } + + private static Flux> doRead(ReadCommand command, StreamReadOptions readOptions, + RedisClusterReactiveCommands cmd) { + + StreamOffset[] streamOffsets = toStreamOffsets(command.getStreamOffsets()); + XReadArgs args = StreamConverters.toReadArgs(readOptions); + + if (command.getConsumer() == null) { + return cmd.xread(args, streamOffsets) + .map(StreamConverters::toStreamMessage); + } + + io.lettuce.core.Consumer lettuceConsumer = toConsumer(command.getConsumer()); + + return cmd.xreadgroup(lettuceConsumer, args, streamOffsets) + .map(StreamConverters::toStreamMessage); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.ReactiveStreamCommands#xRevRange(org.reactivestreams.Publisher) + */ + @Override + public Flux>>> xRevRange( + Publisher commands) { + + return connection.execute(cmd -> Flux.from(commands).map(command -> { + + Assert.notNull(command.getKey(), "Key must not be null!"); + Assert.notNull(command.getRange(), "Range must not be null!"); + Assert.notNull(command.getLimit(), "Limit must not be null!"); + + io.lettuce.core.Range lettuceRange = RangeConverter.toRange(command.getRange(), Function.identity()); + io.lettuce.core.Limit lettuceLimit = LettuceConverters.toLimit(command.getLimit()); + + return new CommandResponse<>(command, cmd.xrevrange(command.getKey(), lettuceRange, lettuceLimit) + .map(StreamConverters::toStreamMessage)); + })); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.ReactiveStreamCommands#xTrim(org.reactivestreams.Publisher) + */ + @Override + public Flux> xTrim(Publisher commands) { + + return connection.execute(cmd -> Flux.from(commands).concatMap(command -> { + + Assert.notNull(command.getKey(), "Key must not be null!"); + Assert.notNull(command.getCount(), "Count must not be null!"); + + return cmd.xtrim(command.getKey(), command.getCount()).map(value -> new NumericResponse<>(command, value)); + })); + } + + @SuppressWarnings("unchecked") + private static StreamOffset[] toStreamOffsets(Collection> streams) { + + return streams.stream().map(it -> StreamOffset.from(it.getKey(), it.getOffset().getOffset())) + .toArray(StreamOffset[]::new); + } + + private static io.lettuce.core.Consumer toConsumer(Consumer consumer) { + return io.lettuce.core.Consumer.from(ByteUtils.getByteBuffer(consumer.getGroup()), + ByteUtils.getByteBuffer(consumer.getName())); + } +} diff --git a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveZSetCommands.java b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveZSetCommands.java index 374b7f510..ae8db0596 100644 --- a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveZSetCommands.java +++ b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveZSetCommands.java @@ -16,13 +16,10 @@ package org.springframework.data.redis.connection.lettuce; import io.lettuce.core.Range; -import io.lettuce.core.Range.Boundary; import io.lettuce.core.ScanStream; import io.lettuce.core.ScoredValue; import io.lettuce.core.ZAddArgs; import io.lettuce.core.ZStoreArgs; -import io.lettuce.core.codec.StringCodec; -import io.lettuce.core.protocol.LettuceCharsets; import reactor.core.publisher.Flux; import reactor.core.publisher.Mono; @@ -30,7 +27,6 @@ import java.nio.ByteBuffer; import java.util.List; import org.reactivestreams.Publisher; -import org.springframework.core.convert.converter.Converter; import org.springframework.data.domain.Sort.Direction; import org.springframework.data.redis.connection.DefaultTuple; import org.springframework.data.redis.connection.ReactiveRedisConnection.CommandResponse; @@ -44,7 +40,6 @@ import org.springframework.data.redis.util.ByteUtils; import org.springframework.lang.Nullable; import org.springframework.util.Assert; import org.springframework.util.ObjectUtils; -import org.springframework.util.StringUtils; /** * @author Christoph Strobl @@ -237,7 +232,7 @@ class LettuceReactiveZSetCommands implements ReactiveZSetCommands { if (ObjectUtils.nullSafeEquals(command.getDirection(), Direction.ASC)) { - Range range = ArgumentConverters.toRange(command.getRange()); + Range range = RangeConverter.toRange(command.getRange()); if (command.isWithScores()) { @@ -262,7 +257,7 @@ class LettuceReactiveZSetCommands implements ReactiveZSetCommands { } } else { - Range range = ArgumentConverters.toRange(command.getRange()); + Range range = RangeConverter.toRange(command.getRange()); if (command.isWithScores()) { @@ -324,7 +319,7 @@ class LettuceReactiveZSetCommands implements ReactiveZSetCommands { Assert.notNull(command.getKey(), "Key must not be null!"); Assert.notNull(command.getRange(), "Range must not be null!"); - Range range = ArgumentConverters.toRange(command.getRange()); + Range range = RangeConverter.toRange(command.getRange()); Mono result = cmd.zcount(command.getKey(), range); return result.map(value -> new NumericResponse<>(command, value)); @@ -397,7 +392,7 @@ class LettuceReactiveZSetCommands implements ReactiveZSetCommands { Assert.notNull(command.getKey(), "Key must not be null!"); Assert.notNull(command.getRange(), "Range must not be null!"); - Range range = ArgumentConverters.toRange(command.getRange()); + Range range = RangeConverter.toRange(command.getRange()); Mono result = cmd.zremrangebyscore(command.getKey(), range); return result.map(value -> new NumericResponse<>(command, value)); @@ -471,17 +466,17 @@ class LettuceReactiveZSetCommands implements ReactiveZSetCommands { if (!command.getLimit().isUnlimited()) { if (ObjectUtils.nullSafeEquals(command.getDirection(), Direction.ASC)) { - result = cmd.zrangebylex(command.getKey(), ArgumentConverters.toRange(command.getRange()), + result = cmd.zrangebylex(command.getKey(), RangeConverter.toRange(command.getRange()), LettuceConverters.toLimit(command.getLimit())); } else { - result = cmd.zrevrangebylex(command.getKey(), ArgumentConverters.toRange(command.getRange()), + result = cmd.zrevrangebylex(command.getKey(), RangeConverter.toRange(command.getRange()), LettuceConverters.toLimit(command.getLimit())); } } else { if (ObjectUtils.nullSafeEquals(command.getDirection(), Direction.ASC)) { - result = cmd.zrangebylex(command.getKey(), ArgumentConverters.toRange(command.getRange())); + result = cmd.zrangebylex(command.getKey(), RangeConverter.toRange(command.getRange())); } else { - result = cmd.zrevrangebylex(command.getKey(), ArgumentConverters.toRange(command.getRange())); + result = cmd.zrevrangebylex(command.getKey(), RangeConverter.toRange(command.getRange())); } } @@ -520,56 +515,4 @@ class LettuceReactiveZSetCommands implements ReactiveZSetCommands { protected LettuceReactiveRedisConnection getConnection() { return connection; } - - /** - * @author Christoph Strobl - * @author Mark Paluch - */ - private static class ArgumentConverters { - - static Range toRange(org.springframework.data.domain.Range range) { - return Range.from(lowerBoundArgOf(range), upperBoundArgOf(range)); - } - - @SuppressWarnings("unchecked") - static Boundary lowerBoundArgOf(org.springframework.data.domain.Range range) { - return (Boundary) rangeToBoundArgumentConverter(false).convert(range); - } - - @SuppressWarnings("unchecked") - static Boundary upperBoundArgOf(org.springframework.data.domain.Range range) { - return (Boundary) rangeToBoundArgumentConverter(true).convert(range); - } - - private static Converter, Boundary> rangeToBoundArgumentConverter( - Boolean upper) { - - return (source) -> { - Boolean inclusive = upper ? source.getUpperBound().isInclusive() : source.getLowerBound().isInclusive(); - Object value = upper ? LettuceConverters.getUpperBound(source).orElse(null) - : LettuceConverters.getLowerBound(source).orElse(null); - - if (value == null) { - return Boundary.unbounded(); - } - - if (value instanceof Number) { - return inclusive ? Boundary.including((Number) value) : Boundary.excluding((Number) value); - } - - if (value instanceof String) { - - StringCodec stringCodec = new StringCodec(LettuceCharsets.UTF8); - if (!StringUtils.hasText((String) value) || ObjectUtils.nullSafeEquals(value, "+") - || ObjectUtils.nullSafeEquals(value, "-")) { - return Boundary.unbounded(); - } - return inclusive ? Boundary.including(stringCodec.encodeValue((String) value)) - : Boundary.excluding(stringCodec.encodeValue((String) value)); - } - - return inclusive ? Boundary.including((ByteBuffer) value) : Boundary.excluding((ByteBuffer) value); - }; - } - } } diff --git a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceStreamCommands.java b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceStreamCommands.java new file mode 100644 index 000000000..88c04ef21 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceStreamCommands.java @@ -0,0 +1,469 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.connection.lettuce; + +import io.lettuce.core.XReadArgs; +import io.lettuce.core.cluster.api.async.RedisClusterAsyncCommands; +import io.lettuce.core.cluster.api.sync.RedisClusterCommands; +import lombok.NonNull; +import lombok.RequiredArgsConstructor; + +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.function.Function; + +import org.springframework.dao.DataAccessException; +import org.springframework.data.domain.Range; +import org.springframework.data.redis.connection.RedisStreamCommands; +import org.springframework.data.redis.connection.RedisZSetCommands.Limit; +import org.springframework.util.Assert; + +/** + * @author Mark Paluch + * @since 2.2 + */ +@RequiredArgsConstructor +class LettuceStreamCommands implements RedisStreamCommands { + + private final @NonNull LettuceConnection connection; + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xAck(byte[], byte[], java.lang.String[]) + */ + @Override + public Long xAck(byte[] key, String group, String... messageIds) { + + Assert.notNull(key, "Key must not be null!"); + Assert.hasText(group, "Group name must not be null or empty!"); + Assert.notNull(messageIds, "MessageIds must not be null!"); + + try { + if (isPipelined()) { + pipeline( + connection.newLettuceResult(getAsyncConnection().xack(key, LettuceConverters.toBytes(group), messageIds))); + return null; + } + if (isQueueing()) { + transaction( + connection.newLettuceResult(getAsyncConnection().xack(key, LettuceConverters.toBytes(group), messageIds))); + return null; + } + return getConnection().xack(key, LettuceConverters.toBytes(group), messageIds); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xAdd(byte[], java.util.Map) + */ + @Override + public String xAdd(byte[] key, Map body) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(body, "Message body must not be null!"); + + try { + if (isPipelined()) { + pipeline(connection.newLettuceResult(getAsyncConnection().xadd(key, body))); + return null; + } + if (isQueueing()) { + transaction(connection.newLettuceResult(getAsyncConnection().xadd(key, body))); + return null; + } + return getConnection().xadd(key, body); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xDel(byte[], java.lang.String[]) + */ + @Override + public Long xDel(byte[] key, String... messageIds) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(messageIds, "MessageIds must not be null!"); + + try { + if (isPipelined()) { + pipeline(connection.newLettuceResult(getAsyncConnection().xdel(key, messageIds))); + return null; + } + if (isQueueing()) { + transaction(connection.newLettuceResult(getAsyncConnection().xdel(key, messageIds))); + return null; + } + return getConnection().xdel(key, messageIds); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xGroupCreate(byte[], org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset, java.lang.String) + */ + @Override + public String xGroupCreate(byte[] key, ReadOffset readOffset, String group) { + + Assert.notNull(key, "Key must not be null!"); + Assert.hasText(group, "Group name must not be null or empty!"); + Assert.notNull(readOffset, "ReadOffset must not be null!"); + + try { + XReadArgs.StreamOffset streamOffset = XReadArgs.StreamOffset.from(key, readOffset.getOffset()); + + if (isPipelined()) { + pipeline(connection + .newLettuceResult(getAsyncConnection().xgroupCreate(streamOffset, LettuceConverters.toBytes(group)))); + return null; + } + if (isQueueing()) { + transaction(connection + .newLettuceResult(getAsyncConnection().xgroupCreate(streamOffset, LettuceConverters.toBytes(group)))); + return null; + } + return getConnection().xgroupCreate(streamOffset, LettuceConverters.toBytes(group)); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xGroupDelConsumer(byte[], org.springframework.data.redis.connection.RedisStreamCommands.Consumer) + */ + @Override + public Boolean xGroupDelConsumer(byte[] key, Consumer consumer) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(consumer, "Consumer must not be null!"); + + try { + io.lettuce.core.Consumer lettuceConsumer = toConsumer(consumer); + + if (isPipelined()) { + pipeline(connection.newLettuceResult(getAsyncConnection().xgroupDelconsumer(key, lettuceConsumer))); + return null; + } + if (isQueueing()) { + transaction(connection.newLettuceResult(getAsyncConnection().xgroupDelconsumer(key, lettuceConsumer))); + return null; + } + return getConnection().xgroupDelconsumer(key, lettuceConsumer); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xGroupDestroy(byte[], java.lang.String) + */ + @Override + public Boolean xGroupDestroy(byte[] key, String group) { + + Assert.notNull(key, "Key must not be null!"); + Assert.hasText(group, "Group name must not be null or empty!"); + + try { + if (isPipelined()) { + pipeline( + connection.newLettuceResult(getAsyncConnection().xgroupDestroy(key, LettuceConverters.toBytes(group)))); + return null; + } + if (isQueueing()) { + transaction( + connection.newLettuceResult(getAsyncConnection().xgroupDestroy(key, LettuceConverters.toBytes(group)))); + return null; + } + return getConnection().xgroupDestroy(key, LettuceConverters.toBytes(group)); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xLen(byte[]) + */ + @Override + public Long xLen(byte[] key) { + + Assert.notNull(key, "Key must not be null!"); + + try { + if (isPipelined()) { + pipeline(connection.newLettuceResult(getAsyncConnection().xlen(key))); + return null; + } + if (isQueueing()) { + transaction(connection.newLettuceResult(getAsyncConnection().xlen(key))); + return null; + } + return getConnection().xlen(key); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xRange(byte[], org.springframework.data.domain.Range, org.springframework.data.redis.connection.RedisZSetCommands.Limit) + */ + @Override + public List> xRange(byte[] key, Range range, Limit limit) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(range, "Range must not be null!"); + Assert.notNull(limit, "Limit must not be null!"); + + io.lettuce.core.Range lettuceRange = RangeConverter.toRange(range, Function.identity()); + io.lettuce.core.Limit lettuceLimit = LettuceConverters.toLimit(limit); + try { + if (isPipelined()) { + pipeline(connection.newLettuceResult(getAsyncConnection().xrange(key, lettuceRange, lettuceLimit), + StreamConverters.streamMessageListConverter())); + return null; + } + if (isQueueing()) { + transaction(connection.newLettuceResult(getAsyncConnection().xrange(key, lettuceRange, lettuceLimit), + StreamConverters.streamMessageListConverter())); + return null; + } + return StreamConverters.toStreamMessages(getConnection().xrange(key, lettuceRange, lettuceLimit)); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xRead(org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions, org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset[]) + */ + @Override + public List> xRead(StreamReadOptions readOptions, StreamOffset... streams) { + + Assert.notNull(readOptions, "StreamReadOptions must not be null!"); + Assert.notNull(streams, "StreamOffsets must not be null!"); + + XReadArgs.StreamOffset[] streamOffsets = toStreamOffsets(streams); + XReadArgs args = StreamConverters.toReadArgs(readOptions); + + if (isBlocking(readOptions)) { + + try { + if (isPipelined()) { + pipeline(connection.newLettuceResult(getAsyncDedicatedConnection().xread(args, streamOffsets), + StreamConverters.streamMessageListConverter())); + return null; + } + if (isQueueing()) { + transaction(connection.newLettuceResult(getAsyncDedicatedConnection().xread(args, streamOffsets), + StreamConverters.streamMessageListConverter())); + return null; + } + return StreamConverters.toStreamMessages(getDedicatedConnection().xread(args, streamOffsets)); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + try { + if (isPipelined()) { + pipeline(connection.newLettuceResult(getAsyncConnection().xread(args, streamOffsets), + StreamConverters.streamMessageListConverter())); + return null; + } + if (isQueueing()) { + transaction(connection.newLettuceResult(getAsyncConnection().xread(args, streamOffsets), + StreamConverters.streamMessageListConverter())); + return null; + } + return StreamConverters.toStreamMessages(getConnection().xread(args, streamOffsets)); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xReadGroup(org.springframework.data.redis.connection.RedisStreamCommands.Consumer, org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions, org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset[]) + */ + @Override + public List> xReadGroup(Consumer consumer, StreamReadOptions readOptions, + StreamOffset... streams) { + + Assert.notNull(consumer, "Consumer must not be null!"); + Assert.notNull(readOptions, "StreamReadOptions must not be null!"); + Assert.notNull(streams, "StreamOffsets must not be null!"); + + XReadArgs.StreamOffset[] streamOffsets = toStreamOffsets(streams); + XReadArgs args = StreamConverters.toReadArgs(readOptions); + io.lettuce.core.Consumer lettuceConsumer = toConsumer(consumer); + + if (isBlocking(readOptions)) { + + try { + if (isPipelined()) { + pipeline(connection.newLettuceResult( + getAsyncDedicatedConnection().xreadgroup(lettuceConsumer, args, streamOffsets), + StreamConverters.streamMessageListConverter())); + return null; + } + if (isQueueing()) { + transaction(connection.newLettuceResult( + getAsyncDedicatedConnection().xreadgroup(lettuceConsumer, args, streamOffsets), + StreamConverters.streamMessageListConverter())); + return null; + } + return StreamConverters + .toStreamMessages(getDedicatedConnection().xreadgroup(lettuceConsumer, args, streamOffsets)); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + try { + if (isPipelined()) { + pipeline(connection.newLettuceResult(getAsyncConnection().xreadgroup(lettuceConsumer, args, streamOffsets), + StreamConverters.streamMessageListConverter())); + return null; + } + if (isQueueing()) { + transaction(connection.newLettuceResult(getAsyncConnection().xreadgroup(lettuceConsumer, args, streamOffsets), + StreamConverters.streamMessageListConverter())); + return null; + } + return StreamConverters.toStreamMessages(getConnection().xreadgroup(lettuceConsumer, args, streamOffsets)); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xRevRange(byte[], org.springframework.data.domain.Range, org.springframework.data.redis.connection.RedisZSetCommands.Limit) + */ + @Override + public List> xRevRange(byte[] key, Range range, Limit limit) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(range, "Range must not be null!"); + Assert.notNull(limit, "Limit must not be null!"); + + io.lettuce.core.Range lettuceRange = RangeConverter.toRange(range); + io.lettuce.core.Limit lettuceLimit = LettuceConverters.toLimit(limit); + try { + if (isPipelined()) { + pipeline(connection.newLettuceResult(getAsyncConnection().xrevrange(key, lettuceRange, lettuceLimit), + StreamConverters.streamMessageListConverter())); + return null; + } + if (isQueueing()) { + transaction(connection.newLettuceResult(getAsyncConnection().xrevrange(key, lettuceRange, lettuceLimit), + StreamConverters.streamMessageListConverter())); + return null; + } + return StreamConverters.toStreamMessages(getConnection().xrevrange(key, lettuceRange, lettuceLimit)); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.connection.RedisStreamCommands#xTrim(byte[], long) + */ + @Override + public Long xTrim(byte[] key, long count) { + + Assert.notNull(key, "Key must not be null!"); + + try { + if (isPipelined()) { + pipeline(connection.newLettuceResult(getAsyncConnection().xtrim(key, count))); + return null; + } + if (isQueueing()) { + transaction(connection.newLettuceResult(getAsyncConnection().xtrim(key, count))); + return null; + } + return getConnection().xtrim(key, count); + } catch (Exception ex) { + throw convertLettuceAccessException(ex); + } + } + + private boolean isPipelined() { + return connection.isPipelined(); + } + + private boolean isQueueing() { + return connection.isQueueing(); + } + + private void pipeline(LettuceResult result) { + connection.pipeline(result); + } + + private void transaction(LettuceResult result) { + connection.transaction(result); + } + + RedisClusterAsyncCommands getAsyncConnection() { + return connection.getAsyncConnection(); + } + + RedisClusterCommands getConnection() { + return connection.getConnection(); + } + + RedisClusterAsyncCommands getAsyncDedicatedConnection() { + return connection.getAsyncDedicatedConnection(); + } + + RedisClusterCommands getDedicatedConnection() { + return connection.getDedicatedConnection(); + } + + private DataAccessException convertLettuceAccessException(Exception ex) { + return connection.convertLettuceAccessException(ex); + } + + private static boolean isBlocking(StreamReadOptions readOptions) { + return readOptions.getBlock() != null && readOptions.getBlock() > 0; + } + + @SuppressWarnings("unchecked") + private static XReadArgs.StreamOffset[] toStreamOffsets(StreamOffset[] streams) { + + return Arrays.stream(streams).map(it -> XReadArgs.StreamOffset.from(it.getKey(), it.getOffset().getOffset())) + .toArray(XReadArgs.StreamOffset[]::new); + } + + private static io.lettuce.core.Consumer toConsumer(Consumer consumer) { + return io.lettuce.core.Consumer.from(LettuceConverters.toBytes(consumer.getGroup()), + LettuceConverters.toBytes(consumer.getName())); + } +} diff --git a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceZSetCommands.java b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceZSetCommands.java index 35a51f95d..8be36a8ca 100644 --- a/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceZSetCommands.java +++ b/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceZSetCommands.java @@ -841,11 +841,13 @@ class LettuceZSetCommands implements RedisZSetCommands { try { if (isPipelined()) { if (limit.isUnlimited()) { - pipeline(connection.newLettuceResult(getAsyncConnection().zrangebylex(key, LettuceConverters.toRange(range)), + pipeline( + connection.newLettuceResult(getAsyncConnection().zrangebylex(key, LettuceConverters.toRange(range, true)), LettuceConverters.bytesListToBytesSet())); } else { pipeline(connection.newLettuceResult( - getAsyncConnection().zrangebylex(key, LettuceConverters.toRange(range), LettuceConverters.toLimit(limit)), + getAsyncConnection().zrangebylex(key, LettuceConverters.toRange(range, true), + LettuceConverters.toLimit(limit)), LettuceConverters.bytesListToBytesSet())); } return null; @@ -853,11 +855,12 @@ class LettuceZSetCommands implements RedisZSetCommands { if (isQueueing()) { if (limit.isUnlimited()) { transaction( - connection.newLettuceResult(getAsyncConnection().zrangebylex(key, LettuceConverters.toRange(range)), + connection.newLettuceResult(getAsyncConnection().zrangebylex(key, LettuceConverters.toRange(range, true)), LettuceConverters.bytesListToBytesSet())); } else { transaction(connection.newLettuceResult( - getAsyncConnection().zrangebylex(key, LettuceConverters.toRange(range), LettuceConverters.toLimit(limit)), + getAsyncConnection().zrangebylex(key, LettuceConverters.toRange(range, true), + LettuceConverters.toLimit(limit)), LettuceConverters.bytesListToBytesSet())); } return null; @@ -865,10 +868,10 @@ class LettuceZSetCommands implements RedisZSetCommands { if (limit.isUnlimited()) { return LettuceConverters.bytesListToBytesSet() - .convert(getConnection().zrangebylex(key, LettuceConverters.toRange(range))); + .convert(getConnection().zrangebylex(key, LettuceConverters.toRange(range, true))); } return LettuceConverters.bytesListToBytesSet().convert( - getConnection().zrangebylex(key, LettuceConverters.toRange(range), LettuceConverters.toLimit(limit))); + getConnection().zrangebylex(key, LettuceConverters.toRange(range, true), LettuceConverters.toLimit(limit))); } catch (Exception ex) { throw convertLettuceAccessException(ex); diff --git a/src/main/java/org/springframework/data/redis/connection/lettuce/RangeConverter.java b/src/main/java/org/springframework/data/redis/connection/lettuce/RangeConverter.java new file mode 100644 index 000000000..21fa719aa --- /dev/null +++ b/src/main/java/org/springframework/data/redis/connection/lettuce/RangeConverter.java @@ -0,0 +1,88 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.connection.lettuce; + +import io.lettuce.core.Range; +import io.lettuce.core.Range.Boundary; +import io.lettuce.core.codec.StringCodec; + +import java.nio.ByteBuffer; +import java.util.function.Function; + +import org.springframework.core.convert.converter.Converter; +import org.springframework.util.ObjectUtils; +import org.springframework.util.StringUtils; + +/** + * @author Christoph Strobl + * @author Mark Paluch + */ +class RangeConverter { + + static Range toRange(org.springframework.data.domain.Range range) { + return toRange(range, StringCodec.UTF8::encodeValue); + } + + static Range toRange(org.springframework.data.domain.Range range, + Function stringEncoder) { + return Range.from(lowerBoundArgOf(range, stringEncoder), upperBoundArgOf(range, stringEncoder)); + } + + @SuppressWarnings("unchecked") + private static Boundary lowerBoundArgOf(org.springframework.data.domain.Range range, + Function stringEncoder) { + return (Boundary) rangeToBoundArgumentConverter(false, stringEncoder).convert(range); + } + + @SuppressWarnings("unchecked") + private static Boundary upperBoundArgOf(org.springframework.data.domain.Range range, + Function stringEncoder) { + return (Boundary) rangeToBoundArgumentConverter(true, stringEncoder).convert(range); + } + + private static Converter, Boundary> rangeToBoundArgumentConverter( + boolean upper, Function stringEncoder) { + + return (source) -> { + + Boolean inclusive = upper ? source.getUpperBound().isInclusive() : source.getLowerBound().isInclusive(); + Object value = upper ? source.getUpperBound().getValue().orElse(null) + : source.getLowerBound().getValue().orElse(null); + + if (value instanceof Number) { + return inclusive ? Boundary.including((Number) value) : Boundary.excluding((Number) value); + } + + if (value instanceof String) { + + if (!StringUtils.hasText((String) value) || ObjectUtils.nullSafeEquals(value, "+") + || ObjectUtils.nullSafeEquals(value, "-")) { + return Boundary.unbounded(); + } + + Object encoded = stringEncoder.apply((String) value); + return inclusive ? Boundary.including(encoded) : Boundary.excluding(encoded); + + } + + if (value == null) { + return Boundary.unbounded(); + } + + return inclusive ? Boundary.including((ByteBuffer) value) : Boundary.excluding((ByteBuffer) value); + }; + } +} diff --git a/src/main/java/org/springframework/data/redis/connection/lettuce/StreamConverters.java b/src/main/java/org/springframework/data/redis/connection/lettuce/StreamConverters.java new file mode 100644 index 000000000..78eb67479 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/connection/lettuce/StreamConverters.java @@ -0,0 +1,136 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.connection.lettuce; + +import io.lettuce.core.StreamMessage; +import io.lettuce.core.XReadArgs; + +import java.util.List; + +import org.springframework.core.convert.converter.Converter; +import org.springframework.data.redis.connection.RedisStreamCommands; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.connection.convert.ListConverter; + +/** + * Converters for Redis Stream-specific types. + *

+ * Converters typically convert between value objects/argument objects retaining the actual types of values (i.e. no + * serialization/deserialization happens here). + * + * @author Mark Paluch + * @since 2.2 + */ +@SuppressWarnings({ "unchecked", "rawtypes" }) +class StreamConverters { + + private static final ListConverter, RedisStreamCommands.StreamMessage> STREAM_LIST_CONVERTER = new ListConverter<>( + StreamMessageConverter.INSTANCE); + + /** + * Convert Lettuce's {@link StreamMessage} to {@link RedisStreamCommands.StreamMessage}. + * + * @param source must not be {@literal null}. + * @return the converted {@link RedisStreamCommands.StreamMessage}. + */ + static RedisStreamCommands.StreamMessage toStreamMessage(StreamMessage source) { + return StreamMessageConverter.INSTANCE.convert((StreamMessage) source); + } + + /** + * Convert Lettuce's {@link List} of {@link StreamMessage} to {@link RedisStreamCommands.StreamMessage}s. + * + * @param source must not be {@literal null}. + * @return the converted {@link List}. + */ + static List> toStreamMessages(List> source) { + return (List) STREAM_LIST_CONVERTER.convert((List) source); + } + + /** + * Convert {@link StreamReadOptions} to Lettuce's {@link XReadArgs}. + * + * @param readOptions must not be {@literal null}. + * @return the converted {@link XReadArgs}. + */ + static XReadArgs toReadArgs(StreamReadOptions readOptions) { + return StreamReadOptionsToXReadArgsConverter.INSTANCE.convert(readOptions); + } + + /** + * @return {@link Converter} to convert Lettuce {@link StreamMessage} into + * {@link org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage}. + */ + public static Converter, RedisStreamCommands.StreamMessage> streamMessageConverter() { + return (Converter) StreamMessageConverter.INSTANCE; + } + + /** + * @return {@link Converter} to convert a{@link List} of Lettuce {@link StreamMessage}s into a {@link List} of + * {@link org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage}. + */ + public static Converter>, List>> streamMessageListConverter() { + return (Converter) STREAM_LIST_CONVERTER; + } + + /** + * {@link Converter} to convert Lettuce's {@link StreamMessage} to {@link RedisStreamCommands.StreamMessage}. + */ + enum StreamMessageConverter + implements Converter, RedisStreamCommands.StreamMessage> { + INSTANCE; + + /* + * (non-Javadoc) + * @see org.springframework.core.convert.converter.Converter#convert(java.lang.Object) + */ + @Override + public RedisStreamCommands.StreamMessage convert(StreamMessage source) { + return new RedisStreamCommands.StreamMessage<>(source.getStream(), source.getId(), source.getBody()); + } + } + + /** + * {@link Converter} to convert {@link StreamReadOptions} to Lettuce's {@link XReadArgs}. + */ + enum StreamReadOptionsToXReadArgsConverter implements Converter { + + INSTANCE; + + /* + * (non-Javadoc) + * @see org.springframework.core.convert.converter.Converter#convert(java.lang.Object) + */ + @Override + public XReadArgs convert(StreamReadOptions source) { + + XReadArgs args = new XReadArgs(); + + if (source.isNoack()) { + args.noack(true); + } + + if (source.getBlock() != null) { + args.block(source.getBlock()); + } + + if (source.getCount() != null) { + args.count(source.getCount()); + } + return args; + } + } +} diff --git a/src/main/java/org/springframework/data/redis/core/AbstractOperations.java b/src/main/java/org/springframework/data/redis/core/AbstractOperations.java index e58770b67..769996e68 100644 --- a/src/main/java/org/springframework/data/redis/core/AbstractOperations.java +++ b/src/main/java/org/springframework/data/redis/core/AbstractOperations.java @@ -91,8 +91,8 @@ abstract class AbstractOperations { } @Nullable - T execute(RedisCallback callback, boolean b) { - return template.execute(callback, b); + T execute(RedisCallback callback, boolean exposeConnection) { + return template.execute(callback, exposeConnection); } public RedisOperations getOperations() { diff --git a/src/main/java/org/springframework/data/redis/core/BoundStreamOperations.java b/src/main/java/org/springframework/data/redis/core/BoundStreamOperations.java new file mode 100644 index 000000000..61c01907d --- /dev/null +++ b/src/main/java/org/springframework/data/redis/core/BoundStreamOperations.java @@ -0,0 +1,209 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.core; + +import java.util.List; +import java.util.Map; + +import org.springframework.data.domain.Range; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.connection.RedisZSetCommands.Limit; +import org.springframework.lang.Nullable; + +/** + * Redis stream specific operations bound to a certain key. + * + * @author Mark Paluch + * @since 2.2 + */ +public interface BoundStreamOperations { + + /** + * Acknowledge one or more messages as processed. + * + * @param group name of the consumer group. + * @param messageIds message Id's to acknowledge. + * @return length of acknowledged messages. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XACK + */ + @Nullable + Long acknowledge(String group, String... messageIds); + + /** + * Append a message to the stream {@code key}. + * + * @param body message body. + * @return the message Id. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XADD + */ + @Nullable + String add(Map body); + + /** + * Removes the specified entries from the stream. Returns the number of items deleted, that may be different from the + * number of IDs passed in case certain IDs do not exist. + * + * @param messageIds stream message Id's. + * @return number of removed entries. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XDEL + */ + @Nullable + Long delete(String... messageIds); + + /** + * Create a consumer group. + * + * @param readOffset + * @param group name of the consumer group. + * @return {@literal true} if successful. {@literal null} when used in pipeline / transaction. + */ + @Nullable + String createGroup(ReadOffset readOffset, String group); + + /** + * Delete a consumer from a consumer group. + * + * @param consumer consumer identified by group name and consumer key. + * @return {@literal true} if successful. {@literal null} when used in pipeline / transaction. + */ + @Nullable + Boolean deleteConsumer(Consumer consumer); + + /** + * Destroy a consumer group. + * + * @param group name of the consumer group. + * @return {@literal true} if successful. {@literal null} when used in pipeline / transaction. + */ + @Nullable + Boolean destroyGroup(String group); + + /** + * Get the length of a stream. + * + * @return length of the stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XLEN + */ + @Nullable + Long size(); + + /** + * Read messages from a stream within a specific {@link Range}. + * + * @param range must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XRANGE + */ + @Nullable + default List> range(Range range) { + return range(range, Limit.unlimited()); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit}. + * + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XRANGE + */ + @Nullable + List> range(Range range, Limit limit); + + /** + * Read messages from {@link ReadOffset}. + * + * @param readOffset the offset to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREAD + */ + @Nullable + default List> read(ReadOffset readOffset) { + return read(StreamReadOptions.empty(), readOffset); + } + + /** + * Read messages starting from {@link ReadOffset}. + * + * @param readOptions read arguments. + * @param readOffset the offset to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREAD + */ + @Nullable + List> read(StreamReadOptions readOptions, ReadOffset readOffset); + + /** + * Read messages starting from {@link ReadOffset}. using a consumer group. + * + * @param consumer consumer/group. + * @param readOffset the offset to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREADGROUP + */ + @Nullable + default List> read(Consumer consumer, ReadOffset readOffset) { + return read(consumer, StreamReadOptions.empty(), readOffset); + } + + /** + * Read messages starting from {@link ReadOffset}. using a consumer group. + * + * @param consumer consumer/group. + * @param readOptions read arguments. + * @param readOffset the offset to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREADGROUP + */ + @Nullable + List> read(Consumer consumer, StreamReadOptions readOptions, ReadOffset readOffset); + + /** + * Read messages from a stream within a specific {@link Range} in reverse order. + * + * @param range must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREVRANGE + */ + @Nullable + default List> reverseRange(Range range) { + return reverseRange(range, Limit.unlimited()); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit} in reverse order. + * + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREVRANGE + */ + @Nullable + List> reverseRange(Range range, Limit limit); + + /** + * Trims the stream to {@code count} elements. + * + * @param count length of the stream. + * @return number of removed entries. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XTRIM + */ + @Nullable + Long trim(long count); +} diff --git a/src/main/java/org/springframework/data/redis/core/DefaultBoundStreamOperations.java b/src/main/java/org/springframework/data/redis/core/DefaultBoundStreamOperations.java new file mode 100644 index 000000000..a7a460b14 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/core/DefaultBoundStreamOperations.java @@ -0,0 +1,182 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.core; + +import java.util.List; +import java.util.Map; + +import org.springframework.data.domain.Range; +import org.springframework.data.redis.connection.DataType; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.connection.RedisZSetCommands.Limit; +import org.springframework.lang.Nullable; + +/** + * Default implementation for {@link BoundStreamOperations}. + * + * @author Mark Paluch + * @since 2.2 + */ +class DefaultBoundStreamOperations extends DefaultBoundKeyOperations implements BoundStreamOperations { + + private final StreamOperations ops; + + /** + * Constructs a new DefaultBoundSetOperations instance. + * + * @param key + * @param operations + */ + DefaultBoundStreamOperations(K key, RedisOperations operations) { + + super(key, operations); + this.ops = operations.opsForStream(); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundStreamOperations#acknowledge(java.lang.String, java.lang.String[]) + */ + @Nullable + @Override + public Long acknowledge(String group, String... messageIds) { + return ops.acknowledge(getKey(), group, messageIds); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundStreamOperations#add(java.util.Map) + */ + @Nullable + @Override + public String add(Map body) { + return ops.add(getKey(), body); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundStreamOperations#delete(java.lang.String[]) + */ + @Nullable + @Override + public Long delete(String... messageIds) { + return ops.delete(getKey(), messageIds); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundStreamOperations#createGroup(org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset, java.lang.String) + */ + @Nullable + @Override + public String createGroup(ReadOffset readOffset, String group) { + return ops.createGroup(getKey(), readOffset, group); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundStreamOperations#deleteConsumer(org.springframework.data.redis.connection.RedisStreamCommands.Consumer) + */ + @Nullable + @Override + public Boolean deleteConsumer(Consumer consumer) { + return ops.deleteConsumer(getKey(), consumer); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundStreamOperations#destroyGroup(java.lang.String) + */ + @Nullable + @Override + public Boolean destroyGroup(String group) { + return ops.destroyGroup(getKey(), group); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundStreamOperations#size() + */ + @Nullable + @Override + public Long size() { + return ops.size(getKey()); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundStreamOperations#range(org.springframework.data.domain.Range, org.springframework.data.redis.connection.RedisZSetCommands.Limit) + */ + @Nullable + @Override + public List> range(Range range, Limit limit) { + return ops.range(getKey(), range, limit); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundStreamOperations#read(org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions, org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset) + */ + @Nullable + @Override + public List> read(StreamReadOptions readOptions, ReadOffset readOffset) { + return ops.read(readOptions, StreamOffset.create(getKey(), readOffset)); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundStreamOperations#read(org.springframework.data.redis.connection.RedisStreamCommands.Consumer, org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions, org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset) + */ + @Nullable + @Override + public List> read(Consumer consumer, StreamReadOptions readOptions, ReadOffset readOffset) { + return ops.read(consumer, readOptions, StreamOffset.create(getKey(), readOffset)); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundStreamOperations#reverseRange(org.springframework.data.domain.Range, org.springframework.data.redis.connection.RedisZSetCommands.Limit) + */ + @Nullable + @Override + public List> reverseRange(Range range, Limit limit) { + return ops.reverseRange(getKey(), range, limit); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundStreamOperations#trim(long) + */ + @Nullable + @Override + public Long trim(long count) { + return ops.trim(getKey(), count); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.BoundKeyOperations#getType() + */ + @Nullable + @Override + public DataType getType() { + return DataType.STREAM; + } +} diff --git a/src/main/java/org/springframework/data/redis/core/DefaultReactiveStreamOperations.java b/src/main/java/org/springframework/data/redis/core/DefaultReactiveStreamOperations.java new file mode 100644 index 000000000..547e2bcc6 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/core/DefaultReactiveStreamOperations.java @@ -0,0 +1,232 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.core; + +import lombok.NonNull; +import lombok.RequiredArgsConstructor; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +import java.nio.ByteBuffer; +import java.util.Arrays; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.function.Function; + +import org.reactivestreams.Publisher; +import org.springframework.data.domain.Range; +import org.springframework.data.redis.connection.ReactiveStreamCommands; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.connection.RedisZSetCommands.Limit; +import org.springframework.data.redis.serializer.RedisSerializationContext; +import org.springframework.util.Assert; + +/** + * Default implementation of {@link ReactiveStreamOperations}. + * + * @author Mark Paluch + * @since 2.2 + */ +@RequiredArgsConstructor +class DefaultReactiveStreamOperations implements ReactiveStreamOperations { + + private final @NonNull ReactiveRedisTemplate template; + private final @NonNull RedisSerializationContext serializationContext; + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.ReactiveStreamOperations#acknowledge(java.lang.Object, java.lang.String, java.lang.String[]) + */ + @Override + public Mono acknowledge(K key, String group, String... messageIds) { + + Assert.notNull(key, "Key must not be null!"); + Assert.hasText(group, "Group must not be null or empty!"); + Assert.notNull(messageIds, "MessageIds must not be null!"); + Assert.notEmpty(messageIds, "MessageIds must not be empty!"); + + return createMono(connection -> connection.xAck(rawKey(key), group, messageIds)); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.ReactiveStreamOperations#add(java.lang.Object, java.util.Map) + */ + @Override + public Mono add(K key, Map body) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(body, "Body must not be null!"); + Assert.isTrue(!body.isEmpty(), "Body must not be empty!"); + + Map rawBody = new LinkedHashMap<>(body.size()); + + body.forEach((k, v) -> rawBody.put(rawKey(k), rawValue(v))); + + return createMono(connection -> connection.xAdd(rawKey(key), rawBody)); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.ReactiveStreamOperations#delete(java.lang.Object, java.lang.String[]) + */ + @Override + public Mono delete(K key, String... messageIds) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(messageIds, "MessageIds must not be null!"); + + return createMono(connection -> connection.xDel(rawKey(key), messageIds)); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.ReactiveStreamOperations#size(java.lang.Object) + */ + @Override + public Mono size(K key) { + + Assert.notNull(key, "Key must not be null!"); + + return createMono(connection -> connection.xLen(rawKey(key))); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.ReactiveStreamOperations#range(java.lang.Object, org.springframework.data.domain.Range, org.springframework.data.redis.connection.RedisZSetCommands.Limit) + */ + @Override + public Flux> range(K key, Range range, Limit limit) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(range, "Range must not be null!"); + Assert.notNull(limit, "Limit must not be null!"); + + return createFlux(connection -> connection.xRange(rawKey(key), range, limit).map(this::readValue)); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.ReactiveStreamOperations#read(org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions, org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset[]) + */ + @Override + public Flux> read(StreamReadOptions readOptions, StreamOffset... streams) { + + Assert.notNull(readOptions, "StreamReadOptions must not be null!"); + Assert.notNull(streams, "Streams must not be null!"); + + return createFlux(connection -> { + + StreamOffset[] streamOffsets = rawStreamOffsets(streams); + + return connection.xRead(readOptions, streamOffsets).map(this::readValue); + }); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.ReactiveStreamOperations#read(org.springframework.data.redis.connection.RedisStreamCommands.Consumer, org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions, org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset[]) + */ + @Override + public Flux> read(Consumer consumer, StreamReadOptions readOptions, StreamOffset... streams) { + + Assert.notNull(consumer, "Consumer must not be null!"); + Assert.notNull(readOptions, "StreamReadOptions must not be null!"); + Assert.notNull(streams, "Streams must not be null!"); + + return createFlux(connection -> { + + StreamOffset[] streamOffsets = rawStreamOffsets(streams); + + return connection.xReadGroup(consumer, readOptions, streamOffsets).map(this::readValue); + }); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.ReactiveStreamOperations#reverseRange(java.lang.Object, org.springframework.data.domain.Range, org.springframework.data.redis.connection.RedisZSetCommands.Limit) + */ + @Override + public Flux> reverseRange(K key, Range range, Limit limit) { + + Assert.notNull(key, "Key must not be null!"); + Assert.notNull(range, "Range must not be null!"); + Assert.notNull(limit, "Limit must not be null!"); + + return createFlux(connection -> connection.xRevRange(rawKey(key), range, limit).map(this::readValue)); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.ReactiveStreamOperations#trim(java.lang.Object, long) + */ + @Override + public Mono trim(K key, long count) { + + Assert.notNull(key, "Key must not be null!"); + + return createMono(connection -> connection.xTrim(rawKey(key), count)); + } + + @SuppressWarnings("unchecked") + private StreamOffset[] rawStreamOffsets(StreamOffset[] streams) { + + return Arrays.stream(streams).map(it -> StreamOffset.create(rawKey(it.getKey()), it.getOffset())) + .toArray(StreamOffset[]::new); + } + + private StreamMessage readValue(StreamMessage message) { + + Map body = new LinkedHashMap<>(message.getBody().size()); + + message.getBody().forEach((k, v) -> body.put(readKey(k), readValue(v))); + + return new StreamMessage<>(readKey(message.getStream()), message.getId(), body); + } + + private Mono createMono(Function> function) { + + Assert.notNull(function, "Function must not be null!"); + + return template.createMono(connection -> function.apply(connection.streamCommands())); + } + + private Flux createFlux(Function> function) { + + Assert.notNull(function, "Function must not be null!"); + + return template.createFlux(connection -> function.apply(connection.streamCommands())); + } + + private ByteBuffer rawKey(K key) { + return serializationContext.getKeySerializationPair().write(key); + } + + private ByteBuffer rawValue(V value) { + return serializationContext.getValueSerializationPair().write(value); + } + + private K readKey(ByteBuffer buffer) { + return serializationContext.getKeySerializationPair().read(buffer); + } + + private V readValue(ByteBuffer buffer) { + return serializationContext.getValueSerializationPair().read(buffer); + } +} diff --git a/src/main/java/org/springframework/data/redis/core/DefaultStreamOperations.java b/src/main/java/org/springframework/data/redis/core/DefaultStreamOperations.java new file mode 100644 index 000000000..517af3be8 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/core/DefaultStreamOperations.java @@ -0,0 +1,249 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.core; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import org.springframework.data.domain.Range; +import org.springframework.data.redis.connection.RedisConnection; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.connection.RedisZSetCommands.Limit; +import org.springframework.lang.Nullable; + +/** + * Default implementation of {@link ListOperations}. + * + * @author Mark Paluch + * @since 2.2 + */ +class DefaultStreamOperations extends AbstractOperations implements StreamOperations { + + DefaultStreamOperations(RedisTemplate template) { + super(template); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.StreamOperations#acknowledge(java.lang.Object, java.lang.String, java.lang.String[]) + */ + @Override + public Long acknowledge(K key, String group, String... messageIds) { + + byte[] rawKey = rawKey(key); + return execute(connection -> connection.xAck(rawKey, group, messageIds), true); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.StreamOperations#add(java.lang.Object, java.util.Map) + */ + @Override + public String add(K key, Map body) { + + byte[] rawKey = rawKey(key); + Map rawBody = new LinkedHashMap<>(body.size()); + + for (Map.Entry entry : body.entrySet()) { + rawBody.put(rawKey(entry.getKey()), rawValue(entry.getValue())); + } + + return execute(connection -> connection.xAdd(rawKey, rawBody), true); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.StreamOperations#delete(java.lang.Object, java.lang.String[]) + */ + @Override + public Long delete(K key, String... messageIds) { + + byte[] rawKey = rawKey(key); + return execute(connection -> connection.xDel(rawKey, messageIds), true); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.StreamOperations#createGroup(java.lang.Object, org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset, java.lang.String) + */ + @Override + public String createGroup(K key, ReadOffset readOffset, String group) { + + byte[] rawKey = rawKey(key); + return execute(connection -> connection.xGroupCreate(rawKey, readOffset, group), true); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.StreamOperations#deleteConsumer(java.lang.Object, org.springframework.data.redis.connection.RedisStreamCommands.Consumer) + */ + @Override + public Boolean deleteConsumer(K key, Consumer consumer) { + + byte[] rawKey = rawKey(key); + return execute(connection -> connection.xGroupDelConsumer(rawKey, consumer), true); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.StreamOperations#destroyGroup(java.lang.Object, java.lang.String) + */ + @Override + public Boolean destroyGroup(K key, String group) { + + byte[] rawKey = rawKey(key); + return execute(connection -> connection.xGroupDestroy(rawKey, group), true); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.StreamOperations#size(java.lang.Object) + */ + @Override + public Long size(K key) { + + byte[] rawKey = rawKey(key); + return execute(connection -> connection.xLen(rawKey), true); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.StreamOperations#range(java.lang.Object, org.springframework.data.domain.Range, org.springframework.data.redis.connection.RedisZSetCommands.Limit) + */ + @Override + public List> range(K key, Range range, Limit limit) { + + return execute(new StreamMessagesDeserializingRedisCallback() { + @Nullable + @Override + List> inRedis(RedisConnection connection) { + return connection.xRange(rawKey(key), range, limit); + } + }, true); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.StreamOperations#read(org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions, org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset[]) + */ + @Override + public List> read(StreamReadOptions readOptions, StreamOffset... streams) { + + return execute(new StreamMessagesDeserializingRedisCallback() { + @Nullable + @Override + List> inRedis(RedisConnection connection) { + return connection.xRead(readOptions, rawStreamOffsets(streams)); + } + }, true); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.StreamOperations#read(org.springframework.data.redis.connection.RedisStreamCommands.Consumer, org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions, org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset[]) + */ + @Override + public List> read(Consumer consumer, StreamReadOptions readOptions, StreamOffset... streams) { + + return execute(new StreamMessagesDeserializingRedisCallback() { + @Nullable + @Override + List> inRedis(RedisConnection connection) { + return connection.xReadGroup(consumer, readOptions, rawStreamOffsets(streams)); + } + }, true); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.StreamOperations#reverseRange(java.lang.Object, org.springframework.data.domain.Range, org.springframework.data.redis.connection.RedisZSetCommands.Limit) + */ + @Override + public List> reverseRange(K key, Range range, Limit limit) { + + return execute(new StreamMessagesDeserializingRedisCallback() { + @Nullable + @Override + List> inRedis(RedisConnection connection) { + return connection.xRevRange(rawKey(key), range, limit); + } + }, true); + } + + @Override + public Long trim(K key, long count) { + + byte[] rawKey = rawKey(key); + return execute(connection -> connection.xTrim(rawKey, count), true); + } + + @SuppressWarnings("unchecked") + private Map deserializeBody(@Nullable Map entries) { + // connection in pipeline/multi mode + + if (entries == null) { + return null; + } + + Map map = new LinkedHashMap<>(entries.size()); + + for (Map.Entry entry : entries.entrySet()) { + map.put(deserializeKey(entry.getKey()), deserializeValue(entry.getValue())); + } + + return map; + } + + @SuppressWarnings("unchecked") + private StreamOffset[] rawStreamOffsets(StreamOffset[] streams) { + + return Arrays.stream(streams) // + .map(it -> StreamOffset.create(rawKey(it.getKey()), it.getOffset())) // + .toArray(it -> new StreamOffset[it]); + } + + abstract class StreamMessagesDeserializingRedisCallback implements RedisCallback>> { + + public final List> doInRedis(RedisConnection connection) { + + List> streamMessages = inRedis(connection); + + if (streamMessages == null) { + return null; + } + + List> result = new ArrayList<>(streamMessages.size()); + + for (StreamMessage streamMessage : streamMessages) { + + result.add(new StreamMessage<>(deserializeKey(streamMessage.getStream()), streamMessage.getId(), + deserializeBody(streamMessage.getBody()))); + } + + return result; + } + + @Nullable + abstract List> inRedis(RedisConnection connection); + } +} diff --git a/src/main/java/org/springframework/data/redis/core/ReactiveRedisOperations.java b/src/main/java/org/springframework/data/redis/core/ReactiveRedisOperations.java index 4d407bc8b..54917536e 100644 --- a/src/main/java/org/springframework/data/redis/core/ReactiveRedisOperations.java +++ b/src/main/java/org/springframework/data/redis/core/ReactiveRedisOperations.java @@ -428,6 +428,23 @@ public interface ReactiveRedisOperations { */ ReactiveSetOperations opsForSet(RedisSerializationContext serializationContext); + /** + * Returns the operations performed on streams. + * + * @return stream operations. + * @since 2.2 + */ + ReactiveStreamOperations opsForStream(); + + /** + * Returns the operations performed on streams given a {@link RedisSerializationContext}. + * + * @param serializationContext serializers to be used with the returned operations, must not be {@literal null}. + * @return stream operations. + * @since 2.2 + */ + ReactiveStreamOperations opsForStream(RedisSerializationContext serializationContext); + /** * Returns the operations performed on simple values (or Strings in Redis terminology). * diff --git a/src/main/java/org/springframework/data/redis/core/ReactiveRedisTemplate.java b/src/main/java/org/springframework/data/redis/core/ReactiveRedisTemplate.java index 87d39b215..05fca8330 100644 --- a/src/main/java/org/springframework/data/redis/core/ReactiveRedisTemplate.java +++ b/src/main/java/org/springframework/data/redis/core/ReactiveRedisTemplate.java @@ -621,6 +621,25 @@ public class ReactiveRedisTemplate implements ReactiveRedisOperations(this, serializationContext); } + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.ReactiveRedisOperations#opsForStream() + */ + @Override + public ReactiveStreamOperations opsForStream() { + return opsForStream(serializationContext); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.ReactiveRedisOperations#opsForStream(org.springframework.data.redis.serializer.RedisSerializationContext) + */ + @Override + public ReactiveStreamOperations opsForStream( + RedisSerializationContext serializationContext) { + return new DefaultReactiveStreamOperations<>(this, serializationContext); + } + /* * (non-Javadoc) * @see org.springframework.data.redis.core.ReactiveRedisOperations#opsForValue() diff --git a/src/main/java/org/springframework/data/redis/core/ReactiveStreamOperations.java b/src/main/java/org/springframework/data/redis/core/ReactiveStreamOperations.java new file mode 100644 index 000000000..ca04e393c --- /dev/null +++ b/src/main/java/org/springframework/data/redis/core/ReactiveStreamOperations.java @@ -0,0 +1,239 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.core; + +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +import java.util.Map; + +import org.reactivestreams.Publisher; +import org.springframework.data.domain.Range; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.connection.RedisZSetCommands.Limit; + +/** + * Redis stream specific operations. + * + * @author Mark Paluch + * @since 2.2 + */ +public interface ReactiveStreamOperations { + + /** + * Acknowledge one or more messages as processed. + * + * @param key the stream key. + * @param group name of the consumer group. + * @param messageIds message Id's to acknowledge. + * @return length of acknowledged messages. + * @see Redis Documentation: XACK + */ + Mono acknowledge(K key, String group, String... messageIds); + + /** + * Append one or more message to the stream {@code key}. + * + * @param key the stream key. + * @param bodyPublisher message body {@link Publisher}. + * @return the message Ids. + * @see Redis Documentation: XADD + */ + default Flux add(K key, Publisher> bodyPublisher) { + return Flux.from(bodyPublisher).flatMap(it -> add(key, it)); + } + + /** + * Append a message to the stream {@code key}. + * + * @param key the stream key. + * @param body message body. + * @return the message Id. + * @see Redis Documentation: XADD + */ + Mono add(K key, Map body); + + /** + * Removes the specified entries from the stream. Returns the number of items deleted, that may be different from the + * number of IDs passed in case certain IDs do not exist. + * + * @param key the stream key. + * @param messageIds stream message Id's. + * @return number of removed entries. + * @see Redis Documentation: XDEL + */ + Mono delete(K key, String... messageIds); + + /** + * Get the length of a stream. + * + * @param key the stream key. + * @return length of the stream. + * @see Redis Documentation: XLEN + */ + Mono size(K key); + + /** + * Read messages from a stream within a specific {@link Range}. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @return list with members of the resulting stream. + * @see Redis Documentation: XRANGE + */ + default Flux> range(K key, Range range) { + return range(key, range, Limit.unlimited()); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit}. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + * @return list with members of the resulting stream. + * @see Redis Documentation: XRANGE + */ + Flux> range(K key, Range range, Limit limit); + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param stream the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREAD + */ + default Flux> read(StreamOffset stream) { + return read(StreamReadOptions.empty(), new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param streams the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREAD + */ + default Flux> read(StreamOffset... streams) { + return read(StreamReadOptions.empty(), streams); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param readOptions read arguments. + * @param stream the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREAD + */ + default Flux> read(StreamReadOptions readOptions, StreamOffset stream) { + return read(readOptions, new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param readOptions read arguments. + * @param streams the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREAD + */ + Flux> read(StreamReadOptions readOptions, StreamOffset... streams); + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param stream the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREADGROUP + */ + default Flux> read(Consumer consumer, StreamOffset stream) { + return read(consumer, StreamReadOptions.empty(), new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param streams the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREADGROUP + */ + default Flux> read(Consumer consumer, StreamOffset... streams) { + return read(consumer, StreamReadOptions.empty(), streams); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param readOptions read arguments. + * @param stream the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREADGROUP + */ + default Flux> read(Consumer consumer, StreamReadOptions readOptions, StreamOffset stream) { + return read(consumer, readOptions, new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param readOptions read arguments. + * @param streams the streams to read from. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREADGROUP + */ + Flux> read(Consumer consumer, StreamReadOptions readOptions, StreamOffset... streams); + + /** + * Read messages from a stream within a specific {@link Range} in reverse order. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREVRANGE + */ + default Flux> reverseRange(K key, Range range) { + return reverseRange(key, range, Limit.unlimited()); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit} in reverse order. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + * @return list with members of the resulting stream. + * @see Redis Documentation: XREVRANGE + */ + Flux> reverseRange(K key, Range range, Limit limit); + + /** + * Trims the stream to {@code count} elements. + * + * @param key the stream key. + * @param count length of the stream. + * @return number of removed entries. + * @see Redis Documentation: XTRIM + */ + Mono trim(K key, long count); +} diff --git a/src/main/java/org/springframework/data/redis/core/RedisOperations.java b/src/main/java/org/springframework/data/redis/core/RedisOperations.java index 16625aa2b..969fb6146 100644 --- a/src/main/java/org/springframework/data/redis/core/RedisOperations.java +++ b/src/main/java/org/springframework/data/redis/core/RedisOperations.java @@ -616,6 +616,22 @@ public interface RedisOperations { */ BoundSetOperations boundSetOps(K key); + /** + * Returns the operations performed on Streams. + * + * @return stream operations. + * @since 2.2 + */ + StreamOperations opsForStream(); + + /** + * Returns the operations performed on Streams bound to the given key. + * + * @return stream operations. + * @since 2.2 + */ + BoundStreamOperations boundStreamOps(K key); + /** * Returns the operations performed on simple values (or Strings in Redis terminology). * diff --git a/src/main/java/org/springframework/data/redis/core/RedisTemplate.java b/src/main/java/org/springframework/data/redis/core/RedisTemplate.java index 672a70846..8851c08d5 100644 --- a/src/main/java/org/springframework/data/redis/core/RedisTemplate.java +++ b/src/main/java/org/springframework/data/redis/core/RedisTemplate.java @@ -104,6 +104,7 @@ public class RedisTemplate extends RedisAccessor implements RedisOperation private @Nullable ValueOperations valueOps; private @Nullable ListOperations listOps; private @Nullable SetOperations setOps; + private @Nullable StreamOperations streamOps; private @Nullable ZSetOperations zSetOps; private @Nullable GeoOperations geoOps; private @Nullable HyperLogLogOperations hllOps; @@ -1300,6 +1301,28 @@ public class RedisTemplate extends RedisAccessor implements RedisOperation return setOps; } + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.RedisOperations#opsForStream() + */ + @Override + public StreamOperations opsForStream() { + + if (streamOps == null) { + streamOps = new DefaultStreamOperations<>(this); + } + return streamOps; + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.core.RedisOperations#boundStreamOps(java.lang.Object) + */ + @Override + public BoundStreamOperations boundStreamOps(K key) { + return new DefaultBoundStreamOperations<>(key, this); + } + /* * (non-Javadoc) * @see org.springframework.data.redis.core.RedisOperations#boundValueOps(java.lang.Object) diff --git a/src/main/java/org/springframework/data/redis/core/StreamOperations.java b/src/main/java/org/springframework/data/redis/core/StreamOperations.java new file mode 100644 index 000000000..4e2076fb6 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/core/StreamOperations.java @@ -0,0 +1,274 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.core; + +import java.util.List; +import java.util.Map; + +import org.springframework.data.domain.Range; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.connection.RedisZSetCommands.Limit; +import org.springframework.lang.Nullable; + +/** + * Redis stream specific operations. + * + * @author Mark Paluch + * @since 2.2 + */ +public interface StreamOperations { + + /** + * Acknowledge one or more messages as processed. + * + * @param key the stream key. + * @param group name of the consumer group. + * @param messageIds message Id's to acknowledge. + * @return length of acknowledged messages. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XACK + */ + @Nullable + Long acknowledge(K key, String group, String... messageIds); + + /** + * Append a message to the stream {@code key}. + * + * @param key the stream key. + * @param body message body. + * @return the message Id. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XADD + */ + @Nullable + String add(K key, Map body); + + /** + * Removes the specified entries from the stream. Returns the number of items deleted, that may be different from the + * number of IDs passed in case certain IDs do not exist. + * + * @param key the stream key. + * @param messageIds stream message Id's. + * @return number of removed entries. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XDEL + */ + @Nullable + Long delete(K key, String... messageIds); + + /** + * Create a consumer group. + * + * @param key + * @param readOffset + * @param group name of the consumer group. + * @return {@literal true} if successful. {@literal null} when used in pipeline / transaction. + */ + @Nullable + String createGroup(K key, ReadOffset readOffset, String group); + + /** + * Delete a consumer from a consumer group. + * + * @param key the stream key. + * @param consumer consumer identified by group name and consumer key. + * @return {@literal true} if successful. {@literal null} when used in pipeline / transaction. + */ + @Nullable + Boolean deleteConsumer(K key, Consumer consumer); + + /** + * Destroy a consumer group. + * + * @param key the stream key. + * @param group name of the consumer group. + * @return {@literal true} if successful. {@literal null} when used in pipeline / transaction. + */ + @Nullable + Boolean destroyGroup(K key, String group); + + /** + * Get the length of a stream. + * + * @param key the stream key. + * @return length of the stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XLEN + */ + @Nullable + Long size(K key); + + /** + * Read messages from a stream within a specific {@link Range}. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XRANGE + */ + @Nullable + default List> range(K key, Range range) { + return range(key, range, Limit.unlimited()); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit}. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XRANGE + */ + @Nullable + List> range(K key, Range range, Limit limit); + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param stream the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREAD + */ + @Nullable + default List> read(StreamOffset stream) { + return read(StreamReadOptions.empty(), new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param streams the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREAD + */ + @Nullable + default List> read(StreamOffset... streams) { + return read(StreamReadOptions.empty(), streams); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param readOptions read arguments. + * @param stream the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREAD + */ + @Nullable + default List> read(StreamReadOptions readOptions, StreamOffset stream) { + return read(readOptions, new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s. + * + * @param readOptions read arguments. + * @param streams the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREAD + */ + @Nullable + List> read(StreamReadOptions readOptions, StreamOffset... streams); + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param stream the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREADGROUP + */ + @Nullable + default List> read(Consumer consumer, StreamOffset stream) { + return read(consumer, StreamReadOptions.empty(), new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param streams the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREADGROUP + */ + @Nullable + default List> read(Consumer consumer, StreamOffset... streams) { + return read(consumer, StreamReadOptions.empty(), streams); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param readOptions read arguments. + * @param stream the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREADGROUP + */ + @Nullable + default List> read(Consumer consumer, StreamReadOptions readOptions, StreamOffset stream) { + return read(consumer, readOptions, new StreamOffset[] { stream }); + } + + /** + * Read messages from one or more {@link StreamOffset}s using a consumer group. + * + * @param consumer consumer/group. + * @param readOptions read arguments. + * @param streams the streams to read from. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREADGROUP + */ + @Nullable + List> read(Consumer consumer, StreamReadOptions readOptions, StreamOffset... streams); + + /** + * Read messages from a stream within a specific {@link Range} in reverse order. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREVRANGE + */ + @Nullable + default List> reverseRange(K key, Range range) { + return reverseRange(key, range, Limit.unlimited()); + } + + /** + * Read messages from a stream within a specific {@link Range} applying a {@link Limit} in reverse order. + * + * @param key the stream key. + * @param range must not be {@literal null}. + * @param limit must not be {@literal null}. + * @return list with members of the resulting stream. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XREVRANGE + */ + @Nullable + List> reverseRange(K key, Range range, Limit limit); + + /** + * Trims the stream to {@code count} elements. + * + * @param key the stream key. + * @param count length of the stream. + * @return number of removed entries. {@literal null} when used in pipeline / transaction. + * @see Redis Documentation: XTRIM + */ + @Nullable + Long trim(K key, long count); +} diff --git a/src/main/java/org/springframework/data/redis/stream/Cancelable.java b/src/main/java/org/springframework/data/redis/stream/Cancelable.java new file mode 100644 index 000000000..c392d15e7 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/stream/Cancelable.java @@ -0,0 +1,34 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import org.springframework.dao.DataAccessResourceFailureException; + +/** + * Cancelable allows stopping long running tasks and freeing underlying resources. + * + * @author Mark Paluch + * @since 2.2 + */ +public interface Cancelable { + + /** + * Abort and free resources. + * + * @throws DataAccessResourceFailureException + */ + void cancel() throws DataAccessResourceFailureException; +} diff --git a/src/main/java/org/springframework/data/redis/stream/DefaultStreamMessageListenerContainer.java b/src/main/java/org/springframework/data/redis/stream/DefaultStreamMessageListenerContainer.java new file mode 100644 index 000000000..2ae72ed0c --- /dev/null +++ b/src/main/java/org/springframework/data/redis/stream/DefaultStreamMessageListenerContainer.java @@ -0,0 +1,319 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import lombok.EqualsAndHashCode; +import lombok.RequiredArgsConstructor; + +import java.time.Duration; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.Executor; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.springframework.dao.DataAccessResourceFailureException; +import org.springframework.data.redis.connection.RedisConnectionFactory; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.core.RedisTemplate; +import org.springframework.data.redis.core.StreamOperations; +import org.springframework.util.Assert; +import org.springframework.util.ErrorHandler; + +/** + * Simple {@link Executor} based {@link StreamMessageListenerContainer} implementation for running {@link Task tasks} to + * poll on Redis Streams. + *

+ * This message container creates long-running tasks that are executed on {@link Executor}. + * + * @author Mark Paluch + * @since 2.2 + */ +class DefaultStreamMessageListenerContainer implements StreamMessageListenerContainer { + + private final Object lifecycleMonitor = new Object(); + + private final Executor taskExecutor; + private final ErrorHandler errorHandler; + private final StreamReadOptions readOptions; + private final RedisTemplate template; + + private final List subscriptions = new ArrayList<>(); + + private boolean running = false; + + /** + * Create a new {@link DefaultStreamMessageListenerContainer}. + * + * @param connectionFactory must not be {@literal null}. + * @param containerOptions must not be {@literal null}. + */ + DefaultStreamMessageListenerContainer(RedisConnectionFactory connectionFactory, + StreamMessageListenerContainerOptions containerOptions) { + + Assert.notNull(connectionFactory, "RedisConnectionFactory must not be null!"); + Assert.notNull(containerOptions, "StreamMessageListenerContainerOptions must not be null!"); + + this.taskExecutor = containerOptions.getExecutor(); + this.errorHandler = containerOptions.getErrorHandler(); + this.readOptions = getStreamReadOptions(containerOptions); + this.template = createRedisTemplate(connectionFactory, containerOptions); + } + + private static StreamReadOptions getStreamReadOptions(StreamMessageListenerContainerOptions options) { + + StreamReadOptions streamReadOptions = StreamReadOptions.empty().count(options.getBatchSize()); + + if (!options.getPollTimeout().isZero()) { + streamReadOptions = streamReadOptions.block(options.getPollTimeout()); + } + + return streamReadOptions; + } + + private RedisTemplate createRedisTemplate(RedisConnectionFactory connectionFactory, + StreamMessageListenerContainerOptions containerOptions) { + + RedisTemplate template = new RedisTemplate<>(); + template.setKeySerializer(containerOptions.getKeySerializer()); + template.setValueSerializer(containerOptions.getBodySerializer()); + template.setConnectionFactory(connectionFactory); + template.afterPropertiesSet(); + + return template; + } + + /* + * (non-Javadoc) + * @see org.springframework.context.SmartLifecycle#isAutoStartup() + */ + @Override + public boolean isAutoStartup() { + return false; + } + + /* + * (non-Javadoc) + * @see org.springframework.context.SmartLifecycle#stop(java.lang.Runnable) + */ + @Override + public void stop(Runnable callback) { + + stop(); + callback.run(); + } + + /* + * (non-Javadoc) + * @see org.springframework.context.Lifecycle#start() + */ + @Override + public void start() { + + synchronized (lifecycleMonitor) { + + if (this.running) { + return; + } + + subscriptions.stream() // + .filter(it -> !it.isActive()) // + .filter(it -> it instanceof TaskSubscription) // + .map(TaskSubscription.class::cast) // + .map(TaskSubscription::getTask) // + .forEach(taskExecutor::execute); + + running = true; + } + } + + /* + * (non-Javadoc) + * @see org.springframework.context.Lifecycle#stop() + */ + @Override + public void stop() { + + synchronized (lifecycleMonitor) { + + if (this.running) { + + subscriptions.forEach(Cancelable::cancel); + + running = false; + } + } + } + + /* + * (non-Javadoc) + * @see org.springframework.context.Lifecycle#isRunning() + */ + @Override + public boolean isRunning() { + + synchronized (this.lifecycleMonitor) { + return running; + } + } + + /* + * (non-Javadoc) + * @see org.springframework.context.Phased#getPhase() + */ + @Override + public int getPhase() { + return Integer.MAX_VALUE; + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.stream.StreamMessageListenerContainer#register(org.springframework.data.redis.stream.StreamMessageListenerContainer.StreamReadRequest, org.springframework.data.redis.stream.StreamListener) + */ + @Override + public Subscription register(StreamReadRequest streamRequest, StreamListener listener) { + return doRegister(getReadTask(streamRequest, listener)); + } + + private StreamPollTask getReadTask(StreamReadRequest streamRequest, StreamListener listener) { + + StreamOperations streamOperations = template.opsForStream(); + + if (streamRequest instanceof StreamMessageListenerContainer.ConsumerStreamReadRequest) { + + ConsumerStreamReadRequest consumerStreamRequest = (ConsumerStreamReadRequest) streamRequest; + + StreamReadOptions readOptions = consumerStreamRequest.isAutoAck() ? this.readOptions : this.readOptions.noack(); + Consumer consumer = consumerStreamRequest.getConsumer(); + + return new StreamPollTask<>(consumerStreamRequest, listener, errorHandler, + (key, offset) -> streamOperations.read(consumer, readOptions, StreamOffset.create(key, offset))); + } + + return new StreamPollTask<>(streamRequest, listener, errorHandler, + (key, offset) -> streamOperations.read(readOptions, StreamOffset.create(key, offset))); + } + + private Subscription doRegister(Task task) { + + Subscription subscription = new TaskSubscription(task); + + synchronized (lifecycleMonitor) { + + this.subscriptions.add(subscription); + + if (this.running) { + taskExecutor.execute(task); + } + } + + return subscription; + } + + /* + * (non-Javadoc) + * @see org.springframework.data.mongodb.monitor.MessageListenerContainer#remove(org.springframework.data.mongodb.monitor.Subscription) + */ + @Override + public void remove(Subscription subscription) { + + synchronized (lifecycleMonitor) { + + if (subscriptions.contains(subscription)) { + + if (subscription.isActive()) { + subscription.cancel(); + } + + subscriptions.remove(subscription); + } + } + } + + /** + * {@link Subscription} wrapping a {@link Task}. + * + * @author Mark Paluch + * @since 2.2 + */ + @EqualsAndHashCode + @RequiredArgsConstructor + static class TaskSubscription implements Subscription { + + private final Task task; + + Task getTask() { + return task; + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.stream.Subscription#isActive() + */ + @Override + public boolean isActive() { + return task.isActive(); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.stream.Subscription#await(java.time.Duration) + */ + @Override + public boolean await(Duration timeout) throws InterruptedException { + return task.awaitStart(timeout); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.stream.Cancelable#cancel() + */ + @Override + public void cancel() throws DataAccessResourceFailureException { + task.cancel(); + } + } + + /** + * Logging {@link ErrorHandler}. + * + * @author Mark Paluch + * @since 2.2 + */ + enum LoggingErrorHandler implements ErrorHandler { + + INSTANCE; + + private final Log logger; + + LoggingErrorHandler() { + this.logger = LogFactory.getLog(LoggingErrorHandler.class); + } + + /* + * (non-Javadoc) + * @see org.springframework.util.ErrorHandler#handleError(java.lang.Throwable) + */ + public void handleError(Throwable t) { + + if (this.logger.isErrorEnabled()) { + this.logger.error("Unexpected error occurred in scheduled task.", t); + } + } + } +} diff --git a/src/main/java/org/springframework/data/redis/stream/DefaultStreamReceiver.java b/src/main/java/org/springframework/data/redis/stream/DefaultStreamReceiver.java new file mode 100644 index 000000000..ca009c95d --- /dev/null +++ b/src/main/java/org/springframework/data/redis/stream/DefaultStreamReceiver.java @@ -0,0 +1,539 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import lombok.RequiredArgsConstructor; +import reactor.core.CoreSubscriber; +import reactor.core.publisher.Flux; +import reactor.core.publisher.FluxSink; +import reactor.core.publisher.Operators; +import reactor.util.concurrent.Queues; +import reactor.util.context.Context; + +import java.util.Optional; +import java.util.Queue; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; +import java.util.concurrent.atomic.AtomicReference; +import java.util.function.BiFunction; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.reactivestreams.Subscription; +import org.springframework.data.redis.connection.ReactiveRedisConnectionFactory; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.core.ReactiveRedisTemplate; +import org.springframework.data.redis.serializer.RedisSerializationContext; + +/** + * Default implementation of {@link StreamReceiver}. + * + * @author Mark Paluch + * @since 2.2 + */ +class DefaultStreamReceiver implements StreamReceiver { + + private final Log logger = LogFactory.getLog(getClass()); + private final ReactiveRedisTemplate template; + private final StreamReadOptions readOptions; + + /** + * Create a new {@link DefaultStreamReceiver} given {@link ReactiveRedisConnectionFactory} and + * {@link org.springframework.data.redis.stream.StreamReceiver.StreamReceiverOptions}. + * + * @param connectionFactory must not be {@literal null}. + * @param options must not be {@literal null}. + */ + DefaultStreamReceiver(ReactiveRedisConnectionFactory connectionFactory, StreamReceiverOptions options) { + + RedisSerializationContext serializationContext = RedisSerializationContext + . newSerializationContext(options.getKeySerializer()) // + .key(options.getKeySerializer()) // + .value(options.getBodySerializer()) // + .build(); + + StreamReadOptions readOptions = StreamReadOptions.empty().count(options.getBatchSize()); + if (!options.getPollTimeout().isZero()) { + readOptions = readOptions.block(options.getPollTimeout()); + } + + this.readOptions = readOptions; + this.template = new ReactiveRedisTemplate<>(connectionFactory, serializationContext); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.stream.StreamReceiver#receive(org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset) + */ + @Override + public Flux> receive(StreamOffset streamOffset) { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("receive(%s)", streamOffset)); + } + + return Flux.defer(() -> { + + PollState pollState = PollState.standalone(streamOffset.getOffset()); + BiFunction>> readFunction = (key, readOffset) -> template.opsForStream() + .read(readOptions, StreamOffset.create(key, readOffset)); + + return Flux.create(sink -> new StreamSubscription(sink, streamOffset.getKey(), pollState, readFunction).arm()); + }); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.stream.StreamReceiver#receiveAutoAck(org.springframework.data.redis.connection.RedisStreamCommands.Consumer, org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset) + */ + @Override + public Flux> receiveAutoAck(Consumer consumer, StreamOffset streamOffset) { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("receiveAutoAck(%s, %s)", consumer, streamOffset)); + } + + return Flux.defer(() -> { + + PollState pollState = PollState.consumer(consumer, streamOffset.getOffset()); + BiFunction>> readFunction = (key, readOffset) -> template.opsForStream() + .read(consumer, readOptions, StreamOffset.create(key, readOffset)); + + return Flux.create(sink -> new StreamSubscription(sink, streamOffset.getKey(), pollState, readFunction).arm()); + }); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.stream.StreamReceiver#receive(org.springframework.data.redis.connection.RedisStreamCommands.Consumer, org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset) + */ + @Override + public Flux> receive(Consumer consumer, StreamOffset streamOffset) { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("receive(%s, %s)", consumer, streamOffset)); + } + + StreamReadOptions noack = readOptions.noack(); + + return Flux.defer(() -> { + + PollState pollState = PollState.consumer(consumer, streamOffset.getOffset()); + BiFunction>> readFunction = (key, readOffset) -> template.opsForStream() + .read(consumer, noack, StreamOffset.create(key, readOffset)); + + return Flux.create(sink -> new StreamSubscription(sink, streamOffset.getKey(), pollState, readFunction).arm()); + }); + } + + /** + * A stateful Redis Stream subscription. + */ + @RequiredArgsConstructor + class StreamSubscription { + + private final Queue> overflow = Queues.> small().get(); + + private final FluxSink> sink; + private final K key; + private final PollState pollState; + private final BiFunction>> readFunction; + + /** + * Arm the subscription so {@link Subscription#request(long) demand} activates polling. + * + * @return + */ + void arm() { + + sink.onRequest(toAdd -> { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] onRequest(%d)", key, toAdd)); + } + + if (pollState.isSubscriptionActive()) { + + long r, u; + for (;;) { + r = pollState.getRequested(); + if (r == Long.MAX_VALUE) { + scheduleIfRequired(); + return; + } + u = Operators.addCap(r, toAdd); + if (pollState.setRequested(r, u)) { + if (u > 0) { + scheduleIfRequired(); + } + return; + } + } + } else { + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] onRequest(%d): Dropping, subscription canceled", key, toAdd)); + } + } + }); + + sink.onCancel(pollState::cancel); + } + + private void scheduleIfRequired() { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] scheduleIfRequired()", key)); + } + if (pollState.isScheduled()) { + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] scheduleIfRequired(): Already scheduled", key)); + } + return; + } + + if (!pollState.isSubscriptionActive()) { + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] scheduleIfRequired(): Subscription cancelled", key)); + } + return; + } + + if (pollState.getRequested() > 0 && !overflow.isEmpty()) { + if (logger.isDebugEnabled()) { + logger.info(String.format("[stream: %s] scheduleIfRequired(): Requested: %d Emit from buffer", key, + pollState.getRequested())); + } + emitBuffer(); + } + + if (pollState.getRequested() == 0) { + + if (logger.isDebugEnabled()) { + logger.debug(String + .format("[stream: %s] scheduleIfRequired(): Subscriber has no demand. Suspending subscription.", key)); + } + return; + } + + if (pollState.getRequested() <= 0) { + return; + } + + if (pollState.activateSchedule()) { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] scheduleIfRequired(): Activating subscription", key)); + } + + ReadOffset readOffset = pollState.getCurrentReadOffset(); + + if (logger.isDebugEnabled()) { + logger.debug( + String.format("[stream: %s] scheduleIfRequired(): Activating subscription, offset %s", key, readOffset)); + } + + Flux> poll = readFunction.apply(key, readOffset); + + poll.subscribe(getSubscriber()); + } + } + + private CoreSubscriber> getSubscriber() { + + return new CoreSubscriber>() { + + @Override + public void onSubscribe(Subscription s) { + s.request(Long.MAX_VALUE); + } + + @Override + public void onNext(StreamMessage message) { + onStreamMessage(message); + } + + @Override + public void onError(Throwable t) { + onStreamError(t); + } + + @Override + public void onComplete() { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] onComplete()", key)); + } + + pollState.scheduleCompleted(); + + scheduleIfRequired(); + } + + @Override + public Context currentContext() { + return sink.currentContext(); + } + }; + } + + private void onStreamMessage(StreamMessage message) { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] onStreamMessage(%s)", key, message)); + } + + pollState.updateReadOffset(message.getId()); + + long requested = pollState.getRequested(); + + if (requested > 0) { + + if (requested == Long.MAX_VALUE) { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] onStreamMessage(%s): Emitting item, fast-path", key, message)); + } + sink.next(message); + } else { + + if (pollState.decrementRequested()) { + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] onStreamMessage(%s): Emitting item, slow-path", key, message)); + } + sink.next(message); + } else { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] onStreamMessage(%s): Buffering overflow", key, message)); + } + overflow.add(message); + } + } + + } else { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] onStreamMessage(%s): Buffering overflow", key, message)); + } + overflow.offer(message); + } + } + + private void onStreamError(Throwable t) { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] onStreamError(%s)", key, t)); + } + + pollState.cancel(); + sink.error(t); + } + + private void emitBuffer() { + + while (!overflow.isEmpty()) { + + long demand = pollState.getRequested(); + + if (demand <= 0) { + break; + } + + if (demand == Long.MAX_VALUE) { + + StreamMessage message = overflow.poll(); + + if (message == null) { + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] emitBuffer(): emission missed", key)); + } + break; + } + + if (logger.isDebugEnabled()) { + logger.debug( + String.format("[stream: %s] emitBuffer(%s): Emitting item from buffer, fast-path", key, message)); + } + + sink.next(message); + + } else if (pollState.setRequested(demand, demand - 1)) { + + StreamMessage message = overflow.poll(); + + if (message == null) { + + if (logger.isDebugEnabled()) { + logger.debug(String.format("[stream: %s] emitBuffer(): emission missed", key)); + } + pollState.incrementRequested(); + break; + } + + if (logger.isDebugEnabled()) { + logger.debug( + String.format("[stream: %s] emitBuffer(%s): Emitting item from buffer, slow-path", key, message)); + } + + sink.next(message); + } + } + } + } + + /** + * Object representing the current polling state for a particular stream subscription. + */ + static class PollState { + + private final AtomicLong requestsPending = new AtomicLong(); + private final AtomicBoolean active = new AtomicBoolean(true); + private final AtomicBoolean scheduled = new AtomicBoolean(false); + + private final ReadOffsetStrategy readOffsetStrategy; + private final AtomicReference currentOffset; + private final Optional consumer; + + private PollState(Optional consumer, ReadOffsetStrategy readOffsetStrategy, ReadOffset currentOffset) { + + this.readOffsetStrategy = readOffsetStrategy; + this.currentOffset = new AtomicReference<>(currentOffset); + this.consumer = consumer; + } + + /** + * Create a new state object for standalone-read. + * + * @param offset + * @return + */ + static PollState standalone(ReadOffset offset) { + + ReadOffsetStrategy strategy = ReadOffsetStrategy.getStrategy(offset); + return new PollState(Optional.empty(), strategy, strategy.getFirst(offset, Optional.empty())); + } + + /** + * Create a new state object for consumergroup-read. + * + * @param consumer + * @param offset + * @return + */ + static PollState consumer(Consumer consumer, ReadOffset offset) { + + ReadOffsetStrategy strategy = ReadOffsetStrategy.getStrategy(offset); + Optional optionalConsumer = Optional.of(consumer); + return new PollState(optionalConsumer, strategy, strategy.getFirst(offset, optionalConsumer)); + } + + /** + * @return {@literal true} if the subscription is active. + */ + public boolean isSubscriptionActive() { + return active.get(); + } + + /** + * Cancel the subscription. + */ + public void cancel() { + active.set(false); + } + + /** + * Decrement request count to indicate that an element was emitted. + * + * @return + */ + boolean decrementRequested() { + + long demand = requestsPending.get(); + + if (demand > 0) { + return requestsPending.compareAndSet(demand, demand - 1); + } + + return false; + } + + /** + * Increment request count. + */ + void incrementRequested() { + requestsPending.incrementAndGet(); + } + + /** + * @return the number of requested items. + */ + public long getRequested() { + return requestsPending.get(); + } + + /** + * Update demand. + * + * @param expect + * @param update + * @return + */ + boolean setRequested(long expect, long update) { + return requestsPending.compareAndSet(expect, update); + } + + /** + * Activate the schedule and return the synchronization state. + * + * @return {@literal true} if the schedule was activated by this call or {@literal false} if a different process + * activated the schedule. + */ + boolean activateSchedule() { + return scheduled.compareAndSet(false, true); + } + + /** + * @return {@literal true} if the schedule is activated. + */ + public boolean isScheduled() { + return scheduled.get(); + } + + /** + * Deactivate the schedule. + */ + void scheduleCompleted() { + scheduled.compareAndSet(true, false); + } + + /** + * Advance the {@link ReadOffset}. + */ + void updateReadOffset(String messageId) { + + ReadOffset next = readOffsetStrategy.getNext(getCurrentReadOffset(), consumer, messageId); + this.currentOffset.set(next); + } + + ReadOffset getCurrentReadOffset() { + return currentOffset.get(); + } + } +} diff --git a/src/main/java/org/springframework/data/redis/stream/ReadOffsetStrategy.java b/src/main/java/org/springframework/data/redis/stream/ReadOffsetStrategy.java new file mode 100644 index 000000000..052d78eb9 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/stream/ReadOffsetStrategy.java @@ -0,0 +1,113 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import java.util.Optional; + +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; + +/** + * Strategy to determine the first and subsequent {@link ReadOffset}. + * + * @author Mark Paluch + * @since 2.2 + */ +enum ReadOffsetStrategy { + + /** + * Use the last seen message Id. + */ + NextMessage { + @Override + public ReadOffset getFirst(ReadOffset readOffset, Optional consumer) { + return readOffset; + } + + @Override + public ReadOffset getNext(ReadOffset readOffset, Optional consumer, String lastConsumedMessageId) { + return ReadOffset.from(lastConsumedMessageId); + } + }, + + /** + * Last consumed strategy. + */ + LastConsumed { + @Override + public ReadOffset getFirst(ReadOffset readOffset, Optional consumer) { + return consumer.map(it -> ReadOffset.lastConsumed()).orElseGet(ReadOffset::latest); + } + + @Override + public ReadOffset getNext(ReadOffset readOffset, Optional consumer, String lastConsumedMessageId) { + return consumer.map(it -> ReadOffset.lastConsumed()).orElseGet(() -> ReadOffset.from(lastConsumedMessageId)); + } + }, + + /** + * Use always the latest stream message. + */ + Latest { + @Override + public ReadOffset getFirst(ReadOffset readOffset, Optional consumer) { + return ReadOffset.latest(); + } + + @Override + public ReadOffset getNext(ReadOffset readOffset, Optional consumer, String lastConsumedMessageId) { + return ReadOffset.latest(); + } + }; + + /** + * Return a {@link ReadOffsetStrategy} given the initial {@link ReadOffset}. + * + * @param offset must not be {@literal null}. + * @return the {@link ReadOffsetStrategy}. + */ + static ReadOffsetStrategy getStrategy(ReadOffset offset) { + + if (ReadOffset.latest().equals(offset)) { + return Latest; + } + + if (ReadOffset.lastConsumed().equals(offset)) { + return LastConsumed; + } + + return NextMessage; + } + + /** + * Determine the first {@link ReadOffset}. + * + * @param readOffset + * @param consumer + * @return + */ + public abstract ReadOffset getFirst(ReadOffset readOffset, Optional consumer); + + /** + * Determine the next {@link ReadOffset} given {@code lastConsumedMessageId}. + * + * @param readOffset + * @param consumer + * @param lastConsumedMessageId + * @return + */ + public abstract ReadOffset getNext(ReadOffset readOffset, Optional consumer, String lastConsumedMessageId); +} diff --git a/src/main/java/org/springframework/data/redis/stream/StreamListener.java b/src/main/java/org/springframework/data/redis/stream/StreamListener.java new file mode 100644 index 000000000..2808e9bd3 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/stream/StreamListener.java @@ -0,0 +1,37 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; + +/** + * Listener interface to receive delivery of {@link StreamMessage messages}. + * + * @author Mark Paluch + * @param Stream key and Stream field type. + * @param Stream value type. + * @since 2.2 + */ +@FunctionalInterface +public interface StreamListener { + + /** + * Callback invoked on receiving a {@link StreamMessage}. + * + * @param message never {@literal null}. + */ + void onMessage(StreamMessage message); +} diff --git a/src/main/java/org/springframework/data/redis/stream/StreamMessageListenerContainer.java b/src/main/java/org/springframework/data/redis/stream/StreamMessageListenerContainer.java new file mode 100644 index 000000000..cefda140a --- /dev/null +++ b/src/main/java/org/springframework/data/redis/stream/StreamMessageListenerContainer.java @@ -0,0 +1,650 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import lombok.AccessLevel; +import lombok.RequiredArgsConstructor; + +import java.time.Duration; +import java.util.concurrent.Executor; +import java.util.function.Predicate; + +import org.springframework.context.SmartLifecycle; +import org.springframework.core.task.SimpleAsyncTaskExecutor; +import org.springframework.data.redis.connection.RedisConnectionFactory; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.serializer.RedisSerializer; +import org.springframework.data.redis.serializer.StringRedisSerializer; +import org.springframework.data.redis.stream.DefaultStreamMessageListenerContainer.LoggingErrorHandler; +import org.springframework.lang.Nullable; +import org.springframework.util.Assert; +import org.springframework.util.ErrorHandler; + +/** + * Abstraction used by the framework representing a message listener container. Not meant to be + * implemented externally. + *

+ * Once created, a {@link StreamMessageListenerContainer} can subscribe to a Redis Stream and consume incoming + * {@link StreamMessage messages}. {@link StreamMessageListenerContainer} allows multiple stream read requests and + * returns a {@link Subscription} handle per read request. Cancelling the {@link Subscription} terminates eventually + * background polling. Messages are converted using {@link RedisSerializer key and value serializers} to support various + * serialization strategies.
+ * {@link StreamMessageListenerContainer} supports multiple modes of stream consumption: + *

    + *
  • Standalone
    • + *
  • Using a {@link Consumer} with external + * {@link org.springframework.data.redis.core.StreamOperations#acknowledge(Object, String, String...)} acknowledge}
    • + *
  • Using a {@link Consumer} with auto-acknowledge
    • + *
+ * Reading from a stream requires polling and a strategy to advance stream offsets. Depending on the initial + * {@link ReadOffset}, {@link StreamMessageListenerContainer} applies an individual strategy to obtain the next + * {@link ReadOffset}:
+ * Standalone + *
    + *
  • {@link ReadOffset#from(String)} Offset using a particular message Id: Start with the given offset and use the + * last seen {@link StreamMessage#getId() message Id}.
    • + *
  • {@link ReadOffset#lastConsumed()} Last consumed: Start with the latest offset ({@code $}) and use the last seen + * {@link StreamMessage#getId() message Id}.
    • + *
  • {@link ReadOffset#latest()} Last consumed: Start with the latest offset ({@code $}) and use latest offset + * ({@code $}) for subsequent reads.
    • + *
+ *
+ * Using {@link Consumer} + *
    + *
  • {@link ReadOffset#from(String)} Offset using a particular message Id: Start with the given offset and use the + * last seen {@link StreamMessage#getId() message Id}.
    • + *
  • {@link ReadOffset#lastConsumed()} Last consumed: Start with the last consumed message by the consumer ({@code >}) + * and use the last consumed message by the consumer ({@code >}) for subsequent reads.
    • + *
  • {@link ReadOffset#latest()} Last consumed: Start with the latest offset ({@code $}) and use latest offset + * ({@code $}) for subsequent reads.
    • + *
+ * Note: Using {@link ReadOffset#latest()} bears the chance of dropped messages as messages can arrive in the + * time during polling is suspended. Use messagedId's as offset or {@link ReadOffset#lastConsumed()} to minimize the + * chance of message loss. + *

+ * {@link StreamMessageListenerContainer} requires a {@link Executor} to fork long-running polling tasks on a different + * {@link Thread}. This thread is used as event loop to poll for stream messages and invoke the + * {@link StreamListener#onMessage(StreamMessage) listener callback}. + *

+ * {@link StreamMessageListenerContainer} tasks propagate errors during stream reads and + * {@link StreamListener#onMessage(StreamMessage) listener notification} to a configurable {@link ErrorHandler}. Errors + * stop a {@link Subscription} by default. Configuring a {@link Predicate} for a {@link StreamReadRequest} allows + * conditional subscription cancelling or continuing on all errors. + *

+ * See the following example code how to use {@link StreamMessageListenerContainer}: + * + * 

+ * RedisConnectionFactory factory = …;
+ *
+ * StreamMessageListenerContainer container = StreamMessageListenerContainer.create(factory);
+ * Subscription subscription = container.receive(StreamOffset.create("my-stream", ReadOffset.from("0-0")), message -> …);
+ *
+ * container.start();
+ *
+ * // later
+ * container.stop();
+ *
+ * + * @author Mark Paluch + * @param Stream key and Stream field type. + * @param Stream value type. + * @since 2.2 + * @see StreamMessageListenerContainerOptions#builder() + * @see StreamListener + * @see StreamReadRequest + * @see ConsumerStreamReadRequest + * @see StreamMessageListenerContainerOptionsBuilder#executor(Executor) + * @see ErrorHandler + * @see org.springframework.data.redis.core.StreamOperations + * @see RedisConnectionFactory + * @see StreamReceiver + */ +public interface StreamMessageListenerContainer extends SmartLifecycle { + + /** + * Create a new {@link StreamMessageListenerContainer} using {@link StringRedisSerializer string serializers} given + * {@link RedisConnectionFactory}. + * + * @param connectionFactory must not be {@literal null}. + * @return the new {@link StreamMessageListenerContainer}. + */ + static StreamMessageListenerContainer create(RedisConnectionFactory connectionFactory) { + + Assert.notNull(connectionFactory, "RedisConnectionFactory must not be null!"); + + return create(connectionFactory, + StreamMessageListenerContainerOptions.builder().serializer(StringRedisSerializer.UTF_8).build()); + } + + /** + * Create a new {@link StreamMessageListenerContainer} given {@link RedisConnectionFactory} and + * {@link StreamMessageListenerContainerOptions}. + * + * @param connectionFactory must not be {@literal null}. + * @param options must not be {@literal null}. + * @return the new {@link StreamMessageListenerContainer}. + */ + static StreamMessageListenerContainer create(RedisConnectionFactory connectionFactory, + StreamMessageListenerContainerOptions options) { + + Assert.notNull(connectionFactory, "RedisConnectionFactory must not be null!"); + Assert.notNull(options, "StreamMessageListenerContainerOptions must not be null!"); + + return new DefaultStreamMessageListenerContainer<>(connectionFactory, options); + } + + /** + * Register a new subscription for a Redis Stream. If the {@link StreamMessageListenerContainer#isRunning() is already + * running} the {@link Subscription} will be added and run immediately, otherwise it'll be scheduled and started once + * the container is actually {@link StreamMessageListenerContainer#start() started}. + *

+ * Errors during {@link org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage} retrieval lead to + * {@link Subscription#cancel() cancellation} of the underlying task. + *

+ * On {@link StreamMessageListenerContainer#stop()} all {@link Subscription subscriptions} are cancelled prior to + * shutting down the container itself. + * + * @param streamOffset the stream along its offset. + * @param listener must not be {@literal null}. + * @return the subscription handle. + * @see StreamOffset#create(Object, ReadOffset) + */ + default Subscription receive(StreamOffset streamOffset, StreamListener listener) { + return register(StreamReadRequest.builder(streamOffset).build(), listener); + } + + /** + * Register a new subscription for a Redis Stream. If the {@link StreamMessageListenerContainer#isRunning() is already + * running} the {@link Subscription} will be added and run immediately, otherwise it'll be scheduled and started once + * the container is actually {@link StreamMessageListenerContainer#start() started}. + *

+ * Every message must be acknowledged using + * {@link org.springframework.data.redis.core.StreamOperations#acknowledge(Object, String, String...)} after + * processing. + *

+ * Errors during {@link org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage} retrieval lead to + * {@link Subscription#cancel() cancellation} of the underlying task. + *

+ * On {@link StreamMessageListenerContainer#stop()} all {@link Subscription subscriptions} are cancelled prior to + * shutting down the container itself. + * + * @param consumer consumer group, must not be {@literal null}. + * @param streamOffset the stream along its offset. + * @param listener must not be {@literal null}. + * @return the subscription handle. + * @see StreamOffset#create(Object, ReadOffset) + * @see ReadOffset#lastConsumed() + */ + default Subscription receive(Consumer consumer, StreamOffset streamOffset, StreamListener listener) { + return register(StreamReadRequest.builder(streamOffset).consumer(consumer).build(), listener); + } + + /** + * Register a new subscription for a Redis Stream. If the {@link StreamMessageListenerContainer#isRunning() is already + * running} the {@link Subscription} will be added and run immediately, otherwise it'll be scheduled and started once + * the container is actually {@link StreamMessageListenerContainer#start() started}. + *

+ * Every message is acknowledged when received. + *

+ * Errors during {@link org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage} retrieval lead to + * {@link Subscription#cancel() cancellation} of the underlying task. + *

+ * On {@link StreamMessageListenerContainer#stop()} all {@link Subscription subscriptions} are cancelled prior to + * shutting down the container itself. + * + * @param consumer consumer group, must not be {@literal null}. + * @param streamOffset the stream along its offset. + * @param listener must not be {@literal null}. + * @return the subscription handle. + * @see StreamOffset#create(Object, ReadOffset) + * @see ReadOffset#lastConsumed() + */ + default Subscription receiveAutoAck(Consumer consumer, StreamOffset streamOffset, StreamListener listener) { + return register(StreamReadRequest.builder(streamOffset).consumer(consumer).autoAck(true).build(), listener); + } + + /** + * Register a new subscription for a Redis Stream. If the {@link StreamMessageListenerContainer#isRunning() is already + * running} the {@link Subscription} will be added and run immediately, otherwise it'll be scheduled and started once + * the container is actually {@link StreamMessageListenerContainer#start() started}. + *

+ * Errors during {@link org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage} are tested + * against test {@link StreamReadRequest#getCancelSubscriptionOnError() cancellation predicate} whether to cancel the + * underlying task. + *

+ * On {@link StreamMessageListenerContainer#stop()} all {@link Subscription subscriptions} are cancelled prior to + * shutting down the container itself. + *

+ * Errors during {@link org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage} retrieval are + * delegated to the given {@link StreamReadRequest#getErrorHandler()}. + * + * @param streamRequest must not be {@literal null}. + * @param listener must not be {@literal null}. + * @return the subscription handle. + * @see StreamReadRequest + * @see ConsumerStreamReadRequest + */ + Subscription register(StreamReadRequest streamRequest, StreamListener listener); + + /** + * Unregister a given {@link Subscription} from the container. This prevents the {@link Subscription} to be restarted + * in a potential {@link SmartLifecycle#stop() stop}/{@link SmartLifecycle#start() start} scenario.
+ * An {@link Subscription#isActive() active} {@link Subscription subcription} is {@link Subscription#cancel() + * cancelled} prior to removal. + * + * @param subscription must not be {@literal null}. + */ + void remove(Subscription subscription); + + /** + * Request to read a Redis Stream. + * + * @param Stream key and Stream field type. + * @see StreamReadRequestBuilder + */ + class StreamReadRequest { + + private final StreamOffset streamOffset; + private final @Nullable ErrorHandler errorHandler; + private final Predicate cancelSubscriptionOnError; + + private StreamReadRequest(StreamOffset streamOffset, @Nullable ErrorHandler errorHandler, + Predicate cancelSubscriptionOnError) { + + this.streamOffset = streamOffset; + this.errorHandler = errorHandler; + this.cancelSubscriptionOnError = cancelSubscriptionOnError; + } + + /** + * @return a new builder for {@link StreamReadRequest}. + */ + static StreamReadRequestBuilder builder(StreamOffset offset) { + return new StreamReadRequestBuilder<>(offset); + } + + public StreamOffset getStreamOffset() { + return streamOffset; + } + + @Nullable + public ErrorHandler getErrorHandler() { + return errorHandler; + } + + public Predicate getCancelSubscriptionOnError() { + return cancelSubscriptionOnError; + } + } + + /** + * Request to read a Redis Stream with a {@link Consumer}. + * + * @param Stream key and Stream field type. + * @see StreamReadRequestBuilder + */ + class ConsumerStreamReadRequest extends StreamReadRequest { + + private final Consumer consumer; + private final boolean autoAck; + + private ConsumerStreamReadRequest(StreamOffset streamOffset, @Nullable ErrorHandler errorHandler, + Predicate cancelSubscriptionOnError, Consumer consumer, boolean autoAck) { + super(streamOffset, errorHandler, cancelSubscriptionOnError); + this.consumer = consumer; + this.autoAck = autoAck; + } + + public Consumer getConsumer() { + return consumer; + } + + public boolean isAutoAck() { + return autoAck; + } + } + + /** + * Builder to build a {@link StreamReadRequest}. + * + * @param Stream key and Stream field type. + */ + class StreamReadRequestBuilder { + + final StreamOffset streamOffset; + @Nullable ErrorHandler errorHandler; + Predicate cancelSubscriptionOnError = t -> true; + + StreamReadRequestBuilder(StreamOffset streamOffset) { + this.streamOffset = streamOffset; + } + + StreamReadRequestBuilder(StreamReadRequestBuilder other) { + + this.streamOffset = other.streamOffset; + this.errorHandler = other.errorHandler; + this.cancelSubscriptionOnError = other.cancelSubscriptionOnError; + } + + /** + * Configure a {@link ErrorHandler} to be notified on {@link Throwable errors}. + * + * @param errorHandler must not be null. + * @return {@code this} {@link StreamReadRequestBuilder}. + */ + public StreamReadRequestBuilder errorHandler(ErrorHandler errorHandler) { + + this.errorHandler = errorHandler; + return this; + } + + /** + * Configure a cancellation {@link Predicate} to be notified on {@link Throwable errors}. The outcome of the + * {@link Predicate} decides whether to cancel the subscription by returning {@literal true}. + * + * @param cancelSubscriptionOnError must not be null. + * @return {@code this} {@link StreamReadRequestBuilder}. + */ + public StreamReadRequestBuilder cancelOnError(Predicate cancelSubscriptionOnError) { + + this.cancelSubscriptionOnError = cancelSubscriptionOnError; + return this; + } + + /** + * Configure a {@link Consumer} to consume stream messages within a consumer group. + * + * @param consumer must not be null. + * @return a new {@link ConsumerStreamReadRequestBuilder}. + */ + public ConsumerStreamReadRequestBuilder consumer(Consumer consumer) { + return new ConsumerStreamReadRequestBuilder<>(this).consumer(consumer); + } + + /** + * Build a new instance of {@link StreamReadRequest}. + * + * @return a new instance of {@link StreamReadRequest}. + */ + public StreamReadRequest build() { + return new StreamReadRequest<>(streamOffset, errorHandler, cancelSubscriptionOnError); + } + } + + /** + * Builder to build a {@link ConsumerStreamReadRequest}. + * + * @param Stream key and Stream field type. + */ + class ConsumerStreamReadRequestBuilder extends StreamReadRequestBuilder { + + private Consumer consumer; + private boolean autoAck = true; + + ConsumerStreamReadRequestBuilder(StreamReadRequestBuilder other) { + super(other); + } + + /** + * Configure a {@link ErrorHandler} to be notified on {@link Throwable errors}. + * + * @param errorHandler must not be null. + * @return {@code this} {@link ConsumerStreamReadRequestBuilder}. + */ + public ConsumerStreamReadRequestBuilder errorHandler(ErrorHandler errorHandler) { + + super.errorHandler(errorHandler); + return this; + } + + /** + * Configure a cancellation {@link Predicate} to be notified on {@link Throwable errors}. The outcome of the + * {@link Predicate} decides whether to cancel the subscription by returning {@literal true}. + * + * @param cancelSubscriptionOnError must not be null. + * @return {@code this} {@link ConsumerStreamReadRequestBuilder}. + */ + public ConsumerStreamReadRequestBuilder cancelOnError(Predicate cancelSubscriptionOnError) { + + super.cancelOnError(cancelSubscriptionOnError); + return this; + } + + /** + * Configure a {@link Consumer} to consume stream messages within a consumer group. + * + * @param consumer must not be null. + * @return {@code this} {@link ConsumerStreamReadRequestBuilder}. + */ + public ConsumerStreamReadRequestBuilder consumer(Consumer consumer) { + + this.consumer = consumer; + return this; + } + + /** + * Configure auto-acknowledgement for stream message consumption. + * + * @param autoAck {@literal true} (default) to auto-acknowledge received messages or {@literal false} for external + * acknowledgement. + * @return {@code this} {@link ConsumerStreamReadRequestBuilder}. + */ + public ConsumerStreamReadRequestBuilder autoAck(boolean autoAck) { + + this.autoAck = autoAck; + return this; + } + + /** + * Build a new instance of {@link ConsumerStreamReadRequest}. + * + * @return a new instance of {@link ConsumerStreamReadRequest}. + */ + public ConsumerStreamReadRequest build() { + return new ConsumerStreamReadRequest<>(streamOffset, errorHandler, cancelSubscriptionOnError, consumer, autoAck); + } + } + + /** + * Options for {@link StreamMessageListenerContainer}. + * + * @param Stream key and Stream field type. + * @param Stream value type. + * @see StreamMessageListenerContainerOptionsBuilder + */ + @RequiredArgsConstructor(access = AccessLevel.PRIVATE) + class StreamMessageListenerContainerOptions { + + private final Duration pollTimeout; + private final int batchSize; + private final RedisSerializer keySerializer; + private final RedisSerializer bodySerializer; + private final ErrorHandler errorHandler; + private final Executor executor; + + /** + * @return a new builder for {@link StreamMessageListenerContainerOptions}. + */ + static StreamMessageListenerContainerOptionsBuilder builder() { + return new StreamMessageListenerContainerOptionsBuilder<>().serializer(StringRedisSerializer.UTF_8); + } + + /** + * Timeout for blocking polling using the {@code BLOCK} option during reads. + * + * @return + */ + public Duration getPollTimeout() { + return pollTimeout; + } + + /** + * Batch size polling using the {@code COUNT} option during reads. + * + * @return + */ + public int getBatchSize() { + return batchSize; + } + + public RedisSerializer getKeySerializer() { + return keySerializer; + } + + public RedisSerializer getBodySerializer() { + return bodySerializer; + } + + /** + * @return the default {@link ErrorHandler}. + */ + public ErrorHandler getErrorHandler() { + return errorHandler; + } + + /** + * @return the {@link Executor} to run stream polling {@link Task}s. Defaults to {@link SimpleAsyncTaskExecutor}. + */ + public Executor getExecutor() { + return executor; + } + } + + /** + * Builder for {@link StreamMessageListenerContainerOptions}. + * + * @param Stream key and Stream field type + * @param Stream value type + */ + class StreamMessageListenerContainerOptionsBuilder { + + private Duration pollTimeout = Duration.ofSeconds(2); + private int batchSize = 1; + private RedisSerializer keySerializer; + private RedisSerializer bodySerializer; + private ErrorHandler errorHandler = LoggingErrorHandler.INSTANCE; + private Executor executor = new SimpleAsyncTaskExecutor(); + + private StreamMessageListenerContainerOptionsBuilder() {} + + /** + * Configure a poll timeout for the {@code BLOCK} option during reading. + * + * @param pollTimeout must not be {@literal null} or negative. + * @return {@code this} {@link StreamMessageListenerContainerOptionsBuilder}. + */ + public StreamMessageListenerContainerOptionsBuilder pollTimeout(Duration pollTimeout) { + + Assert.notNull(pollTimeout, "Poll timeout must not be null!"); + Assert.isTrue(!pollTimeout.isNegative(), "Poll timeout must not be negative!"); + + this.pollTimeout = pollTimeout; + return this; + } + + /** + * Configure a batch size for the {@code COUNT} option during reading. + * + * @param messagesPerPoll must not be greater zero. + * @return {@code this} {@link StreamMessageListenerContainerOptionsBuilder}. + */ + public StreamMessageListenerContainerOptionsBuilder batchSize(int messagesPerPoll) { + + Assert.isTrue(messagesPerPoll > 0, "Batch size must be greater zero!"); + + this.batchSize = messagesPerPoll; + return this; + } + + /** + * Configure a {@link Executor} to run stream polling {@link Task}s. + * + * @param executor must not be null. + * @return {@code this} {@link StreamMessageListenerContainerOptionsBuilder}. + */ + public StreamMessageListenerContainerOptionsBuilder executor(Executor executor) { + + Assert.notNull(executor, "Executor must not be null!"); + + this.executor = executor; + return this; + } + + /** + * Configure a {@link ErrorHandler} to be notified on {@link Throwable errors}. + * + * @param errorHandler must not be null. + * @return {@code this} {@link StreamMessageListenerContainerOptionsBuilder}. + */ + public StreamMessageListenerContainerOptionsBuilder errorHandler(ErrorHandler errorHandler) { + + Assert.notNull(errorHandler, "ErrorHandler must not be null!"); + + this.errorHandler = errorHandler; + return this; + } + + /** + * Configure a key and value serializer. + * + * @param serializer must not be {@literal null}. + * @return {@code this} {@link StreamMessageListenerContainerOptionsBuilder}. + */ + public StreamMessageListenerContainerOptionsBuilder serializer(RedisSerializer serializer) { + + this.keySerializer = (RedisSerializer) serializer; + this.bodySerializer = (RedisSerializer) serializer; + return (StreamMessageListenerContainerOptionsBuilder) this; + } + + /** + * Configure a key serializer. + * + * @param serializer must not be {@literal null}. + * @return {@code this} {@link StreamMessageListenerContainerOptionsBuilder}. + */ + public StreamMessageListenerContainerOptionsBuilder keySerializer(RedisSerializer serializer) { + + this.keySerializer = (RedisSerializer) serializer; + return (StreamMessageListenerContainerOptionsBuilder) this; + } + + /** + * Configure a value serializer. + * + * @param serializer must not be {@literal null}. + * @return {@code this} {@link StreamMessageListenerContainerOptionsBuilder}. + */ + public StreamMessageListenerContainerOptionsBuilder bodySerializer(RedisSerializer serializer) { + + this.bodySerializer = (RedisSerializer) serializer; + return (StreamMessageListenerContainerOptionsBuilder) this; + } + + /** + * Build new {@link StreamMessageListenerContainerOptions}. + * + * @return new {@link StreamMessageListenerContainerOptions}. + */ + public StreamMessageListenerContainerOptions build() { + return new StreamMessageListenerContainerOptions<>(pollTimeout, batchSize, keySerializer, bodySerializer, + errorHandler, executor); + } + } +} diff --git a/src/main/java/org/springframework/data/redis/stream/StreamPollTask.java b/src/main/java/org/springframework/data/redis/stream/StreamPollTask.java new file mode 100644 index 000000000..c562bf6ad --- /dev/null +++ b/src/main/java/org/springframework/data/redis/stream/StreamPollTask.java @@ -0,0 +1,267 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import java.time.Duration; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.function.BiFunction; +import java.util.function.Predicate; + +import org.springframework.dao.DataAccessResourceFailureException; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.stream.StreamMessageListenerContainer.ConsumerStreamReadRequest; +import org.springframework.data.redis.stream.StreamMessageListenerContainer.StreamReadRequest; +import org.springframework.util.ErrorHandler; + +/** + * {@link Task} that invokes a {@link BiFunction read function} to poll on a Redis Stream. + * + * @author Mark Paluch + * @see 2.2 + */ +class StreamPollTask implements Task { + + private final StreamReadRequest request; + private final StreamListener listener; + private final ErrorHandler errorHandler; + private final Predicate cancelSubscriptionOnError; + private final BiFunction>> readFunction; + + private final PollState pollState; + private volatile boolean isInEventLoop = false; + + StreamPollTask(StreamReadRequest streamRequest, StreamListener listener, ErrorHandler errorHandler, + BiFunction>> readFunction) { + + this.request = streamRequest; + this.listener = listener; + this.errorHandler = Optional.ofNullable(streamRequest.getErrorHandler()).orElse(errorHandler); + this.cancelSubscriptionOnError = streamRequest.getCancelSubscriptionOnError(); + this.readFunction = readFunction; + this.pollState = createPollState(streamRequest); + } + + private static PollState createPollState(StreamReadRequest streamRequest) { + + StreamOffset streamOffset = streamRequest.getStreamOffset(); + + if (streamRequest instanceof ConsumerStreamReadRequest) { + return PollState.consumer(((ConsumerStreamReadRequest) streamRequest).getConsumer(), streamOffset.getOffset()); + } + + return PollState.standalone(streamOffset.getOffset()); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.stream.Cancelable#cancel() + */ + @Override + public void cancel() throws DataAccessResourceFailureException { + this.pollState.cancel(); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.stream.Task#getState() + */ + @Override + public State getState() { + return pollState.getState(); + } + + /* + * (non-Javadoc) + * @see org.springframework.data.redis.stream.Task#awaitStart(java.time.Duration) + */ + @Override + public boolean awaitStart(Duration timeout) throws InterruptedException { + return pollState.awaitStart(timeout.toNanos(), TimeUnit.NANOSECONDS); + } + + /* + * (non-Javadoc) + * @see org.springframework.scheduling.SchedulingAwareRunnable#isLongLived() + */ + @Override + public boolean isLongLived() { + return true; + } + + /* + * (non-Javadoc) + * @see java.lang.Runnable#run() + */ + @Override + public void run() { + + pollState.starting(); + pollState.running(); + + try { + + isInEventLoop = true; + doLoop(request.getStreamOffset().getKey()); + } finally { + isInEventLoop = false; + } + } + + private void doLoop(K key) { + + do { + + try { + + // allow interruption + Thread.sleep(0); + + List> read = readFunction.apply(key, pollState.getCurrentReadOffset()); + + for (StreamMessage message : read) { + + listener.onMessage(message); + pollState.updateReadOffset(message.getId()); + } + } catch (InterruptedException e) { + + pollState.cancel(); + + Thread.currentThread().interrupt(); + } catch (RuntimeException e) { + + errorHandler.handleError(e); + + if (cancelSubscriptionOnError.test(e)) { + cancel(); + } + } + } while (pollState.isSubscriptionActive()); + } + + @Override + public boolean isActive() { + return State.RUNNING.equals(getState()) || isInEventLoop; + } + + /** + * Object representing the current polling state for a particular stream subscription. + */ + static class PollState { + + private final ReadOffsetStrategy readOffsetStrategy; + private final Optional consumer; + private volatile ReadOffset currentOffset; + private volatile State state = State.CREATED; + private volatile CountDownLatch awaitStart = new CountDownLatch(1); + + private PollState(Optional consumer, ReadOffsetStrategy readOffsetStrategy, ReadOffset currentOffset) { + + this.readOffsetStrategy = readOffsetStrategy; + this.currentOffset = currentOffset; + this.consumer = consumer; + } + + /** + * Create a new state object for standalone-read. + * + * @param offset + * @return + */ + static PollState standalone(ReadOffset offset) { + + ReadOffsetStrategy strategy = ReadOffsetStrategy.getStrategy(offset); + return new PollState(Optional.empty(), strategy, strategy.getFirst(offset, Optional.empty())); + } + + /** + * Create a new state object for consumergroup-read. + * + * @param consumer + * @param offset + * @return + */ + static PollState consumer(Consumer consumer, ReadOffset offset) { + + ReadOffsetStrategy strategy = ReadOffsetStrategy.getStrategy(offset); + Optional optionalConsumer = Optional.of(consumer); + return new PollState(optionalConsumer, strategy, strategy.getFirst(offset, optionalConsumer)); + } + + boolean awaitStart(long timeout, TimeUnit unit) throws InterruptedException { + return awaitStart.await(timeout, unit); + } + + public State getState() { + return state; + } + + /** + * @return {@literal true} if the subscription is active. + */ + boolean isSubscriptionActive() { + return state == State.STARTING || state == State.RUNNING; + } + + /** + * Set the state to {@link org.springframework.data.redis.stream.Task.State#STARTING}. + */ + void starting() { + state = State.STARTING; + } + + /** + * Switch the state to {@link org.springframework.data.redis.stream.Task.State#RUNNING}. + */ + void running() { + + state = State.RUNNING; + + CountDownLatch awaitStart = this.awaitStart; + + if (awaitStart.getCount() == 1) { + awaitStart.countDown(); + } + } + + /** + * Set the state to {@link org.springframework.data.redis.stream.Task.State#CANCELLED} and re-arm the + * {@link #awaitStart(long, TimeUnit) await synchronizer}. + */ + void cancel() { + + awaitStart = new CountDownLatch(1); + state = State.CANCELLED; + } + + /** + * Advance the {@link ReadOffset}. + */ + void updateReadOffset(String messageId) { + currentOffset = readOffsetStrategy.getNext(getCurrentReadOffset(), consumer, messageId); + } + + ReadOffset getCurrentReadOffset() { + return currentOffset; + } + } +} diff --git a/src/main/java/org/springframework/data/redis/stream/StreamReceiver.java b/src/main/java/org/springframework/data/redis/stream/StreamReceiver.java new file mode 100644 index 000000000..89d04fec3 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/stream/StreamReceiver.java @@ -0,0 +1,320 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import reactor.core.publisher.Flux; + +import java.nio.ByteBuffer; +import java.time.Duration; + +import org.springframework.data.redis.connection.ReactiveRedisConnectionFactory; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.serializer.RedisSerializationContext.SerializationPair; +import org.springframework.data.redis.serializer.StringRedisSerializer; +import org.springframework.util.Assert; + +/** + * A receiver to consume Redis Streams using reactive infrastructure. + *

+ * Once created, a {@link StreamReceiver} can subscribe to a Redis Stream and consume incoming {@link StreamMessage + * messages}. Consider a {@link Flux} of {@link StreamMessage} infinite. Cancelling the + * {@link org.reactivestreams.Subscription} terminates eventually background polling. Messages are converted using + * {@link SerializationPair key and value serializers} to support various serialization strategies.
+ * {@link StreamReceiver} supports three modes of stream consumption: + *

    + *
  • Standalone
    • + *
  • Using a {@link Consumer} with external + * {@link org.springframework.data.redis.core.ReactiveStreamOperations#acknowledge(Object, String, String...) + * acknowledge}
    • + *
  • Using a {@link Consumer} with auto-acknowledge
    • + *
+ * Reading from a stream requires polling and a strategy to advance stream offsets. Depending on the initial + * {@link ReadOffset}, {@link StreamReceiver} applies an individual strategy to obtain the next {@link ReadOffset}: + *
+ * Standalone + *
    + *
  • {@link ReadOffset#from(String)} Offset using a particular message Id: Start with the given offset and use the + * last seen {@link StreamMessage#getId() message Id}.
    • + *
  • {@link ReadOffset#lastConsumed()} Last consumed: Start with the latest offset ({@code $}) and use the last seen + * {@link StreamMessage#getId() message Id}.
    • + *
  • {@link ReadOffset#latest()} Last consumed: Start with the latest offset ({@code $}) and use latest offset + * ({@code $}) for subsequent reads.
    • + *
+ *
+ * Using {@link Consumer} + *
    + *
  • {@link ReadOffset#from(String)} Offset using a particular message Id: Start with the given offset and use the + * last seen {@link StreamMessage#getId() message Id}.
    • + *
  • {@link ReadOffset#lastConsumed()} Last consumed: Start with the last consumed message by the consumer ({@code >}) + * and use the last consumed message by the consumer ({@code >}) for subsequent reads.
    • + *
  • {@link ReadOffset#latest()} Last consumed: Start with the latest offset ({@code $}) and use latest offset + * ({@code $}) for subsequent reads.
    • + *
+ * Note: Using {@link ReadOffset#latest()} bears the chance of dropped messages as messages can arrive in the + * time during polling is suspended. Use messagedId's as offset or {@link ReadOffset#lastConsumed()} to minimize the + * chance of message loss. + *

+ * See the following example code how to use {@link StreamReceiver}: + * + * 

+ * ReactiveRedisConnectionFactory factory = …;
+ *
+ * StreamReceiver receiver = StreamReceiver.create(factory);
+ * Flux> messages = receiver.receive(StreamOffset.create("my-stream", ReadOffset.from("0-0")));
+ *
+ * messageFlux.doOnNext(message -> …);
+ *
+ * + * @author Mark Paluch + * @param Stream key and Stream field type. + * @param Stream value type. + * @since 2.2 + * @see StreamReceiverOptions#builder() + * @see org.springframework.data.redis.core.ReactiveStreamOperations + * @see ReactiveRedisConnectionFactory + * @see StreamMessageListenerContainer + */ +public interface StreamReceiver { + + /** + * Create a new {@link StreamReceiver} using {@link StringRedisSerializer string serializers} given + * {@link ReactiveRedisConnectionFactory}. + * + * @param connectionFactory must not be {@literal null}. + * @return the new {@link StreamReceiver}. + */ + static StreamReceiver create(ReactiveRedisConnectionFactory connectionFactory) { + + Assert.notNull(connectionFactory, "ReactiveRedisConnectionFactory must not be null!"); + + SerializationPair serializationPair = SerializationPair.fromSerializer(StringRedisSerializer.UTF_8); + return create(connectionFactory, StreamReceiverOptions.builder().serializer(serializationPair).build()); + } + + /** + * Create a new {@link StreamReceiver} given {@link ReactiveRedisConnectionFactory} and {@link StreamReceiverOptions}. + * + * @param connectionFactory must not be {@literal null}. + * @param options must not be {@literal null}. + * @return the new {@link StreamReceiver}. + */ + static StreamReceiver create(ReactiveRedisConnectionFactory connectionFactory, + StreamReceiverOptions options) { + + Assert.notNull(connectionFactory, "ReactiveRedisConnectionFactory must not be null!"); + Assert.notNull(options, "StreamReceiverOptions must not be null!"); + + return new DefaultStreamReceiver<>(connectionFactory, options); + } + + /** + * Starts a Redis Stream consumer that consumes {@link StreamMessage messages} from the {@link StreamOffset stream}. + * Messages are consumed from Redis and delivered on the returned {@link Flux} when requests are made on the Flux. The + * receiver is closed when the returned {@link Flux} terminates. + *

+ * Every message must be acknowledged using + * {@link org.springframework.data.redis.connection.ReactiveStreamCommands#xAck(ByteBuffer, String, String...)} + * + * @param streamOffset the stream along its offset. + * @return Flux of inbound {@link StreamMessage}s. + * @see StreamOffset#create(Object, ReadOffset) + */ + Flux> receive(StreamOffset streamOffset); + + /** + * Starts a Redis Stream consumer that consumes {@link StreamMessage messages} from the {@link StreamOffset stream}. + * Messages are consumed from Redis and delivered on the returned {@link Flux} when requests are made on the Flux. The + * receiver is closed when the returned {@link Flux} terminates. + *

+ * Every message is acknowledged when received. + * + * @param consumer consumer group, must not be {@literal null}. + * @param streamOffset the stream along its offset. + * @return Flux of inbound {@link StreamMessage}s. + * @see StreamOffset#create(Object, ReadOffset) + * @see ReadOffset#lastConsumed() + */ + Flux> receiveAutoAck(Consumer consumer, StreamOffset streamOffset); + + /** + * Starts a Redis Stream consumer that consumes {@link StreamMessage messages} from the {@link StreamOffset stream}. + * Messages are consumed from Redis and delivered on the returned {@link Flux} when requests are made on the Flux. The + * receiver is closed when the returned {@link Flux} terminates. + *

+ * Every message must be acknowledged using + * {@link org.springframework.data.redis.core.ReactiveStreamOperations#acknowledge(Object, String, String...)} after + * processing. + * + * @param consumer consumer group, must not be {@literal null}. + * @param streamOffset the stream along its offset. + * @return Flux of inbound {@link StreamMessage}s. + * @see StreamOffset#create(Object, ReadOffset) + * @see ReadOffset#lastConsumed() + */ + Flux> receive(Consumer consumer, StreamOffset streamOffset); + + /** + * Options for {@link StreamReceiver}. + * + * @param Stream key and Stream field type. + * @param Stream value type. + * @see StreamReceiverOptionsBuilder + */ + class StreamReceiverOptions { + + private final Duration pollTimeout; + private final int batchSize; + private final SerializationPair keySerializer; + private final SerializationPair bodySerializer; + + private StreamReceiverOptions(Duration pollTimeout, int batchSize, SerializationPair keySerializer, + SerializationPair bodySerializer) { + this.pollTimeout = pollTimeout; + this.batchSize = batchSize; + this.keySerializer = keySerializer; + this.bodySerializer = bodySerializer; + } + + /** + * @return a new builder for {@link StreamReceiverOptions}. + */ + static StreamReceiverOptionsBuilder builder() { + + SerializationPair serializer = SerializationPair.fromSerializer(StringRedisSerializer.UTF_8); + return new StreamReceiverOptionsBuilder<>().serializer(serializer); + } + + /** + * Timeout for blocking polling using the {@code BLOCK} option during reads. + * + * @return + */ + public Duration getPollTimeout() { + return pollTimeout; + } + + /** + * Batch size polling using the {@code COUNT} option during reads. + * + * @return + */ + public int getBatchSize() { + return batchSize; + } + + public SerializationPair getKeySerializer() { + return keySerializer; + } + + public SerializationPair getBodySerializer() { + return bodySerializer; + } + } + + /** + * Builder for {@link StreamReceiverOptions}. + * + * @param Stream key and Stream field type. + * @param Stream value type. + */ + class StreamReceiverOptionsBuilder { + + private Duration pollTimeout = Duration.ofSeconds(2); + private int batchSize = 1; + private SerializationPair keySerializer; + private SerializationPair bodySerializer; + + private StreamReceiverOptionsBuilder() {} + + /** + * Configure a poll timeout for the {@code BLOCK} option during reading. + * + * @param pollTimeout must not be {@literal null} or negative. + * @return {@code this} {@link StreamReceiverOptionsBuilder}. + */ + public StreamReceiverOptionsBuilder pollTimeout(Duration pollTimeout) { + + Assert.notNull(pollTimeout, "Poll timeout must not be null!"); + Assert.isTrue(!pollTimeout.isNegative(), "Poll timeout must not be negative!"); + + this.pollTimeout = pollTimeout; + return this; + } + + /** + * Configure a batch size for the {@code COUNT} option during reading. + * + * @param messagesPerPoll must not be greater zero. + * @return {@code this} {@link StreamReceiverOptionsBuilder}. + */ + public StreamReceiverOptionsBuilder batchSize(int messagesPerPoll) { + + Assert.isTrue(messagesPerPoll > 0, "Batch size must be greater zero!"); + + this.batchSize = messagesPerPoll; + return this; + } + + /** + * Configure a key and value serializer. + * + * @param pair must not be {@literal null}. + * @return {@code this} {@link StreamReceiverOptionsBuilder}. + */ + public StreamReceiverOptionsBuilder serializer(SerializationPair pair) { + + this.keySerializer = (SerializationPair) pair; + this.bodySerializer = (SerializationPair) pair; + return (StreamReceiverOptionsBuilder) this; + } + + /** + * Configure a key serializer. + * + * @param pair must not be {@literal null}. + * @return {@code this} {@link StreamReceiverOptionsBuilder}. + */ + public StreamReceiverOptionsBuilder keySerializer(SerializationPair pair) { + + this.keySerializer = (SerializationPair) pair; + return (StreamReceiverOptionsBuilder) this; + } + + /** + * Configure a value serializer. + * + * @param pair must not be {@literal null}. + * @return {@code this} {@link StreamReceiverOptionsBuilder}. + */ + public StreamReceiverOptionsBuilder bodySerializer(SerializationPair pair) { + + this.bodySerializer = (SerializationPair) pair; + return (StreamReceiverOptionsBuilder) this; + } + + /** + * Build new {@link StreamReceiverOptions}. + * + * @return new {@link StreamReceiverOptions}. + */ + public StreamReceiverOptions build() { + return new StreamReceiverOptions<>(pollTimeout, batchSize, keySerializer, bodySerializer); + } + } +} diff --git a/src/main/java/org/springframework/data/redis/stream/Subscription.java b/src/main/java/org/springframework/data/redis/stream/Subscription.java new file mode 100644 index 000000000..1da7006e3 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/stream/Subscription.java @@ -0,0 +1,47 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import java.time.Duration; + +/** + * The {@link Subscription} is the link to the actual running {@link Task}. + *

+ * Due to the asynchronous nature of the {@link Task} execution a {@link Subscription} might not immediately become + * active. {@link #isActive()} provides an answer if the underlying {@link Task} is already running. + *

+ * + * @author Mark Paluch + * @since 2.2 + */ +public interface Subscription extends Cancelable { + + /** + * @return {@literal true} if the subscription is currently executed. + */ + boolean isActive(); + + /** + * Synchronous, blocking call returns once the {@link Subscription} becomes {@link #isActive() + * active} or {@link Duration timeout} exceeds. + * + * @param timeout must not be {@literal null}. + * @return {@code true} if the subscription was activated. {@code false} if the waiting time elapsed before task was + * activated. + * @throws InterruptedException if the current thread is interrupted while waiting. + */ + boolean await(Duration timeout) throws InterruptedException; +} diff --git a/src/main/java/org/springframework/data/redis/stream/Task.java b/src/main/java/org/springframework/data/redis/stream/Task.java new file mode 100644 index 000000000..61178d071 --- /dev/null +++ b/src/main/java/org/springframework/data/redis/stream/Task.java @@ -0,0 +1,63 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import java.time.Duration; + +import org.springframework.scheduling.SchedulingAwareRunnable; + +/** + * The actual {@link Task} to run within the {@link StreamMessageListenerContainer}. + * + * @author Mark Paluch + * @since 2.2 + */ +public interface Task extends SchedulingAwareRunnable, Cancelable { + + /** + * @return {@literal true} if the task is currently {@link State#RUNNING running}. + */ + default boolean isActive() { + return State.RUNNING.equals(getState()); + } + + /** + * Get the current lifecycle phase. + * + * @return never {@literal null}. + */ + State getState(); + + /** + * Synchronous, blocking call that awaits until this {@link Task} becomes active. Start awaiting is + * rearmed after {@link #cancel() cancelling} to support restart. + * + * @param timeout must not be {@literal null}. + * @return {@code true} if the task was started. {@code false} if the waiting time elapsed before task was started. + * @throws InterruptedException if the current thread is interrupted while waiting. + */ + boolean awaitStart(Duration timeout) throws InterruptedException; + + /** + * The {@link Task.State} defining the lifecycle phase the actual {@link Task}. + * + * @author Mark Paluch + * @since 2.2 + */ + enum State { + CREATED, STARTING, RUNNING, CANCELLED; + } +} diff --git a/src/test/java/org/springframework/data/redis/RedisTestProfileValueSource.java b/src/test/java/org/springframework/data/redis/RedisTestProfileValueSource.java index 630c46b99..f9c6143ac 100644 --- a/src/test/java/org/springframework/data/redis/RedisTestProfileValueSource.java +++ b/src/test/java/org/springframework/data/redis/RedisTestProfileValueSource.java @@ -40,6 +40,7 @@ public class RedisTestProfileValueSource implements ProfileValueSource { private static final String REDIS_28 = "2.8"; private static final String REDIS_30 = "3.0"; private static final String REDIS_32 = "3.2"; + private static final String REDIS_50 = "4.9"; private static final String REDIS_VERSION_KEY = "redisVersion"; private static RedisTestProfileValueSource INSTANCE; @@ -96,6 +97,10 @@ public class RedisTestProfileValueSource implements ProfileValueSource { return System.getProperty(key); } + if (redisVersion.compareTo(RedisVersionUtils.parseVersion(REDIS_50)) >= 0) { + return REDIS_50; + } + if (redisVersion.compareTo(RedisVersionUtils.parseVersion(REDIS_32)) >= 0) { return REDIS_32; } diff --git a/src/test/java/org/springframework/data/redis/connection/AbstractConnectionIntegrationTests.java b/src/test/java/org/springframework/data/redis/connection/AbstractConnectionIntegrationTests.java index 58125757e..aa89ce64a 100644 --- a/src/test/java/org/springframework/data/redis/connection/AbstractConnectionIntegrationTests.java +++ b/src/test/java/org/springframework/data/redis/connection/AbstractConnectionIntegrationTests.java @@ -64,6 +64,10 @@ import org.springframework.data.redis.RedisVersionUtils; import org.springframework.data.redis.TestCondition; import org.springframework.data.redis.connection.RedisGeoCommands.GeoLocation; import org.springframework.data.redis.connection.RedisListCommands.Position; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; import org.springframework.data.redis.connection.RedisStringCommands.BitOperation; import org.springframework.data.redis.connection.RedisStringCommands.SetOption; import org.springframework.data.redis.connection.RedisZSetCommands.Aggregate; @@ -1156,11 +1160,11 @@ public abstract class AbstractConnectionIntegrationTests { @IfProfileValue(name = "redisVersion", value = "4.0+") public void unlinkReturnsNrOfKeysRemoved() { - connection.set("unlink.this", "Can't track this!"); + actual.add(connection.set("unlink.this", "Can't track this!")); actual.add(connection.unlink("unlink.this", "unlink.that")); - verifyResults(Arrays.asList(new Object[] { 1L })); + verifyResults(Arrays.asList(new Object[] { true, 1L })); } @Test // DATAREDIS-693 @@ -2999,6 +3003,101 @@ public abstract class AbstractConnectionIntegrationTests { assertThat((List) results.get(1), contains(100L, -56L)); } + + @Test // DATAREDIS-864 + @IfProfileValue(name = "redisVersion", value = "5.0") + @WithRedisDriver({ RedisDriver.LETTUCE }) + public void xAddShouldCreateStream() { + + actual.add(connection.xAdd(KEY_1, Collections.singletonMap(KEY_2, VALUE_2))); + actual.add(connection.type(KEY_1)); + + List results = getResults(); + assertThat(results, hasSize(2)); + assertThat((String) results.get(0), containsString("-")); + assertThat(results.get(1), is(DataType.STREAM)); + } + + @Test // DATAREDIS-864 + @IfProfileValue(name = "redisVersion", value = "5.0") + @WithRedisDriver({ RedisDriver.LETTUCE }) + public void xReadShouldReadMessage() { + + actual.add(connection.xAdd(KEY_1, Collections.singletonMap(KEY_2, VALUE_2))); + actual.add(connection.xReadAsString(StreamOffset.create(KEY_1, ReadOffset.from("0")))); + + List results = getResults(); + + List> messages = (List) results.get(1); + + assertThat(messages.get(0).getStream(), is(KEY_1)); + assertThat(messages.get(0).getBody(), is(Collections.singletonMap(KEY_2, VALUE_2))); + } + + @Test // DATAREDIS-864 + @IfProfileValue(name = "redisVersion", value = "5.0") + @WithRedisDriver({ RedisDriver.LETTUCE }) + public void xReadGroupShouldReadMessage() { + + actual.add(connection.xAdd(KEY_1, Collections.singletonMap(KEY_2, VALUE_2))); + actual.add(connection.xGroupCreate(KEY_1, ReadOffset.from("0"), "my-group")); + actual.add(connection.xReadGroupAsString(Consumer.from("my-group", "my-consumer"), StreamOffset.create(KEY_1, ReadOffset.lastConsumed()))); + actual.add(connection.xReadGroupAsString(Consumer.from("my-group", "my-consumer"), StreamOffset.create(KEY_1, ReadOffset.lastConsumed()))); + + List results = getResults(); + + List> messages = (List) results.get(2); + + assertThat(messages.get(0).getStream(), is(KEY_1)); + assertThat(messages.get(0).getBody(), is(Collections.singletonMap(KEY_2, VALUE_2))); + + assertThat((List) results.get(3), is(empty())); + } + + @Test // DATAREDIS-864 + @IfProfileValue(name = "redisVersion", value = "5.0") + @WithRedisDriver({ RedisDriver.LETTUCE }) + public void xRangeShouldReportMessages() { + + actual.add(connection.xAdd(KEY_1, Collections.singletonMap(KEY_2, VALUE_2))); + actual.add(connection.xAdd(KEY_1, Collections.singletonMap(KEY_3, VALUE_3))); + actual.add(connection.xRange(KEY_1, org.springframework.data.domain.Range.unbounded())); + + List results = getResults(); + assertThat(results, hasSize(3)); + assertThat((String) results.get(0), containsString("-")); + + List> messages = (List) results.get(2); + + assertThat(messages.get(0).getStream(), is(KEY_1)); + assertThat(messages.get(0).getBody(), is(Collections.singletonMap(KEY_2, VALUE_2))); + + assertThat(messages.get(1).getStream(), is(KEY_1)); + assertThat(messages.get(1).getBody(), is(Collections.singletonMap(KEY_3, VALUE_3))); + } + + @Test // DATAREDIS-864 + @IfProfileValue(name = "redisVersion", value = "5.0") + @WithRedisDriver({ RedisDriver.LETTUCE }) + public void xRevRangeShouldReportMessages() { + + actual.add(connection.xAdd(KEY_1, Collections.singletonMap(KEY_2, VALUE_2))); + actual.add(connection.xAdd(KEY_1, Collections.singletonMap(KEY_3, VALUE_3))); + actual.add(connection.xRevRange(KEY_1, org.springframework.data.domain.Range.unbounded())); + + List results = getResults(); + assertThat(results, hasSize(3)); + assertThat((String) results.get(0), containsString("-")); + + List> messages = (List) results.get(2); + + assertThat(messages.get(0).getStream(), is(KEY_1)); + assertThat(messages.get(0).getBody(), is(Collections.singletonMap(KEY_3, VALUE_3))); + + assertThat(messages.get(1).getStream(), is(KEY_1)); + assertThat(messages.get(1).getBody(), is(Collections.singletonMap(KEY_2, VALUE_2))); + } + protected void verifyResults(List expected) { assertEquals(expected, getResults()); } diff --git a/src/test/java/org/springframework/data/redis/connection/jedis/JedisSentinelIntegrationTests.java b/src/test/java/org/springframework/data/redis/connection/jedis/JedisSentinelIntegrationTests.java index be5c14266..9f4696b07 100644 --- a/src/test/java/org/springframework/data/redis/connection/jedis/JedisSentinelIntegrationTests.java +++ b/src/test/java/org/springframework/data/redis/connection/jedis/JedisSentinelIntegrationTests.java @@ -29,6 +29,7 @@ import org.junit.ClassRule; import org.junit.Ignore; import org.junit.Rule; import org.junit.Test; +import org.junit.runner.RunWith; import org.springframework.dao.InvalidDataAccessApiUsageException; import org.springframework.data.redis.connection.AbstractConnectionIntegrationTests; import org.springframework.data.redis.connection.RedisSentinelConfiguration; @@ -37,7 +38,9 @@ import org.springframework.data.redis.connection.RedisServer; import org.springframework.data.redis.connection.ReturnType; import org.springframework.data.redis.test.util.MinimumRedisVersionRule; import org.springframework.data.redis.test.util.RedisSentinelRule; +import org.springframework.data.redis.test.util.RelaxedJUnit4ClassRunner; import org.springframework.test.annotation.IfProfileValue; +import org.springframework.test.context.ContextConfiguration; import org.springframework.test.util.ReflectionTestUtils; /** @@ -45,6 +48,8 @@ import org.springframework.test.util.ReflectionTestUtils; * @author Thomas Darimont * @author Mark Paluch */ +@RunWith(RelaxedJUnit4ClassRunner.class) +@ContextConfiguration("JedisConnectionIntegrationTests-context.xml") public class JedisSentinelIntegrationTests extends AbstractConnectionIntegrationTests { private static final String MASTER_NAME = "mymaster"; diff --git a/src/test/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveCommandsTestsBase.java b/src/test/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveCommandsTestsBase.java index 30e7c71c4..677b7edc8 100644 --- a/src/test/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveCommandsTestsBase.java +++ b/src/test/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveCommandsTestsBase.java @@ -83,7 +83,7 @@ public abstract class LettuceReactiveCommandsTestsBase { RedisClusterCommands nativeCommands; RedisClusterCommands nativeBinaryCommands; - @Parameterized.Parameters(name = "{2}") + @Parameterized.Parameters(name = "{3}") public static List parameters() { LettuceRedisClientProvider standalone = LettuceRedisClientProvider.local(); diff --git a/src/test/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveStreamCommandsTests.java b/src/test/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveStreamCommandsTests.java new file mode 100644 index 000000000..55ae96574 --- /dev/null +++ b/src/test/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveStreamCommandsTests.java @@ -0,0 +1,173 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.connection.lettuce; + +import static org.assertj.core.api.Assertions.*; +import static org.junit.Assume.*; + +import io.lettuce.core.XReadArgs; +import reactor.test.StepVerifier; + +import java.util.Collections; + +import org.junit.Before; +import org.junit.Test; +import org.springframework.data.domain.Range; +import org.springframework.data.redis.RedisTestProfileValueSource; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.RedisZSetCommands.Limit; + +/** + * Integration tests for {@link LettuceReactiveStreamCommands}. + * + * @author Mark Paluch + */ +public class LettuceReactiveStreamCommandsTests extends LettuceReactiveCommandsTestsBase { + + @Before + public void before() { + + // TODO: Upgrade to 5.0 + assumeTrue(RedisTestProfileValueSource.matches("redisVersion", "4.9")); + } + + @Test // DATAREDIS-864 + public void xAddShouldAddMessage() { + + connection.streamCommands().xAdd(KEY_1_BBUFFER, Collections.singletonMap(KEY_2_BBUFFER, VALUE_2_BBUFFER)) // + .as(StepVerifier::create) // + .expectNextCount(1) // + .verifyComplete(); + + connection.streamCommands().xLen(KEY_1_BBUFFER) // + .as(StepVerifier::create) // + .expectNext(1L) // + .verifyComplete(); + } + + @Test // DATAREDIS-864 + public void xDelShouldRemoveMessage() { + + String messageId = connection.streamCommands() + .xAdd(KEY_1_BBUFFER, Collections.singletonMap(KEY_2_BBUFFER, VALUE_2_BBUFFER)).block(); + + connection.streamCommands().xDel(KEY_1_BBUFFER, messageId) // + .as(StepVerifier::create) // + .expectNext(1L) // + .verifyComplete(); + + connection.streamCommands().xLen(KEY_1_BBUFFER) // + .as(StepVerifier::create) // + .expectNext(0L) // + .verifyComplete(); + } + + @Test // DATAREDIS-864 + public void xRangeShouldReportMessages() { + + connection.streamCommands().xAdd(KEY_1_BBUFFER, Collections.singletonMap(KEY_1_BBUFFER, VALUE_1_BBUFFER)) // + .as(StepVerifier::create) // + .expectNextCount(1) // + .verifyComplete(); + + connection.streamCommands().xAdd(KEY_1_BBUFFER, Collections.singletonMap(KEY_2_BBUFFER, VALUE_2_BBUFFER)) // + .as(StepVerifier::create) // + .expectNextCount(1) // + .verifyComplete(); + + connection.streamCommands().xRange(KEY_1_BBUFFER, Range.unbounded()) // + .as(StepVerifier::create) // + .assertNext(it -> { + + assertThat(it.getStream()).isEqualTo(KEY_1_BBUFFER); + assertThat(it.getBody()).containsEntry(KEY_1_BBUFFER, VALUE_1_BBUFFER); + + }) // + .expectNextCount(1).verifyComplete(); + + connection.streamCommands().xRange(KEY_1_BBUFFER, Range.unbounded(), Limit.limit().count(1)) // + .as(StepVerifier::create) // + .assertNext(it -> { + + assertThat(it.getStream()).isEqualTo(KEY_1_BBUFFER); + assertThat(it.getBody()).containsEntry(KEY_1_BBUFFER, VALUE_1_BBUFFER); + }) // + .verifyComplete(); + } + + @Test // DATAREDIS-864 + public void xReadShouldReadMessage() { + + connection.streamCommands().xAdd(KEY_1_BBUFFER, Collections.singletonMap(KEY_1_BBUFFER, VALUE_1_BBUFFER)) // + .as(StepVerifier::create) // + .expectNextCount(1) // + .verifyComplete(); + + connection.streamCommands().xRead(StreamOffset.create(KEY_1_BBUFFER, ReadOffset.from("0-0"))) // + .as(StepVerifier::create) // + .assertNext(it -> { + + assertThat(it.getStream()).isEqualTo(KEY_1_BBUFFER); + assertThat(it.getBody()).containsEntry(KEY_1_BBUFFER, VALUE_1_BBUFFER); + }) // + .verifyComplete(); + } + + @Test // DATAREDIS-864 + public void xReadGroupShouldReadMessage() { + + String initialMessage = nativeCommands.xadd(KEY_1, KEY_1, VALUE_1); + nativeCommands.xgroupCreate(XReadArgs.StreamOffset.from(KEY_1, initialMessage), "my-group"); + + nativeCommands.xadd(KEY_1, KEY_2, VALUE_2); + + connection.streamCommands() + .xReadGroup(Consumer.from("my-group", "my-consumer"), + StreamOffset.create(KEY_1_BBUFFER, ReadOffset.lastConsumed())) // + .as(StepVerifier::create) // + .assertNext(it -> { + + assertThat(it.getStream()).isEqualTo(KEY_1_BBUFFER); + assertThat(it.getBody()).containsEntry(KEY_2_BBUFFER, VALUE_2_BBUFFER); + }) // + .verifyComplete(); + } + + @Test // DATAREDIS-864 + public void xRevRangeShouldReportMessages() { + + connection.streamCommands().xAdd(KEY_1_BBUFFER, Collections.singletonMap(KEY_1_BBUFFER, VALUE_1_BBUFFER)) // + .as(StepVerifier::create) // + .expectNextCount(1) // + .verifyComplete(); + + connection.streamCommands().xAdd(KEY_1_BBUFFER, Collections.singletonMap(KEY_2_BBUFFER, VALUE_2_BBUFFER)) // + .as(StepVerifier::create) // + .expectNextCount(1) // + .verifyComplete(); + + connection.streamCommands().xRevRange(KEY_1_BBUFFER, Range.unbounded(), Limit.limit().count(1)) // + .as(StepVerifier::create) // + .assertNext(it -> { + + assertThat(it.getStream()).isEqualTo(KEY_1_BBUFFER); + assertThat(it.getBody()).containsEntry(KEY_2_BBUFFER, VALUE_2_BBUFFER); + }) // + .verifyComplete(); + } +} diff --git a/src/test/java/org/springframework/data/redis/core/AbstractOperationsTestParams.java b/src/test/java/org/springframework/data/redis/core/AbstractOperationsTestParams.java index 7aaa5b0ba..16677959f 100644 --- a/src/test/java/org/springframework/data/redis/core/AbstractOperationsTestParams.java +++ b/src/test/java/org/springframework/data/redis/core/AbstractOperationsTestParams.java @@ -27,7 +27,7 @@ import org.springframework.data.redis.RawObjectFactory; import org.springframework.data.redis.SettingsUtils; import org.springframework.data.redis.StringObjectFactory; import org.springframework.data.redis.connection.RedisStandaloneConfiguration; -import org.springframework.data.redis.connection.jedis.JedisConnectionFactory; +import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory; import org.springframework.data.redis.serializer.GenericJackson2JsonRedisSerializer; import org.springframework.data.redis.serializer.GenericToStringSerializer; import org.springframework.data.redis.serializer.Jackson2JsonRedisSerializer; @@ -65,7 +65,7 @@ abstract public class AbstractOperationsTestParams { throw new RuntimeException("Cannot init XStream", ex); } - JedisConnectionFactory jedisConnectionFactory = new JedisConnectionFactory(standaloneConfiguration); + LettuceConnectionFactory jedisConnectionFactory = new LettuceConnectionFactory(standaloneConfiguration); jedisConnectionFactory.afterPropertiesSet(); RedisTemplate stringTemplate = new StringRedisTemplate(); diff --git a/src/test/java/org/springframework/data/redis/core/DefaultReactiveStreamOperationsTests.java b/src/test/java/org/springframework/data/redis/core/DefaultReactiveStreamOperationsTests.java new file mode 100644 index 000000000..a11fc10f7 --- /dev/null +++ b/src/test/java/org/springframework/data/redis/core/DefaultReactiveStreamOperationsTests.java @@ -0,0 +1,228 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.core; + +import static org.assertj.core.api.Assertions.*; +import static org.junit.Assume.*; + +import reactor.test.StepVerifier; + +import java.util.Collection; +import java.util.Collections; + +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; +import org.junit.runners.Parameterized.Parameters; +import org.springframework.data.domain.Range; +import org.springframework.data.domain.Range.Bound; +import org.springframework.data.redis.ConnectionFactoryTracker; +import org.springframework.data.redis.ObjectFactory; +import org.springframework.data.redis.PersonObjectFactory; +import org.springframework.data.redis.RedisTestProfileValueSource; +import org.springframework.data.redis.connection.RedisConnection; +import org.springframework.data.redis.connection.RedisConnectionFactory; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.connection.RedisZSetCommands.Limit; +import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory; +import org.springframework.data.redis.serializer.RedisSerializer; + +/** + * Integration tests for {@link DefaultReactiveStreamOperations}. + * + * @author Mark Paluch + */ +@RunWith(Parameterized.class) +@SuppressWarnings("unchecked") +public class DefaultReactiveStreamOperationsTests { + + private final ReactiveRedisTemplate redisTemplate; + private final ReactiveStreamOperations streamOperations; + + private final ObjectFactory keyFactory; + private final ObjectFactory valueFactory; + + @Parameters(name = "{4}") + public static Collection testParams() { + return ReactiveOperationsTestParams.testParams(); + } + + @AfterClass + public static void cleanUp() { + ConnectionFactoryTracker.cleanUp(); + } + + /** + * @param redisTemplate + * @param keyFactory + * @param valueFactory + * @param label parameterized test label, no further use besides that. + */ + public DefaultReactiveStreamOperationsTests(ReactiveRedisTemplate redisTemplate, ObjectFactory keyFactory, + ObjectFactory valueFactory, RedisSerializer serializer, String label) { + + // Currently, only Lettuce supports Redis Streams. + // See https://github.com/xetorthio/jedis/issues/1820 + assumeTrue(redisTemplate.getConnectionFactory() instanceof LettuceConnectionFactory); + + // TODO: Change to 5.0 after Redis 5 GA + assumeTrue(RedisTestProfileValueSource.matches("redisVersion", "4.9")); + + this.redisTemplate = redisTemplate; + this.streamOperations = redisTemplate.opsForStream(); + this.keyFactory = keyFactory; + this.valueFactory = valueFactory; + + ConnectionFactoryTracker.add(redisTemplate.getConnectionFactory()); + } + + @Before + public void before() { + + RedisConnectionFactory connectionFactory = (RedisConnectionFactory) redisTemplate.getConnectionFactory(); + RedisConnection connection = connectionFactory.getConnection(); + connection.flushAll(); + connection.close(); + } + + @Test // DATAREDIS-864 + public void addShouldAddMessage() { + + assumeFalse(valueFactory instanceof PersonObjectFactory); + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + String messageId = streamOperations.add(key, Collections.singletonMap(key, value)).block(); + + streamOperations.range(key, Range.unbounded()) // + .as(StepVerifier::create) // + .consumeNextWith(actual -> { + + assertThat(actual.getId()).isEqualTo(messageId); + assertThat(actual.getStream()).isEqualTo(key); + + if (!(key instanceof byte[] || value instanceof byte[])) { + assertThat(actual.getBody()).containsEntry(key, value); + } + }) // + .verifyComplete(); + } + + @Test // DATAREDIS-864 + public void rangeShouldReportMessages() { + + assumeFalse(valueFactory instanceof PersonObjectFactory); + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + String messageId1 = streamOperations.add(key, Collections.singletonMap(key, value)).block(); + String messageId2 = streamOperations.add(key, Collections.singletonMap(key, value)).block(); + + streamOperations + .range(key, Range.from(Bound.inclusive(messageId1)).to(Bound.inclusive(messageId2)), Limit.limit().count(1)) // + .as(StepVerifier::create) // + .consumeNextWith(actual -> { + + assertThat(actual.getId()).isEqualTo(messageId1); + }) // + .verifyComplete(); + } + + @Test // DATAREDIS-864 + public void reverseRangeShouldReportMessages() { + + assumeFalse(valueFactory instanceof PersonObjectFactory); + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + String messageId1 = streamOperations.add(key, Collections.singletonMap(key, value)).block(); + String messageId2 = streamOperations.add(key, Collections.singletonMap(key, value)).block(); + + streamOperations.reverseRange(key, Range.unbounded()).map(StreamMessage::getId) // + .as(StepVerifier::create) // + .expectNext(messageId2, messageId1) // + .verifyComplete(); + } + + @Test // DATAREDIS-864 + public void readShouldReadMessage() { + + assumeFalse(valueFactory instanceof PersonObjectFactory); + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + String messageId = streamOperations.add(key, Collections.singletonMap(key, value)).block(); + + streamOperations.read(StreamOffset.create(key, ReadOffset.from("0-0"))) // + .as(StepVerifier::create) // + .consumeNextWith(actual -> { + + assertThat(actual.getId()).isEqualTo(messageId); + assertThat(actual.getStream()).isEqualTo(key); + + if (!(key instanceof byte[] || value instanceof byte[])) { + assertThat(actual.getBody()).containsEntry(key, value); + } + }).verifyComplete(); + } + + @Test // DATAREDIS-864 + public void readShouldReadMessages() { + + assumeFalse(valueFactory instanceof PersonObjectFactory); + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + streamOperations.add(key, Collections.singletonMap(key, value)).block(); + streamOperations.add(key, Collections.singletonMap(key, value)).block(); + + streamOperations.read(StreamReadOptions.empty().count(2), StreamOffset.create(key, ReadOffset.from("0-0"))) // + .as(StepVerifier::create) // + .expectNextCount(2) // + .verifyComplete(); + } + + @Test // DATAREDIS-864 + public void sizeShouldReportStreamSize() { + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + streamOperations.add(key, Collections.singletonMap(key, value)).as(StepVerifier::create).expectNextCount(1) + .verifyComplete(); + streamOperations.size(key) // + .as(StepVerifier::create) // + .expectNext(1L) // + .verifyComplete(); + + streamOperations.add(key, Collections.singletonMap(key, value)).as(StepVerifier::create).expectNextCount(1) + .verifyComplete(); + streamOperations.size(key) // + .as(StepVerifier::create) // + .expectNext(2L) // + .verifyComplete(); + } +} diff --git a/src/test/java/org/springframework/data/redis/core/DefaultStreamOperationsTests.java b/src/test/java/org/springframework/data/redis/core/DefaultStreamOperationsTests.java new file mode 100644 index 000000000..aae8fe097 --- /dev/null +++ b/src/test/java/org/springframework/data/redis/core/DefaultStreamOperationsTests.java @@ -0,0 +1,225 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.core; + +import static org.assertj.core.api.Assertions.*; +import static org.junit.Assume.*; + +import java.util.Collection; +import java.util.Collections; +import java.util.List; + +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; +import org.junit.runners.Parameterized.Parameters; +import org.springframework.data.domain.Range; +import org.springframework.data.domain.Range.Bound; +import org.springframework.data.redis.ConnectionFactoryTracker; +import org.springframework.data.redis.ObjectFactory; +import org.springframework.data.redis.RedisTestProfileValueSource; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamReadOptions; +import org.springframework.data.redis.connection.RedisZSetCommands.Limit; +import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory; + +/** + * Integration test of {@link DefaultStreamOperations} + * + * @author Mark Paluch + */ +@RunWith(Parameterized.class) +public class DefaultStreamOperationsTests { + + private RedisTemplate redisTemplate; + + private ObjectFactory keyFactory; + + private ObjectFactory valueFactory; + + private StreamOperations streamOps; + + public DefaultStreamOperationsTests(RedisTemplate redisTemplate, ObjectFactory keyFactory, + ObjectFactory valueFactory) { + + // Currently, only Lettuce supports Redis Streams. + // See https://github.com/xetorthio/jedis/issues/1820 + assumeTrue(redisTemplate.getConnectionFactory() instanceof LettuceConnectionFactory); + + // TODO: Change to 5.0 after Redis 5 GA + assumeTrue(RedisTestProfileValueSource.matches("redisVersion", "4.9")); + + this.redisTemplate = redisTemplate; + this.keyFactory = keyFactory; + this.valueFactory = valueFactory; + + ConnectionFactoryTracker.add(redisTemplate.getConnectionFactory()); + } + + @Parameters + public static Collection testParams() { + return AbstractOperationsTestParams.testParams(); + } + + @AfterClass + public static void cleanUp() { + ConnectionFactoryTracker.cleanUp(); + } + + @Before + public void setUp() { + streamOps = redisTemplate.opsForStream(); + + redisTemplate.execute((RedisCallback) connection -> { + connection.flushDb(); + return null; + }); + } + + @Test // DATAREDIS-864 + public void addShouldAddMessage() { + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + String messageId = streamOps.add(key, Collections.singletonMap(key, value)); + + List> messages = streamOps.range(key, Range.unbounded()); + + assertThat(messages).hasSize(1); + + StreamMessage message = messages.get(0); + + assertThat(message.getId()).isEqualTo(messageId); + assertThat(message.getStream()).isEqualTo(key); + + if (!(key instanceof byte[] || value instanceof byte[])) { + assertThat(message.getBody()).containsEntry(key, value); + } + } + + @Test // DATAREDIS-864 + public void rangeShouldReportMessages() { + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + String messageId1 = streamOps.add(key, Collections.singletonMap(key, value)); + String messageId2 = streamOps.add(key, Collections.singletonMap(key, value)); + + List> messages = streamOps.range(key, + Range.from(Bound.inclusive(messageId1)).to(Bound.inclusive(messageId2)), Limit.limit().count(1)); + + assertThat(messages).hasSize(1); + + StreamMessage message = messages.get(0); + + assertThat(message.getId()).isEqualTo(messageId1); + } + + @Test // DATAREDIS-864 + public void reverseRangeShouldReportMessages() { + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + String messageId1 = streamOps.add(key, Collections.singletonMap(key, value)); + String messageId2 = streamOps.add(key, Collections.singletonMap(key, value)); + + List> messages = streamOps.reverseRange(key, Range.unbounded()); + + assertThat(messages).hasSize(2).extracting("id").containsSequence(messageId2, messageId1); + } + + @Test // DATAREDIS-864 + public void readShouldReadMessage() { + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + String messageId = streamOps.add(key, Collections.singletonMap(key, value)); + + List> messages = streamOps.read(StreamOffset.create(key, ReadOffset.from("0-0"))); + + assertThat(messages).hasSize(1); + + StreamMessage message = messages.get(0); + + assertThat(message.getId()).isEqualTo(messageId); + assertThat(message.getStream()).isEqualTo(key); + + if (!(key instanceof byte[] || value instanceof byte[])) { + assertThat(message.getBody()).containsEntry(key, value); + } + } + + @Test // DATAREDIS-864 + public void readShouldReadMessages() { + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + streamOps.add(key, Collections.singletonMap(key, value)); + streamOps.add(key, Collections.singletonMap(key, value)); + + List> messages = streamOps.read(StreamReadOptions.empty().count(2), + StreamOffset.create(key, ReadOffset.from("0-0"))); + + assertThat(messages).hasSize(2); + } + + @Test // DATAREDIS-864 + public void readShouldReadMessageWithConsumerGroup() { + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + String messageId = streamOps.add(key, Collections.singletonMap(key, value)); + streamOps.createGroup(key, ReadOffset.from("0-0"), "my-group"); + + List> messages = streamOps.read(Consumer.from("my-group", "my-consumer"), + StreamOffset.create(key, ReadOffset.lastConsumed())); + + assertThat(messages).hasSize(1); + + StreamMessage message = messages.get(0); + + assertThat(message.getId()).isEqualTo(messageId); + assertThat(message.getStream()).isEqualTo(key); + + if (!(key instanceof byte[] || value instanceof byte[])) { + assertThat(message.getBody()).containsEntry(key, value); + } + } + + @Test // DATAREDIS-864 + public void sizeShouldReportStreamSize() { + + K key = keyFactory.instance(); + V value = valueFactory.instance(); + + streamOps.add(key, Collections.singletonMap(key, value)); + assertThat(streamOps.size(key)).isEqualTo(1); + + streamOps.add(key, Collections.singletonMap(key, value)); + assertThat(streamOps.size(key)).isEqualTo(2); + } +} diff --git a/src/test/java/org/springframework/data/redis/core/RedisTemplateTests.java b/src/test/java/org/springframework/data/redis/core/RedisTemplateTests.java index 899dfa0ab..d49521309 100644 --- a/src/test/java/org/springframework/data/redis/core/RedisTemplateTests.java +++ b/src/test/java/org/springframework/data/redis/core/RedisTemplateTests.java @@ -342,6 +342,11 @@ public class RedisTemplateTests { operations.opsForList().rightPop(key1); operations.opsForList().size(key1); operations.exec(); + + try { + // Await EXEC completion as it's executed on a dedicated connection. + Thread.sleep(100); + } catch (InterruptedException e) {} operations.opsForValue().set(key1, value1); operations.opsForValue().get(key1); return null; @@ -704,11 +709,7 @@ public class RedisTemplateTests { } }); - if (redisTemplate.getConnectionFactory() instanceof JedisConnectionFactory) { - assertThat(results, is(empty())); - } else { - assertNull(results); - } + assertThat(results, is(empty())); assertThat(redisTemplate.opsForValue().get(key1), isEqual(value2)); } @@ -777,11 +778,7 @@ public class RedisTemplateTests { } }); - if (redisTemplate.getConnectionFactory() instanceof JedisConnectionFactory) { assertThat(results, is(empty())); - } else { - assertNull(results); - } assertThat(redisTemplate.opsForValue().get(key1), isEqual(value2)); } diff --git a/src/test/java/org/springframework/data/redis/stream/ReadOffsetStrategyUnitTests.java b/src/test/java/org/springframework/data/redis/stream/ReadOffsetStrategyUnitTests.java new file mode 100644 index 000000000..3f724a324 --- /dev/null +++ b/src/test/java/org/springframework/data/redis/stream/ReadOffsetStrategyUnitTests.java @@ -0,0 +1,97 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import static org.assertj.core.api.Assertions.*; + +import java.util.Optional; + +import org.junit.Test; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; + +/** + * Unit tests for {@link ReadOffsetStrategy}. + * + * @author Mark Paluch + */ +public class ReadOffsetStrategyUnitTests { + + static Optional consumer = Optional.of(Consumer.from("foo", "bar")); + + @Test // DATAREDIS-864 + public void nextMessageStandaloneShouldReturnLastSeenMessageId() { + + ReadOffset offset = ReadOffset.from("foo"); + + assertThat(ReadOffsetStrategy.NextMessage.getFirst(offset, Optional.empty())).isEqualTo(offset); + assertThat(ReadOffsetStrategy.NextMessage.getNext(offset, Optional.empty(), "42")).isEqualTo(ReadOffset.from("42")); + } + + @Test // DATAREDIS-864 + public void lastConsumedStandaloneShouldReturnLastSeenMessageId() { + + ReadOffset offset = ReadOffset.lastConsumed(); + + assertThat(ReadOffsetStrategy.LastConsumed.getFirst(offset, Optional.empty())).isEqualTo(ReadOffset.latest()); + assertThat(ReadOffsetStrategy.LastConsumed.getNext(offset, Optional.empty(), "42")) + .isEqualTo(ReadOffset.from("42")); + } + + @Test // DATAREDIS-864 + public void latestStandaloneShouldReturnLatest() { + + ReadOffset offset = ReadOffset.latest(); + + assertThat(ReadOffsetStrategy.Latest.getFirst(offset, Optional.empty())).isEqualTo(ReadOffset.latest()); + assertThat(ReadOffsetStrategy.Latest.getNext(offset, Optional.empty(), "42")).isEqualTo(ReadOffset.latest()); + } + + @Test // DATAREDIS-864 + public void nextMessageConsumerGroupShouldReturnLastSeenMessageId() { + + ReadOffset offset = ReadOffset.from("foo"); + + assertThat(ReadOffsetStrategy.NextMessage.getFirst(offset, consumer)).isEqualTo(offset); + assertThat(ReadOffsetStrategy.NextMessage.getNext(offset, consumer, "42")).isEqualTo(ReadOffset.from("42")); + } + + @Test // DATAREDIS-864 + public void lastConsumedConsumerGroupShouldReturnLastSeenMessageId() { + + ReadOffset offset = ReadOffset.lastConsumed(); + + assertThat(ReadOffsetStrategy.LastConsumed.getFirst(offset, consumer)).isEqualTo(ReadOffset.lastConsumed()); + assertThat(ReadOffsetStrategy.LastConsumed.getNext(offset, consumer, "42")).isEqualTo(ReadOffset.lastConsumed()); + } + + @Test // DATAREDIS-864 + public void latestConsumerGroupShouldReturnLatest() { + + ReadOffset offset = ReadOffset.latest(); + + assertThat(ReadOffsetStrategy.Latest.getFirst(offset, consumer)).isEqualTo(ReadOffset.latest()); + assertThat(ReadOffsetStrategy.Latest.getNext(offset, consumer, "42")).isEqualTo(ReadOffset.latest()); + } + + @Test // DATAREDIS-864 + public void getStrategyShouldReturnAppropriateStrategy() { + + assertThat(ReadOffsetStrategy.getStrategy(ReadOffset.from("foo"))).isEqualTo(ReadOffsetStrategy.NextMessage); + assertThat(ReadOffsetStrategy.getStrategy(ReadOffset.lastConsumed())).isEqualTo(ReadOffsetStrategy.LastConsumed); + assertThat(ReadOffsetStrategy.getStrategy(ReadOffset.latest())).isEqualTo(ReadOffsetStrategy.Latest); + } +} diff --git a/src/test/java/org/springframework/data/redis/stream/StreamMessageListenerContainerIntegrationTests.java b/src/test/java/org/springframework/data/redis/stream/StreamMessageListenerContainerIntegrationTests.java new file mode 100644 index 000000000..d533464de --- /dev/null +++ b/src/test/java/org/springframework/data/redis/stream/StreamMessageListenerContainerIntegrationTests.java @@ -0,0 +1,286 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import static org.assertj.core.api.Assertions.*; +import static org.junit.Assume.*; + +import java.time.Duration; +import java.util.Collections; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.TimeUnit; + +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.springframework.data.redis.ConnectionFactoryTracker; +import org.springframework.data.redis.RedisVersionUtils; +import org.springframework.data.redis.SettingsUtils; +import org.springframework.data.redis.connection.RedisConnection; +import org.springframework.data.redis.connection.RedisConnectionFactory; +import org.springframework.data.redis.connection.RedisStandaloneConfiguration; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.lettuce.LettuceClientConfiguration; +import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory; +import org.springframework.data.redis.connection.lettuce.LettuceTestClientResources; +import org.springframework.data.redis.core.StringRedisTemplate; +import org.springframework.data.redis.stream.StreamMessageListenerContainer.StreamMessageListenerContainerOptions; +import org.springframework.data.redis.stream.StreamMessageListenerContainer.StreamReadRequest; + +/** + * Integration tests for {@link StreamMessageListenerContainer}. + * + * @author Mark Paluch + */ +public class StreamMessageListenerContainerIntegrationTests { + + private static final RedisStandaloneConfiguration standaloneConfiguration = new RedisStandaloneConfiguration( + SettingsUtils.getHost(), SettingsUtils.getPort()); + + private static RedisConnectionFactory connectionFactory; + + StringRedisTemplate redisTemplate = new StringRedisTemplate(connectionFactory); + StreamMessageListenerContainerOptions containerOptions = StreamMessageListenerContainerOptions + .builder().pollTimeout(Duration.ofMillis(100)).build(); + + @BeforeClass + public static void beforeClass() { + + LettuceClientConfiguration clientConfiguration = LettuceClientConfiguration.builder() // + .shutdownTimeout(Duration.ZERO) // + .clientResources(LettuceTestClientResources.getSharedClientResources()) // + .build(); + + LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(standaloneConfiguration, + clientConfiguration); + lettuceConnectionFactory.afterPropertiesSet(); + + ConnectionFactoryTracker.add(lettuceConnectionFactory); + + connectionFactory = lettuceConnectionFactory; + + // TODO: Upgrade to 5.0 + assumeTrue(RedisVersionUtils.atLeast("4.9", connectionFactory.getConnection())); + } + + @AfterClass + public static void tearDown() { + ConnectionFactoryTracker.cleanUp(); + } + + @Before + public void before() { + + RedisConnection connection = connectionFactory.getConnection(); + connection.flushDb(); + connection.close(); + } + + @Test // DATAREDIS-864 + public void shouldReceiveMessages() throws InterruptedException { + + StreamMessageListenerContainer container = StreamMessageListenerContainer.create(connectionFactory, + containerOptions); + BlockingQueue> queue = new LinkedBlockingQueue<>(); + + container.start(); + Subscription subscription = container.receive(StreamOffset.create("my-stream", ReadOffset.from("0-0")), queue::add); + + subscription.await(Duration.ofSeconds(2)); + + redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value1")); + redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value2")); + redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value3")); + + assertThat(queue.poll(1, TimeUnit.SECONDS)).isNotNull(); + assertThat(queue.poll(1, TimeUnit.SECONDS)).isNotNull(); + assertThat(queue.poll(1, TimeUnit.SECONDS)).isNotNull(); + + cancelAwait(subscription); + + assertThat(subscription.isActive()).isFalse(); + } + + @Test // DATAREDIS-864 + public void shouldReceiveMessagesInConsumerGroup() throws InterruptedException { + + StreamMessageListenerContainer container = StreamMessageListenerContainer.create(connectionFactory, + containerOptions); + BlockingQueue> queue = new LinkedBlockingQueue<>(); + String messageId = redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value1")); + redisTemplate.opsForStream().createGroup("my-stream", ReadOffset.from(messageId), "my-group"); + + container.start(); + Subscription subscription = container.receive(Consumer.from("my-group", "my-consumer"), + StreamOffset.create("my-stream", ReadOffset.lastConsumed()), queue::add); + + subscription.await(Duration.ofSeconds(2)); + + redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value2")); + + StreamMessage message = queue.poll(1, TimeUnit.SECONDS); + assertThat(message).isNotNull(); + assertThat(message.getBody()).containsEntry("key", "value2"); + + cancelAwait(subscription); + } + + @Test // DATAREDIS-864 + public void shouldUseCustomErrorHandler() throws InterruptedException { + + BlockingQueue failures = new LinkedBlockingQueue<>(); + + StreamMessageListenerContainerOptions containerOptions = StreamMessageListenerContainerOptions + .builder().errorHandler(failures::add).pollTimeout(Duration.ofMillis(100)).build(); + StreamMessageListenerContainer container = StreamMessageListenerContainer.create(connectionFactory, + containerOptions); + + container.start(); + Subscription subscription = container.receive(Consumer.from("my-group", "my-consumer"), + StreamOffset.create("my-stream", ReadOffset.lastConsumed()), it -> {}); + + subscription.await(Duration.ofSeconds(2)); + + Throwable error = failures.poll(1, TimeUnit.SECONDS); + assertThat(failures).isEmpty(); + assertThat(error).isNotNull(); + + cancelAwait(subscription); + } + + @Test // DATAREDIS-864 + public void errorShouldStopListening() throws InterruptedException { + + BlockingQueue failures = new LinkedBlockingQueue<>(); + + StreamMessageListenerContainer container = StreamMessageListenerContainer.create(connectionFactory, + containerOptions); + + StreamReadRequest readRequest = StreamReadRequest + .builder(StreamOffset.create("my-stream", ReadOffset.lastConsumed())).errorHandler(failures::add) + .consumer(Consumer.from("my-group", "my-consumer")).build(); + + String messageId = redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value1")); + redisTemplate.opsForStream().createGroup("my-stream", ReadOffset.from(messageId), "my-group"); + + container.start(); + Subscription subscription = container.register(readRequest, it -> {}); + + subscription.await(Duration.ofSeconds(1)); + + redisTemplate.delete("my-stream"); + + subscription.await(Duration.ofSeconds(1)); + + assertThat(failures.poll(1, TimeUnit.SECONDS)).isNotNull(); + assertThat(subscription.isActive()).isFalse(); + + cancelAwait(subscription); + } + + @Test // DATAREDIS-864 + public void customizedCancelPredicateShouldNotStopListening() throws InterruptedException { + + BlockingQueue failures = new LinkedBlockingQueue<>(); + + StreamMessageListenerContainer container = StreamMessageListenerContainer.create(connectionFactory, + containerOptions); + + StreamReadRequest readRequest = StreamReadRequest + .builder(StreamOffset.create("my-stream", ReadOffset.lastConsumed())) // + .errorHandler(failures::add) // // + .cancelOnError(t -> false) // + .consumer(Consumer.from("my-group", "my-consumer")) // + .build(); + + String messageId = redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value1")); + redisTemplate.opsForStream().createGroup("my-stream", ReadOffset.from(messageId), "my-group"); + + container.start(); + Subscription subscription = container.register(readRequest, it -> {}); + + subscription.await(Duration.ofSeconds(2)); + + redisTemplate.delete("my-stream"); + + assertThat(failures.poll(1, TimeUnit.SECONDS)).isNotNull(); + assertThat(failures.poll(1, TimeUnit.SECONDS)).isNotNull(); + assertThat(subscription.isActive()).isTrue(); + + cancelAwait(subscription); + } + + @Test // DATAREDIS-864 + public void cancelledStreamShouldNotReceiveMessages() throws InterruptedException { + + StreamMessageListenerContainer container = StreamMessageListenerContainer.create(connectionFactory, + containerOptions); + BlockingQueue> queue = new LinkedBlockingQueue<>(); + + container.start(); + Subscription subscription = container.receive(StreamOffset.create("my-stream", ReadOffset.from("0-0")), queue::add); + + subscription.await(Duration.ofSeconds(2)); + cancelAwait(subscription); + + redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value4")); + + assertThat(queue.poll(200, TimeUnit.MILLISECONDS)).isNull(); + } + + @Test // DATAREDIS-864 + public void containerRestartShouldRestartSubscription() throws InterruptedException { + + StreamMessageListenerContainer container = StreamMessageListenerContainer.create(connectionFactory, + containerOptions); + BlockingQueue> queue = new LinkedBlockingQueue<>(); + + container.start(); + Subscription subscription = container.receive(StreamOffset.create("my-stream", ReadOffset.from("0-0")), queue::add); + + subscription.await(Duration.ofSeconds(2)); + + container.stop(); + + while (subscription.isActive()) { + Thread.sleep(10); + } + + container.start(); + + subscription.await(Duration.ofSeconds(2)); + + redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value1")); + + assertThat(queue.poll(1, TimeUnit.SECONDS)).isNotNull(); + + cancelAwait(subscription); + } + + private static void cancelAwait(Subscription subscription) throws InterruptedException { + + subscription.cancel(); + + while (subscription.isActive()) { + Thread.sleep(10); + } + } +} diff --git a/src/test/java/org/springframework/data/redis/stream/StreamReceiverIntegrationTests.java b/src/test/java/org/springframework/data/redis/stream/StreamReceiverIntegrationTests.java new file mode 100644 index 000000000..a3feabfc1 --- /dev/null +++ b/src/test/java/org/springframework/data/redis/stream/StreamReceiverIntegrationTests.java @@ -0,0 +1,199 @@ +/* + * Copyright 2018 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.springframework.data.redis.stream; + +import static org.assertj.core.api.Assertions.*; +import static org.junit.Assume.*; + +import reactor.core.publisher.Flux; +import reactor.test.StepVerifier; + +import java.time.Duration; +import java.util.Collections; + +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.springframework.data.redis.ConnectionFactoryTracker; +import org.springframework.data.redis.RedisSystemException; +import org.springframework.data.redis.RedisVersionUtils; +import org.springframework.data.redis.SettingsUtils; +import org.springframework.data.redis.connection.RedisConnection; +import org.springframework.data.redis.connection.RedisStandaloneConfiguration; +import org.springframework.data.redis.connection.RedisStreamCommands.Consumer; +import org.springframework.data.redis.connection.RedisStreamCommands.ReadOffset; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamMessage; +import org.springframework.data.redis.connection.RedisStreamCommands.StreamOffset; +import org.springframework.data.redis.connection.lettuce.LettuceClientConfiguration; +import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory; +import org.springframework.data.redis.connection.lettuce.LettuceTestClientResources; +import org.springframework.data.redis.core.StringRedisTemplate; +import org.springframework.data.redis.stream.StreamReceiver.StreamReceiverOptions; + +/** + * Integration tests for {@link StreamReceiver}. + * + * @author Mark Paluch + */ +public class StreamReceiverIntegrationTests { + + private static final RedisStandaloneConfiguration standaloneConfiguration = new RedisStandaloneConfiguration( + SettingsUtils.getHost(), SettingsUtils.getPort()); + + private static LettuceConnectionFactory connectionFactory; + StringRedisTemplate redisTemplate = new StringRedisTemplate(connectionFactory); + + @BeforeClass + public static void beforeClass() { + + LettuceClientConfiguration clientConfiguration = LettuceClientConfiguration.builder() // + .shutdownTimeout(Duration.ZERO) // + .clientResources(LettuceTestClientResources.getSharedClientResources()) // + .build(); + + LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(standaloneConfiguration, + clientConfiguration); + lettuceConnectionFactory.afterPropertiesSet(); + + ConnectionFactoryTracker.add(lettuceConnectionFactory); + + connectionFactory = lettuceConnectionFactory; + + // TODO: Upgrade to 5.0 + assumeTrue(RedisVersionUtils.atLeast("4.9", connectionFactory.getConnection())); + } + + @AfterClass + public static void tearDown() { + ConnectionFactoryTracker.cleanUp(); + } + + @Before + public void before() { + + RedisConnection connection = connectionFactory.getConnection(); + connection.flushDb(); + connection.close(); + } + + @Test // DATAREDIS-864 + public void shouldReceiveMessages() { + + StreamReceiver receiver = StreamReceiver.create(connectionFactory); + + Flux> messages = receiver + .receive(StreamOffset.create("my-stream", ReadOffset.from("0-0"))); + + messages.as(StepVerifier::create) // + .then(() -> redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value"))) + .consumeNextWith(it -> { + + assertThat(it.getStream()).isEqualTo("my-stream"); + assertThat(it.getBody()).containsEntry("key", "value"); + }) // + .thenCancel() // + .verify(Duration.ofSeconds(5)); + } + + @Test // DATAREDIS-864 + public void latestModeLosesMessages() { + + // XADD/XREAD highly timing-dependent as this tests require a poll subscription to receive messages using $ offset. + + StreamReceiverOptions options = StreamReceiverOptions.builder().pollTimeout(Duration.ofSeconds(4)) + .build(); + StreamReceiver receiver = StreamReceiver.create(connectionFactory, options); + + Flux> messages = receiver + .receive(StreamOffset.create("my-stream", ReadOffset.latest())); + + messages.as(publisher -> StepVerifier.create(publisher, 0)) // + .thenRequest(1) // + .then(() -> { + try { + Thread.sleep(500); + redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value1")); + } catch (InterruptedException e) {} + }) // + .expectNextCount(1) // + .then(() -> { + redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value2")); + }) // + .thenRequest(1) // + .then(() -> { + try { + Thread.sleep(500); + redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value3")); + } catch (InterruptedException e) {} + }).consumeNextWith(it -> { + + assertThat(it.getStream()).isEqualTo("my-stream"); + assertThat(it.getBody()).containsEntry("key", "value3"); + }) // + .thenCancel() // + .verify(Duration.ofSeconds(5)); + } + + @Test // DATAREDIS-864 + public void shouldReceiveAsConsumerGroupMessages() { + + StreamReceiver receiver = StreamReceiver.create(connectionFactory); + + Flux> messages = receiver.receive(Consumer.from("my-group", "my-consumer-id"), + StreamOffset.create("my-stream", ReadOffset.lastConsumed())); + + // required to initialize stream + redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value")); + redisTemplate.opsForStream().createGroup("my-stream", ReadOffset.from("0-0"), "my-group"); + redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key2", "value2")); + + messages.as(StepVerifier::create) // + .consumeNextWith(it -> { + + assertThat(it.getStream()).isEqualTo("my-stream"); + assertThat(it.getBody()).containsEntry("key", "value"); + }).consumeNextWith(it -> { + + assertThat(it.getStream()).isEqualTo("my-stream"); + assertThat(it.getBody()).containsEntry("key2", "value2"); + }) // + .thenCancel() // + .verify(Duration.ofSeconds(5)); + } + + @Test // DATAREDIS-864 + public void shouldStopReceivingOnError() { + + StreamReceiverOptions options = StreamReceiverOptions.builder().pollTimeout(Duration.ofMillis(100)) + .build(); + + StreamReceiver receiver = StreamReceiver.create(connectionFactory, options); + + Flux> messages = receiver.receive(Consumer.from("my-group", "my-consumer-id"), + StreamOffset.create("my-stream", ReadOffset.lastConsumed())); + + // required to initialize stream + redisTemplate.opsForStream().add("my-stream", Collections.singletonMap("key", "value")); + redisTemplate.opsForStream().createGroup("my-stream", ReadOffset.from("0-0"), "my-group"); + + messages.as(StepVerifier::create) // + .expectNextCount(1) // + .then(() -> redisTemplate.delete("my-stream")) // + .expectError(RedisSystemException.class) // + .verify(Duration.ofSeconds(5)); + } +} diff --git a/src/test/resources/logback.xml b/src/test/resources/logback.xml index 897f4591b..9c7903aef 100644 --- a/src/test/resources/logback.xml +++ b/src/test/resources/logback.xml @@ -13,4 +13,4 @@ - \ No newline at end of file +