INT-3045: Add ZeroMqChannel support (#3355)

* INT-3045: Add `ZeroMqChannel` support

JIRA: https://jira.spring.io/browse/INT-3045

Provide a `SubscribableChannel` implementation for ZeroMQ

The general idea is to let to have a distributed channel implementation
where every client can connect to a single server backed by the channel.

The logic in the channel is fully transparent for end-user and there is just
enough to send message to it and subscribe for receiving on the other side.
If PUB/SUB model is used, all the subscribes (even over the network) going to
receive the same published message.
In case of PUSH/PULL only one subscriber in the whole cluster is going to get
the published message

* Use Reactor for better threading control
* JeroMQ is not interruptible-friendly: use control sockets to stop proxy loop
* Name Reactor's schedulers to avoid daemon threads

* * Use try-catch-with-resource to close sockets automatically
* Fix Checkstyle violations
* Use `Mono.handle()` to receive data from the socket

* * Optimize local for just a couple of PAIR sockets
* Implement TCP binding
* Add PUB/SUB tests

* * Fix subscriber scheduler name
* Optimize socket create logic
* Add PUSH/PULL over TCP test

* * Fix subscriber scheduler name
* Optimize socket create logic
* Add PUSH/PULL over TCP test
* Implement PUB/SUB over TCP

* * Introduce `ZeroMqProxy` - Spring-friendly component to configure and manage ZeroMq proxy
* Use this `ZeroMqProxy` logic as an external component for `ZeroMqChannel` testing

* * Fix Checkstyle
* Apply docs polishing
* Expose a capture socket on the proxy
* Implement `DisposableBean` in the `ZeroMqProxy` to destroy an internal executor service
* Add JavaDocs to `ZeroMqChannel`
* Add one more `ZeroMqChannel` to TCP test to be sure that proxy distribution works well

* * Add `hamcrest-core` dependency for Awatility

* * Add more JavaDocs to `ZeroMqProxy` and `ZeroMqChannel`
* Expose `ZeroMqChannel.setZeroMqProxy()` option for easier
configuration within the same application context
* Make `ZeroMqChannel` sockets configuration and connection
dependant on provided `ZeroMqProxy` (if any)
* Add `Consumer<ZMQ.Socket>` configuration callbacks to the `ZeroMqChannel`
* Expose `ZeroMqChannel.consumeDelay` option

* * Add docs for ZeroMQ
* Some additions into a `reactive-streams.adoc`
* Fix typo in the `xmpp.adoc`

* * Add `optional` `jackson-databind` since `ZeroMqChannel` uses it by default
* More words into docs

* * Fix language in docs according review

* Fix language in docs according review

Co-authored-by: Gary Russell <grussell@vmware.com>

* Apply suggestions from code review

Co-authored-by: Oliver <oli-ver@users.noreply.github.com>

* * Fix threading using a `publishOn()` for specific scheduler after `cache()`

* * Remove unused import

* * Change proxy port check from static `Mono.just()` to `Mono.fromCallable()`
to really evaluate the current port state on every repeat
* Add finite `100` repeat number to avoid infinite blocking when proxy is not started at all
* Add `doOnError()` for proxy `Mono` to log `ERROR` when repeat is exhausted

* * Fix Checkstyle violation

Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Oliver <oli-ver@users.noreply.github.com>

src/reference/asciidoc/reactive-streams.adoc

@@ -155,7 +155,8 @@ A reactive outbound channel adapter implementation is about initiation (or conti
 An inbound payload could be a reactive type per se or as an event of the whole integration flow which is a part of reactive stream on top.
 A returned reactive type can be subscribed immediately if we are in one-way, fire-and-forget scenario, or it is propagated downstream (request-reply scenarios) for further integration flow or an explicit subscription in the target business logic, but still downstream preserving reactive streams semantics.
 
-Currently Spring Integration provides channel adapter (or gateway) implementations for <<./webflux.adoc#webflux,WebFlux>>, <<./rsocket.adoc#rsocket,RSocket>> and <<./mongodb.adoc#mongodb,MongoDb>>.
+Currently Spring Integration provides channel adapter (or gateway) implementations for <<./webflux.adoc#webflux,WebFlux>>, <<./rsocket.adoc#rsocket,RSocket>>, <<./mongodb.adoc#mongodb,MongoDb>> and <<./r2dbc.adoc#r2dbc,R2DBC>>.
+The <<./redis.adoc#redis-stream-outbound,Redis Stream Channel Adapters>> are also reactive and uses `ReactiveStreamOperations` from Spring Data.
 Also an https://github.com/spring-projects/spring-integration-extensions/tree/master/spring-integration-cassandra[Apache Cassandra Extension] provides a `MessageHandler` implementation for the Cassandra reactive driver.
-More reactive channel adapters are coming, for example for https://r2dbc.io/[R2DBC], for Apache Kafka in https://github.com/spring-projects/spring-integration-kafka[Spring Integration Kafka] based on the `ReactiveKafkaProducerTemplate` and `ReactiveKafkaConsumerTemplate` from https://spring.io/projects/spring-kafka[Spring for Apache Kafka] etc.
+More reactive channel adapters are coming, for example for Apache Kafka in <<./kafka.adoc#kafka,Kafka>> based on the `ReactiveKafkaProducerTemplate` and `ReactiveKafkaConsumerTemplate` from https://spring.io/projects/spring-kafka[Spring for Apache Kafka] etc.
 For many other non-reactive channel adapters thread pools are recommended to avoid blocking during reactive stream processing.

src/reference/asciidoc/redis.adoc

@@ -793,3 +793,8 @@ Starting with version 5.0, the `RedisLockRegistry` implements `ExpirableLockRegi
 === Redis Stream Outbound Channel Adapter
 
 TBD
+
+[[redis-stream-inbound]]
+=== Redis Stream Inbound Channel Adapter
+
+TBD

src/reference/asciidoc/whats-new.adoc

@@ -23,21 +23,30 @@ See <<./kafka.adoc#kafka,Spring for Apache Kafka Support>> for more information.
 The `KafkaProducerMessageHandler` `sendTimeoutExpression` default has changed.
 See <<./kafka.adoc#kafka-outbound,Kafka Outbound Channel Adapter>> for more information.
 
+[[x5.4-r2dbc]]
 ==== R2DBC Channel Adapters
 
 The Channel Adapters for R2DBC database interaction have been introduced.
 See <<./r2dbc.adoc#r2dbc,R2DBC Support>> for more information.
 
+[[x5.4-redis-stream]]
 ==== Redis Stream Support
 
 The Channel Adapters for Redis Stream support have been introduced.
 See <<./redis.adoc#redis-stream-outbound,Redis Stream Outbound Channel Adapter>> for more information.
 
+[[x5.4-renewable-lock]]
 ==== Renewable Lock Registry
 
 A Renewable lock registry has been introduced to allow renew lease of a distributed lock.
 See <<./jdbc.adoc#jdbc-lock-registry,JDBC implementation>> for more information.
 
+[[x5.4-zeromq]]
+==== ZeroMQ Support
+
+A `ZeroMqChannel` has been introduced.
+See <<./zeromq.adoc#zeromq,ZeroMQ Support>> for more information.
+
 [[x5.4-general]]
 === General Changes
 
@@ -63,6 +72,7 @@ See <<./ip.adoc#ip-collaborating-adapters,Collaborating Channel Adapters>> and <
 The `spring-integration-rmi` module is deprecated with no replacement and is going to be removed in the next major version.
 See <<./rmi.adoc#rmi, RMI Support>> for more information.
 
+[[x5.4-amqp]]
 === AMQP Changes
 
 The outbound endpoints now have a new mechanism for handling publisher confirms and returns.

src/reference/asciidoc/xmpp.adoc

@@ -314,7 +314,7 @@ public class CustomConnectionConfiguration {
 	SASLAuthentication.supportSASLMechanism("EXTERNAL", 0); // static initializer
 
 	ConnectionConfiguration config = new ConnectionConfiguration("localhost", 5223);
-	config.setTrustorePath("path_to_truststore.jks");
+	config.setKeystorePath("path_to_truststore.jks");
 	config.setSecurityEnabled(true);
 	config.setSocketFactory(SSLSocketFactory.getDefault());
 	return new XMPPConnection(config);

src/reference/asciidoc/zeromq.adoc

@@ -0,0 +1,104 @@
+[[zeromq]]
+== ZeroMQ Support
+
+Spring Integration provides components to support https://zeromq.org/[ZeroMQ] communication in the application.
+The implementation is based on the well-supported Java API of the https://github.com/zeromq/jeromq[JeroMQ] library.
+All components encapsulate ZeroMQ socket lifecycles and manage threads for them internally making interactions with these components lock-free and thread-safe.
+
+You need to include this dependency into your project:
+
+====
+.Maven
+[source, xml, subs="normal"]
+----
+<dependency>
+    <groupId>org.springframework.integration</groupId>
+    <artifactId>spring-integration-zeromq</artifactId>
+    <version>{project-version}</version>
+</dependency>
+----
+
+.Gradle
+[source, groovy, subs="normal"]
+----
+compile "org.springframework.integration:spring-integration-zeromq:{project-version}"
+----
+====
+
+[[zeromq-proxy]]
+=== ZeroMQ Proxy
+
+The `ZeroMqProxy` is a Spring-friendly wrapper for the built-in `ZMQ.proxy()` https://zguide.zeromq.org/page:chapter2#toc15[function].
+It encapsulates socket lifecycles and thread management.
+The clients of this proxy still can use a standard ZeroMQ socket connection and interaction API.
+Alongside with the standard `ZContext` it requires one of the well-known ZeroMQ proxy modes: SUB/PUB, PULL/PUSH or ROUTER/DEALER.
+This way an appropriate pair of ZeroMQ socket types are used for the frontend and backend of the proxy.
+See `ZeroMqProxy.Type` for details.
+
+The `ZeroMqProxy` implements `SmartLifecycle` to create, bind and configure the sockets and to start `ZMQ.proxy()` in a dedicated thread from an `Executor` (if any).
+The binding for frontend and backend sockets is done over the `tcp://` protocol onto all of the available network interfaces with the provided ports.
+Otherwise they are bound to random ports which can be obtained later via the respective `getFrontendPort()` and `getBackendPort()` API methods.
+
+The control socket is exposed as a `SocketType.PAIR` with an inter-thread transport on the `"inproc://" + beanName + ".control"` address; it can be obtained via `getControlAddress()`.
+It should be used with the same application from another `SocketType.PAIR` socket to send `ZMQ.PROXY_TERMINATE`, `ZMQ.PROXY_PAUSE` and/or `ZMQ.PROXY_RESUME` commands.
+The `ZeroMqProxy` performs a `ZMQ.PROXY_TERMINATE` command when `stop()` is called for its lifecycle to terminate the `ZMQ.proxy()` loop and close all the bound sockets gracefully.
+
+The `setExposeCaptureSocket(boolean)` option causes this component to bind an additional inter-thread socket with `SocketType.PUB` to capture and publish all the communication between the frontend and backend sockets as it states with `ZMQ.proxy()` implementation.
+This socket is bound to the `"inproc://" + beanName + ".capture"` address and doesn't expect any specific subscription for filtering.
+
+The frontend and backend sockets can be customized with additional properties, such as read/write timeout or security.
+This customization is available through `setFrontendSocketConfigurer(Consumer<ZMQ.Socket>)` and `setBackendSocketConfigurer(Consumer<ZMQ.Socket>)` callbacks, respectively.
+
+The `ZeroMqProxy` could be provided as simple bean like this:
+
+====
+[source,java]
+----
+@Bean
+ZeroMqProxy zeroMqProxy() {
+    ZeroMqProxy proxy = new ZeroMqProxy(CONTEXT, ZeroMqProxy.Type.SUB_PUB);
+    proxy.setExposeCaptureSocket(true);
+    proxy.setFrontendPort(6001);
+    proxy.setBackendPort(6002);
+    return proxy;
+}
+----
+====
+
+All the client nodes should connect to the host of this proxy via `tcp://` and use the respective port of their interest.
+
+[[zeromq-message-channel]]
+=== ZeroMQ Message Channel
+
+The `ZeroMqChannel` is a `SubscribableChannel` which uses a pair of ZeroMQ sockets to connect publishers and subscribers for messaging interaction.
+It can work in a PUB/SUB mode (defaults to PUSH/PULL); it can also be used as a local inter-thread channel (uses `PAIR` sockets) - the `connectUrl` is not provided in this case.
+In distributed mode it has to be connected to an externally managed ZeroMQ proxy, where it can exchange messages with other similar channels connected to the same proxy.
+The connect url option is a standard ZeroMQ connection string with the protocol and host and a pair of ports over colon for frontend and backend sockets of the ZeroMQ proxy.
+For convenience, the channel could be supplied with the `ZeroMqProxy` instance instead of connection string, if it is configured in the same application as the proxy.
+
+Both sending and receiving sockets are managed in their own dedicated threads making this channel concurrency-friendly.
+This way we can publish and consume to/from a `ZeroMqChannel` from different threads without synchronization.
+
+By default the `ZeroMqChannel` uses an `EmbeddedJsonHeadersMessageMapper` to (de)serialize the `Message` (including headers) from/to `byte[]` using a Jackson JSON processor.
+This logic can be configured via `setMessageMapper(BytesMessageMapper)`.
+
+Sending and receiving sockets can be customized for any options (read/write timeout, security etc.) via respective `setSendSocketConfigurer(Consumer<ZMQ.Socket>)` and `setSubscribeSocketConfigurer(Consumer<ZMQ.Socket>)` callbacks.
+
+The internal logic of the `ZeroMqChannel` is based on the reactive streams via Project Reactor `Flux` and `Mono` operators.
+This provides easier threading control and allows lock-free concurrent publication and consumption to/from the channel.
+Local PUB/SUB logic is implemented as a `Flux.publish()` operator to allow all of the local subscribers to this channel to receive the same published message, as distributed subscribers to the `PUB` socket.
+
+The following is a simple example of a `ZeroMqChannel` configuration:
+
+====
+[source,java]
+----
+@Bean
+ZeroMqChannel zeroMqPubSubChannel(ZContext context) {
+    ZeroMqChannel channel = new ZeroMqChannel(context, true);
+    channel.setConnectUrl("tcp://localhost:6001:6002");
+    channel.setConsumeDelay(Duration.ofMillis(100));
+    return channel;
+}
+----
+====