Commit Graph

10460 Commits

Author SHA1 Message Date
Artem Bilan
a76bb24965 INT-3045: Add ZeroMqChannel support (#3355)
* INT-3045: Add `ZeroMqChannel` support

JIRA: https://jira.spring.io/browse/INT-3045

Provide a `SubscribableChannel` implementation for ZeroMQ

The general idea is to let to have a distributed channel implementation
where every client can connect to a single server backed by the channel.

The logic in the channel is fully transparent for end-user and there is just
enough to send message to it and subscribe for receiving on the other side.
If PUB/SUB model is used, all the subscribes (even over the network) going to
receive the same published message.
In case of PUSH/PULL only one subscriber in the whole cluster is going to get
the published message

* Use Reactor for better threading control
* JeroMQ is not interruptible-friendly: use control sockets to stop proxy loop
* Name Reactor's schedulers to avoid daemon threads

* * Use try-catch-with-resource to close sockets automatically
* Fix Checkstyle violations
* Use `Mono.handle()` to receive data from the socket

* * Optimize local for just a couple of PAIR sockets
* Implement TCP binding
* Add PUB/SUB tests

* * Fix subscriber scheduler name
* Optimize socket create logic
* Add PUSH/PULL over TCP test

* * Fix subscriber scheduler name
* Optimize socket create logic
* Add PUSH/PULL over TCP test
* Implement PUB/SUB over TCP

* * Introduce `ZeroMqProxy` - Spring-friendly component to configure and manage ZeroMq proxy
* Use this `ZeroMqProxy` logic as an external component for `ZeroMqChannel` testing

* * Fix Checkstyle
* Apply docs polishing
* Expose a capture socket on the proxy
* Implement `DisposableBean` in the `ZeroMqProxy` to destroy an internal executor service
* Add JavaDocs to `ZeroMqChannel`
* Add one more `ZeroMqChannel` to TCP test to be sure that proxy distribution works well

* * Add `hamcrest-core` dependency for Awatility

* * Add more JavaDocs to `ZeroMqProxy` and `ZeroMqChannel`
* Expose `ZeroMqChannel.setZeroMqProxy()` option for easier
configuration within the same application context
* Make `ZeroMqChannel` sockets configuration and connection
dependant on provided `ZeroMqProxy` (if any)
* Add `Consumer<ZMQ.Socket>` configuration callbacks to the `ZeroMqChannel`
* Expose `ZeroMqChannel.consumeDelay` option

* * Add docs for ZeroMQ
* Some additions into a `reactive-streams.adoc`
* Fix typo in the `xmpp.adoc`

* * Add `optional` `jackson-databind` since `ZeroMqChannel` uses it by default
* More words into docs

* * Fix language in docs according review

* Fix language in docs according review

Co-authored-by: Gary Russell <grussell@vmware.com>

* Apply suggestions from code review

Co-authored-by: Oliver <oli-ver@users.noreply.github.com>

* * Fix threading using a `publishOn()` for specific scheduler after `cache()`

* * Remove unused import

* * Change proxy port check from static `Mono.just()` to `Mono.fromCallable()`
to really evaluate the current port state on every repeat
* Add finite `100` repeat number to avoid infinite blocking when proxy is not started at all
* Add `doOnError()` for proxy `Mono` to log `ERROR` when repeat is exhausted

* * Fix Checkstyle violation

Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Oliver <oli-ver@users.noreply.github.com>
2020-08-11 15:04:02 -04:00
Artem Bilan
217e43b194 Add NOSONAR to IntReactiveUtils#emitNext() loop 2020-08-11 09:54:34 -04:00
Attoumane
3cb13af813 More tests for Redis Stream support
Related to https://github.com/spring-projects/spring-integration/issues/3226
2020-08-11 09:30:09 -04:00
Artem Bilan
60f3273bb1 Add diagnostics into StompServerIntegrationTests
Add `n` number for iteration to avoid infinity wait
2020-08-10 14:57:48 -04:00
Artem Bilan
a10e317696 Fix unused import
* Fix new Sonar smells
2020-08-10 14:39:26 -04:00
rohanmukesh12
b54085c99d INT-4566: UPDATE for R2DBC In Channel Adapter
JIRA: https://jira.spring.io/browse/INT-4566

* Rework UPDATE logic according deprecations
* Use `ColumnMapRowMapper` for default `Map` payload type
* Clean up tests
2020-08-10 14:12:28 -04:00
Artem Bilan
17fc4eae99 Fix IntReactiveUtils for the proper emission
* Enable `MessageChannelReactiveUtilsTests.testOverproducingWithSubscribableChannel()` back
* Disable `StompServerIntegrationTests` again since build on CI is stalled again
2020-08-10 12:32:47 -04:00
Artem Bilan
d14c5d5841 Fix Checkstyle violation 2020-08-10 12:10:25 -04:00
Artem Bilan
a180ca8c5f MessagingGW: replace MonoProcessor with Sinks.One
The `MonoProcessor.create()` is deprecated in the Reactor in favor of `Sinks.one()`
2020-08-10 11:58:34 -04:00
Gary Russell
0bae11c60e GH-3359: Add Timer to MessagingGatewaySupport
Resolves https://github.com/spring-projects/spring-integration/issues/3359

- replace removed simple counter with a `Timer`
- register DSL GPFB as beans so they can be discovered for wiring metrics

* Apply suggestions from code review
* Remove left over deprecation suppression
* Fix import
2020-08-10 10:46:41 -04:00
Artem Bilan
fcf7f761a5 Enable STOMP test back to check the solution
* Remove more `ManageableSmartLifecycle` from non-channel adapter components
2020-08-09 08:11:15 -04:00
Artem Bilan
8e5bd4cc80 Disable STOMP test until the fix reactor-netty 2020-08-08 14:28:30 -04:00
Artem Bilan
da7fc4d812 Fix STOMP integration test to close stompClient
The `ReactorNettyTcpStompClient` is based on the `ReactorNettyTcpClient`
which uses a `Schedulers.newParallel()` which has to be shut downed in the end
* Migrate all the STOMP tests to JUnit 5
2020-08-08 14:12:54 -04:00
Oliver
5e74c9c82d Update local maven install in Readme.md
The task `install` does not exist, `./gradlew publishToMavenLocal` needs to be used instead
2020-08-08 13:39:09 -04:00
Artem Bilan
f9fe988556 Downgrade Lettuce version to 5.3.2 2020-08-08 13:38:37 -04:00
Artem Bilan
a49a8848cb Revert some dependencies to be sure in build 2020-08-08 13:05:03 -04:00
Artem Bilan
9d557426b5 * Fix new Sonar smells
* Remove redundant `@SuppressWarnings("deprecation")`
* Add `Duration.ofSeconds(10)` to `StepVerifier.verify()`
to avoid infinite wait and lose failing text context on the hang CI build
2020-08-08 12:29:24 -04:00
Artem Bilan
65ad76232e Fix Checkstyle issues; upgrade some dependencies 2020-08-07 16:40:49 -04:00
Gary Russell
3de0445aaa GH-3358: Kafka IB Gateway - implement Pausable
Resolves https://github.com/spring-projects/spring-integration/issues/3358

Also add `isPaused()` to `Pausable`.

* Fix `@since` to the proper version
2020-08-07 15:26:57 -04:00
Attoumane Ahamadi
4ae6b52c00 GH-3226: Add ReactiveRedisStreamMessageProducer
Fixes https://github.com/spring-projects/spring-integration/issues/3226

* Implement a `ReactiveRedisStreamMessageProducer` to consume Redis streams
* Add support for consumer groups and group auto-creation
* Add `@author` to `RedisHeaders`
* Rename test methods in the `ReactiveRedisStreamMessageHandlerTests` to project code style
* Implement `hashCode()` & `equals()` for `Person` & `Address` testing models
* Fix `ReactiveRedisStreamMessageProducerTests` not creating bean for `ConnectionFactory`.
Otherwise it caused to create one more not controlled `RedisClient`
* Code style clean up in the `ReactiveRedisStreamMessageProducer`
* Implement a group creation logic as a reactive stream deferring the call until
a subscription happens on the final `messageFlux`
* Move a common code for message building as the last `map()` operator in the final `Flux`
* Remove an `IntegrationFlow` definition in the `ReactiveRedisStreamMessageProducerTests`
as redundant
2020-08-07 15:23:20 -04:00
Gary Russell
43950d16ae Restore and deprecate @IntManConf properties 2020-08-07 13:40:57 -04:00
Gary Russell
1beb854fb4 Remove Legacy Metrics
- Simplify MBeans - instead of wrapping to expose lifecycle methods,
  implement `ManageableLifecycle`. Register an additional MBean for
  polled endpoints to control the lifecycle.

* Polishing

- Move `QueueChannel` `@ManagedAttribute`s to `QueueChannelOperations`
- Make all `AbstractEndpoints` `IntegrationManagedResource`s and remove `ManagedEndpoint`
  to allow exposure of any `@Managed*` methods (including those on `Pausable`)
- Revert to `Lifecycle` for classes that are not related to endpoints
- Remove legacy metrics from docs
2020-08-07 12:56:57 -04:00
Gary Russell
da5d002d64 AMQP: Batch mode option to receive element headers
In preparation for https://github.com/spring-cloud/spring-cloud-stream-binder-rabbit/issues/290

SCSt is not set up to receive a `Message<Message<?>>`.

Similar to the kafka endpoint, add the batched headers in a header.
2020-08-06 20:39:56 -04:00
Artem Bilan
094eb2e021 Fix new Sonar smells 2020-08-06 10:45:44 -04:00
Artem Bilan
9989344c0b Remove usage of removed SF deprecated classes
Related to https://github.com/spring-projects/spring-framework/issues/25548
2020-08-06 10:21:15 -04:00
Artem Bilan
cd63763bb4 Fix compatibility with Reactor & Spring Data 2020-08-05 15:33:24 -04:00
Gary Russell
acdc25a172 Upgrade to spring-kafka 2.6.0 2020-08-04 12:14:16 -04:00
Gary Russell
0e2652fdf6 More Cron test fixes 2020-08-03 16:51:55 -04:00
Gary Russell
2356324e9b Fix for internal changes to CronTrigger 2020-08-03 16:30:27 -04:00
Artem Bilan
f26d4c0a5a Fix FtpServerOutboundTests race condition
The `ArrayList` cannot be used in the test-case for checking
its size because the property is not `volatile`: even if
content of the list is what we expect (test logs confirms that),
the non-`volatile` property doesn't give us an actual value

* Use `CopyOnWriteArrayList` for this concurrent test to
be sure that internal `array.length` reflects the reality
after each interaction with the list
2020-08-03 10:06:37 -04:00
Artem Bilan
a62d75f1f0 Fix Reactor Netty dependencies 2020-08-02 09:30:57 -04:00
Artem Bilan
78bb84d07b Disable WebFluxDslTests till Security fixes
Related to https://github.com/spring-projects/spring-security/issues/8881
2020-07-31 11:00:27 -04:00
Gary Russell
8cf6ff0462 AMQP: Support CorrelationData Message Headers
In preparation for https://github.com/spring-cloud/spring-cloud-stream-binder-rabbit/issues/303

Now that `CorrelationData` has a `Future<?>`, users might simply add
correlation data in a header and not receive confirm/return messages.

- No longer require channels for returns and confirms
- don't build the confirm message if there are no channels
- reduce the log level for no channels to DEBUG
- complete the user's future when a message is returned (async GW)

* Fix typo.
2020-07-31 10:00:18 -04:00
Artem Bilan
f5522949f3 WebFluxDslTests: wait for reply for 10 seconds 2020-07-31 09:59:09 -04:00
Artem Bilan
5e01ec1c9c Use SNAPSHOT for Reactor dependencies 2020-07-31 09:07:56 -04:00
Artem Bilan
7ce020dfa7 Upgrade Gradle and plugins 2020-07-30 16:01:31 -04:00
Artem Bilan
564617bff0 Fix id for section in mail.adoc 2020-07-30 14:15:26 -04:00
Artem Bilan
ea546c52d0 Fix compatibility with latest Spring AMQP (#3352)
* Fix compatibility with latest Spring AMQP

* * Fix typo in the `amqp.adoc`
2020-07-30 11:00:28 -04:00
Michel Jung
ba5c80aff0 Fix wording in meter descriptions
Related to https://github.com/spring-projects/spring-integration/pull/3349
2020-07-29 09:57:35 -04:00
Artem Bilan
f23f3ea63d Add gauges for queue channel size (#3349)
* Add gauges for queue channel size

The `QueueChannel` provides a current size and remaining capacity metrics

* Add Micrometer gauges into `QueueChannel` to expose the current values
of the size and remaining capacity

**Cherry-pick to 5.3.x, 5.2.x & 5.1.x**

* * Revert `@SuppressWarnings("unchecked")` for test
* Document new gauges for queue channel

* * Fix IntegrationManagementConfigurer for NPE on `metricsCaptor`

* Fix wording in meter descriptions

Co-authored-by: Michel Jung <michel.jung89@gmail.com>

Co-authored-by: Michel Jung <michel.jung89@gmail.com>
2020-07-28 16:59:35 -04:00
Artem Bilan
818f5a5efe Synchronize list of events for FTP server tests 2020-07-27 09:59:24 -04:00
Artem Bilan
92410b942c Deprecate RMI module
* Remove tests from SI-RMI since the module is deprecated
* Mention such a deprecation in the docs
2020-07-24 11:03:39 -04:00
Artem Bilan
5ec71d4b4a Improve MessageKryoRegistrar for registrations 2020-07-22 13:36:17 -04:00
Artem Bilan
5e45b09cd5 Fix typo in KryoCodecTests 2020-07-22 12:33:24 -04:00
Artem Bilan
6a02b5abe9 Improve Kryo Codec for registrations 2020-07-22 12:07:05 -04:00
Artem Bilan
064cb57ac0 Fix new Sonar smells 2020-07-22 11:32:53 -04:00
Artem Bilan
21dcdbce6d Upgrade to MongoDb driver 4.1.0-rc0 2020-07-22 11:28:13 -04:00
Artem Bilan
730c157064 Introduce ClientRSocketConnector.getRequester()
* Restore `Mono<RSocketRequester> getRSocketRequester()` on
`ClientRSocketConnector` and deprecate it in favor of newly introduced
`RSocketRequester getRequester()`: there is no need in wrapping a
`RSocketRequester` to deferred `Mono` since internal logic of the
`RSocketRequester` on client side is based on lazy-load API of the
`RSocketClient`
2020-07-22 10:25:47 -04:00
Artem Bilan
2c42c9181d Direct access to requester from ClientRSConnector
Since `RSocketRequester` is now lazy load on client side
there is no need to wrap it into a `Mono`.

* Change `getRSocketRequester()` to a plain getter

If there is a requirement to force connect to the server for receiving
requests from there, the `ClientRSocketConnector.connect()` should be used
2020-07-22 09:24:05 -04:00
Artem Bilan
ff44382d94 Use RSocketRequester.rsocketClient() to connect
Related to https://github.com/spring-projects/spring-framework/issues/25332

* Delegate subscription (and therefore auto-connection) into underlying `RSocketClient`
in the `ClientRSocketConnector`
2020-07-22 08:59:00 -04:00