https://build.spring.io/browse/INT-MASTER-1042
The `verifyShutdownWithWait()` test uses extra thread for waiting for
the `TaskScheduler` to finish its tasks.
Also the test doesn't verify the actual behavior at all
* Move `awaitTermination()` to the assert
* Decrease `delay` for the message
* Assert the latch from the handler to be sure that message is scheduled
properly even if `destroy()` call, thanks to required
`waitForTasksToCompleteOnShutdown = true`
* INT-4397: Fix headers filtering for @Transformer
JIRA: https://jira.spring.io/browse/INT-4397
The `AbstractMessageProcessingTransformer` doesn't honor a configured
`notPropagatedHeaders` and copies all the request headers to the
message to return
* Add `setNotPropagatedHeaders()` into the `AbstractMessageProcessingTransformer`
and implement there a logic to filter headers, similar to what we have
in the `AbstractMessageProducingHandler`
* Overrider `updateNotPropagatedHeaders()` in the `MessageTransformingHandler`
to propagate `notPropagatedHeaders` to the `AbstractMessageProcessingTransformer`
delegate
* * Revert `final` for the `AbstractMessageProducingHandler.updateNotPropagatedHeaders()`
* Override `addNotPropagatedHeaders()` for the `MessageTransformingHandler()`
and populate `notPropagatedHeaders` into the target `AbstractMessageProcessingTransformer`
from there
* Also populate `notPropagatedHeaders` from the `AbstractMessageProcessingTransformer.doInit()`
* Implement a `AbstractIntegrationMessageBuilder.filterAndCopyHeadersIfAbsent()`
for a general logic to filter `notPropagatedHeaders` and copy the result
headers set into the target message if they are absent
* Use an new `filterAndCopyHeadersIfAbsent()` in the `AbstractMessageProducingHandler`
and `AbstractMessageProcessingTransformer`to avoid code block duplication
https://build.spring.io/browse/INT-FATS5IC-509
Looks like `busyWait` period as `5 secs` is too aggressive and may be
two or more cycles pass until the election happens.
At this time the waiting for the `granted` latch in `10 secs`
may expire already, therefore we fail in the assertion
* Decrease `busyWait` to the `1 secs` to let the election cycle to pass
quickly
* Increase latch wait time to the `20 secs`
**Cherry-pick to 5.0.x**
https://build.spring.io/browse/INT-MASTER-1038
When we don't provide an `output-processor` for the `<barrier>`
definition, the new logic in the `IntegrationNamespaceUtils.constructAdapter()`
ends up with the `null` injection into the `BarrierMessageHandler` ctor.
From here there is no guarantee which ctor will be selected:
```
BarrierMessageHandler(long timeout, MessageGroupProcessor outputProcessor)
...
BarrierMessageHandler(long timeout, CorrelationStrategy correlationStrategy)
```
From reflection perspective they both are equal and there is no predictable
outcome which is is going to be selected.
Looks like on Windows and OSX, the second (expected) is selected, but on
Linux it is the first one.
* Fix `IntegrationNamespaceUtils.injectCtorWithAdapter()` do not inject
`adapter` in to the target ctor if it is `null`.
This way the `BarrierMessageHandler(long timeout)` ctor is selected
without any ambiguity
* INT-4389: Default to SimpleSeqSizeReleaseStrategy
JIRA: https://jira.spring.io/browse/INT-4389
An `AbstractCorrelatingMessageHandler` fallback to the
`SimpleSequenceSizeReleaseStrategy` when `releaseStrategy` isn't
provided.
* Make `ReleaseStrategyFactoryBean` to fallback to the
`SimpleSequenceSizeReleaseStrategy` for consistency.
* * Fix `ResequencerParser` to fallback to the null `releaseStrategy`.
This way a subsequent `releasePartialSequences` will set a
`SequenceSizeReleaseStrategy` as a default one
* * Fix `BarrierMessageHandler` to populate a default `CorrelationStrategy`
* * More polishing to `BarrierMessageHandler`
* INT-4398: Replace Gemfire dependency with Geode
JIRA: https://jira.spring.io/browse/INT-4398
* Polishing `gemfire.adoc` and mention the procedure for an
exclusion/inclusion appropriate dependency
* More Docs polishing
* * Fix tests to be based on SD Geode XML namespace
* Fix Docs according PR comments
* Polishing Docs according PR comments
JIRA: https://jira.spring.io/browse/INT-4466
When we schedule group for force complete in the
`AbstractCorrelatingMessageHandler`, we don't track a group `timestamp`
and its `lastModified` before the scheduled task.
This way, in the cluster environment, we may schedule several tasks
for different messages and the first started may release the group too
early.
Just because we extract a `MessageGroup` from the store already in the
task per se.
* Propagate the actual `timestamp` and `lastModified` from group before
scheduling task.
Compare these value with the actual group metadata in the
`processForceRelease()` before performing real `forceRelease()`.
This way we restore behavior before fixing memory leak, when we
propagated full `MessageGroup` to the scheduled task
* Implement `ResequencingMessageHandler.getComponentType()` for
consistency
* Remove unnecessary overhead with the `volatile` on many
`AbstractCorrelatingMessageHandler` properties, which hardly ever can be
changed at runtime
**Cherry-pick to 5.0.x and 4.3.x**
JIRA: https://jira.spring.io/browse/INT-4467
The package tangles is caused by the explicit classes declarations.
* Since we use only method definitions from those classes, there is just
enough to use class names and use reflection to get class objects and
then get methods from them
_The fix is fully compatible for back-porting_
**Cherry-pick to 5.0.x**
* Wrap non-`StandardIntegrationFlow` into Proxy
In previous version all the `IntegrationFlow` beans have been replaced
by the `StandardIntegrationFlow` i the `IntegrationFlowBeanPostProcessor`
if they are lambda.
That works for Java, but doesn't with Kotlin, since lambdas i last one
are not synthetic classes.
Therefore some Java DSL definitions (especially `.subFlowMapping()`)
don't work consistently in two languages.
* Introduce `IntegrationFlowLifecycleAdvice` to wrap all the
non-`StandardIntegrationFlow`s (excluding `IntegrationFlowAdapter`)
into the `Proxy` to expose `SmartLifecycle` and `getInputChannel()`
operations and delegate them to the internal `StandardIntegrationFlow`
created by the `IntegrationFlowBeanPostProcessor`.
* This way any custom `IntegrationFlow` implementations can be used
for manual flow registration via `IntegrationFlowContext`
* Polish `RouterDslTests.kt` for the `@Bean`s for sub-flows.
* Document in the `dsl.adoc` a request-reply approach for the case
when `.subFlowMapping()` refers to an `IntegrationFlow` `@Bean`.
* Polishing for the `FlowServiceTests` since all the
non-`StandardIntegrationFlow`s and not-`IntegrationFlowAdapter`s are
wrapped now to the Proxy.
* Add missing `from()` delegations into the `IntegrationFlowAdapter`
* Polishing for the `ManualFlowTests` since all the `IntegrationFlow`
now are `Lifecycle` after wrapping to the Proxy.
* Add JavaDocs to the IntegrationFlowLifecycleAdvice and polishing for the dsl.adoc
* * Add JavaDocs for the `IntegrationFlowBeanPostProcessor.processIntegrationFlowImpl()`
* Improve JavaDoc for the `RouterSpec.subFlowMapping()`
* Assert in the `FlowServiceTests` that proxied custom flow implements
all the expected interfaces
JIRA: https://jira.spring.io/browse/INT-4465
There is a one second delay before a socket close is propagated if there is an active
assembler. This is generally only a problem with deserializers that use EOF to signal
message end (such as the `ByteArrayElasticRawDeserializer`).
Attempt to insert an EOF marker into the buffer queue so that the `getNextBuffer()` will
exit immediately on `close()` if it is blocked awaiting a buffer.
**cherry-pick to 5.0.x, 4.3.x**
https://build.spring.io/browse/INT-SI50X-JOB1-56
We can't wait for the latch in the interruptable code flow;
we can't have a round-robing election guarantees.
* Add `Thread.sleep(LockRegistryLeaderInitiator.this.busyWaitMillis)`
to the `LockRegistryLeaderInitiator` when we restart the main task
* Remove latches waiting and thread shifting from the
`RedisLockRegistryLeaderInitiatorTests`
* Use long `busyWaitMillis` for yielding initiator to let the second
candidate to be elected
**Cherry-pick to 5.0.x**
https://build.spring.io/browse/INT-FATS5IC-501/
When we try to wait for the `Latch` in the interruptable code flow,
it is a fact that we step away from the waiting and end up with the
race condition downstream.
* Wrap `Latch` in the interruptable `publishOnRevoked()` code to the
`Executor.execute()`
* Remove `deleteTimeoutMillis` option from the `RedisLockRegistry`
since it doesn't make sense in the interruptable code.
* Add `RedisLockRegistry.setExecutor()` to allow to inject an external
`Executor`
* Add more debug logging into the `LockRegistryLeaderInitiator`
**Cherry-pick to 5.0.x**
https://build.spring.io/browse/INT-MASTER-1024
When the Lettuce Redis client catches an `InterruptedException`, it is
wrapped to the `RedisCommandInterruptedException`, therefore when we
catch an exception on our code level it is not an `InterruptedException`
anymore and we can't proceed in the loop because the tread is
`interrupted` already.
* Check the `interrupted` alongside with the `InterruptedException`
to restart a loop from a fresh thread.
**Cherry-pick to 5.0.x**
JIRA: https://jira.spring.io/browse/INT-4462
When the HTTP request body is empty, the `HttpMessageReader` ends up
with the empty `Mono` which can't be evaluated to any reasonable value.
* Add fallback to `requestParams` when `Mono` for body is empty and
also when `payloadExpression` returns null
**Cherry-pick to 5.0.x**
JIRA: https://jira.spring.io/browse/INT-4463
Certain options, such as `maxInFlight` were not exposed.
Deprecate the setters on the factory and allow the user to inject a pre-configured
`MqttConnectOptions`, thus making all (and any new) properties available to be
configured.
JIRA: https://jira.spring.io/browse/INT-4464
A `ZookeeperMetadataStore.get()` is based on the `this.cache` variable.
This one is initialized in the `start()`.
* Assert `isRunning()` in the `get()` before using.
**Cherry-pick to 5.0.x and 4.3.x**
* Use Lettuce Redis client; Redis module fixes
The Lettuce client is based on the Netty and more stable, than Jedis
therefore we get a performance improvement for tests
(it saves us at about 30 seconds).
Also this client doesn't fail for me on Windows sporadically (very often)
with the `ConnectionClosedException`
* After switching to the Netty-based client, we expose the interrupted
Thread issue in the `LockRegistryLeaderInitiator`.
If we interrupted (expected behavior), we try to unlock calling
`RedisLockRegistry`, but Netty client reject our request because the
thread is interrupted, therefore we never delete the lock when we yield
our leadership.
Fix the issue with shifting a `RedisTemplate.delete()` operation to the
`ExecutorService` when the current thread is interrupted
* Allow to configure such an `ExecutorService` and timeout to wait for
the `submit()` result
* Refactor `RedisAvailableRule` and all the Redis tests do not expose
the target `RedisConnectionFactory` implementation.
* Make all the test-cases based on the `connectionFactory` created by
the `RedisAvailableTests.setupConnectionFactory()`
* Tweak some unnecessary timeouts and sleeps for better tests task
throughput
* Add a `Log4j2LevelAdjuster` into the `RedisLockRegistryLeaderInitiatorTests`
https://build.spring.io/browse/INT-FATS5IC-497
Fixes `JdbcMessageStoreTests` sporadic failure where there is no
guarantee that message are going to be polled from the group the same
order they've been inserted
Cherry-pick to `5.0.x`
https://build.spring.io/browse/INT-MASTER-1017
Looks like non-`volatile` `size` property in the `LinkedList` is not
updated properly for the `contains()` assertion, therefore we end up
with the `ConcurrentModificationException`
* Move `assertThat()` into the `synchronized (overridePresent)` block
to enforce CPU cache flushing and, therefore, have an actual value
for the `LinkedList.size` during iteration
JIRA: https://jira.spring.io/browse/INT-4461
Convert `byte[]` to `String` using `URF-8` by default.
Add optional evaluate methods to JsonPathUtils with a Charset to use when
converting `byte[]` to `String`.
This is not currently exposed using SpEL. It can be done, but probably not worth
the effort until somebody asks for it.
**cherry-pick to 5.0.x, 4.3.x**
* Use `BAIS`
For better back-pressure handling do not perform active operations
directly on the current `Flux` (e.g. via `doOnNext()`).
It's better to postpone such a handling to back-pressure ready operators
Like in this `FluxMessageChannel` case, the `.handle()` operator is
wrapping the "hard" `send()` operation.
Also include `.errorStrategyContinue()` do not stop during message
processing
* INT-4458: Do not expose recursive generics API
JIRA: https://jira.spring.io/browse/INT-4458
According Kotlin generics system restrictions we can't expose API
based on the recursive generics, like we have with Java DSL for JMS
* Replace explicit recursive generics on factory method with the wildcard (`?`).
This way both Kotlin and Java are able to instantiate target object and perform
the proper chain API auto-completion
* Fix `JmsInboundChannelAdapterSpec` to deal with provided `S` type for the
`configureListenerContainer()` - the way we can configure a `JmsDefaultListenerContainerSpec`
* Add `kotlin-spring` Gradle plugin to avoid extra `open` modificator on `@Bean` methods
* Add `JmsDslKotlinTests.kt`
**Cherry-pick to 5.0.x excluding Kotlin support**
* Fix `Amqp` DSL factory for recursive generics
* Polishing for Kotlin tests
We don't change JDBC scripts too often and even if we do that, it's not
so hard to copy/paste the script throughout all the supported platforms
instead of rely on the pretty old VPP tool and suffer from performance
degradation during regular day-to-day development.
Plus there is some bug in the VPP when it generates a script for the
Derby, so we end up with erroneous braces in the sentence:
```
MESSAGE_SEQUENCE BIGINT NOT NULL GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1),
```
I have to remove them manually each time I switch between Gradle versions
JIRA: https://jira.spring.io/browse/INT-4456
Since Kotlin compiles lambdas different way than Java, they are not
synthetic classes at runtime anymore.
Therefore we fallback to the method invocation logic, but since Java 8
interfaces has `default` methods as well the `MessagingMethodInvokerHelper`
can't select the target method for invocation
* Use explicit `Function.apply()` for non-synthetic implementations
* Add `RouterDslTests` Kotlin-based test-case
**Cherry-pick to 5.0.x**
* Add more explicit methods for invokers
The `transactionWithCommitAndAdvices()` uses `SimpleRepeatAdvice`
which essentially performs `doPoll()` twice.
In this case we don't have enough messages in the `goodInputWithAdvice`
queue
* Add one more message to the `goodInputWithAdvice` queue during testing
* Increase all the timeouts
* Increase capacity in queues for the `transactionWithCommitAndAdvices()`
**Cherry-pick to 5.0.x and 4.3.x**
JIRA: https://jira.spring.io/browse/INT-4351
The `WatchService` reacts to the events in the file system and keep
track ove the events until we poll them, e.g. via `listEligibleFiles()`
in the `WatchServiceDirectoryScanner`.
On the other hand the `SourcePollingChannelAdapter` calls the mentioned
`listEligibleFiles()` according its polling period.
At this moment the `FileListFilter` is applied to the polled files.
It may happen that `LastModifiedFileListFilter` can't accept too young
files yet and they are lost for the future consideration.
* To allow, for example, to retain young files by the
`LastModifiedFileListFilter` judgment for the future cycles add
`DiscardAwareFileListFilter` with the `DiscardCallback` support.
The `WatchServiceDirectoryScanner` now registers such a callback into
the filter and stores discarded files into the `filesToPoll` queue
for the future poll cycle.
**Cherry-pick to 5.0**
Fix compilation warnings
* Replace `DiscardCallback` with the plain `Consumer`
* Ensure uniqueness in the `WatchServiceDirectoryScanner` internal
queue via a `Set` implementation, since discard callback may be called
several times for the same file from the `CompositeFileListFilter`
according to its nature
* Add JavaDocs and Docs
Change `@since` to `5.0.5`
JIRA: https://jira.spring.io/browse/INT-4455
When using java config, `setChannelMapping` NPEs due to no AC.
**cherry-pick to 4.3.x, and to master, removing AC check**
* Polishing - PR comment
* Resolve TODO in the `ErrorMessageExceptionTypeRouter`
Looks like Derby is much slower than H2, plus it doesn't look so stable
having so much sporadic failures.
Looks like moving to H2 here saves for us `500 ms` test working time
**Cherry-pick to 5.0.x**
* Upgrade to Gradle 4.7 and others, polishing build
* Fix `MqttAdapterTests` according requirement of the latest Mockito
* Move `int-derby.properties` to the `/resources` from `/java`
* Fix Checkstyle violations
The message in the `QueueChannel` appears for consuming a bit earlier
than TX is committed
* Introduce `afterCommitLatch` into the test verify the state
when TX is really committed and data is removed from DB
**Cherry-pick to 5.0.x and 4.3.x**
Looks like we have started to fail much often
since upgrade to the `Derby-10.14.1.0`:
cb0d43db6b
That is also related to this https://jira.spring.io/browse/INT-4445
* Remove all the `drop` mentioning for Derby scripts -
it just doesn't support `IF EXISTS`.
More over we don't need to worry about drops since each test now starts
its own fresh embedded DB
**Cherry-pick to 5.0.x**
* INT-4446 Improve EmbeddedJsonHeadersMessageMapper
JIRA: https://jira.spring.io/browse/INT-4446
* Do not recreate message if not necessarily
* Do not let to generate `id` and `timestamp` if they are not mapped
* Use `smartMatch` to allow to configure negative patterns
* Introduce `PatternMatchUtils.smartMatchIgnoreCase()` for convenience
**Cherry-pick to 5.0.x**
* * Polishing `EmbeddedJsonHeadersMessageMapper`
JIRA: https://jira.spring.io/browse/INT-4447
When we get an exception during `this.lock.unlock()`, we don't revoke
leadership.
In case of external resource (e.g. JDBC) this may cause a race condition
when the second candidate is selected as leader when connection comes back
* Catch `this.lock.unlock()` exceptions and log them under DEBUG.
This way we proceed to the `handleRevoked()` logic
**Cherry-pick to 5.0.x and 4.3.x**
https://build.spring.io/browse/INT-MASTER-999/
Since `BeanFactoryTypeConverterTests` and `CallerBlocksPolicyTests`
uses too long `Thread.sleep()` there is no guarantee that they are going
to be performed after expected 10 seconds
* Decrease `Thread.sleep()` in those tests
* Increase timeouts in the `RoundRobinDispatcherConcurrentTests` and
`ManualFlowTests`
* Fix `PollerAdviceTests` to re-use `TaskScheduler` from the ctx
instead of local, not closed instance
* Use `OnlyOnceTrigger` instead of local implementations
* Change the `primary` `Trigger` bean to the `PeriodicTrigger` as well.
The minimum interval for the `CronTrigger` is 1 seconds - it doesn't
matter for this test-case
JIRA: https://jira.spring.io/browse/INT-4445
Looks like `/dataSource` temporary directory is busy in between different
tests.
The thought is like a high-frequently polling endpoint keeps DB resource
from removing.
* Explicitly `stop()` endpoint in the `JdbcMessageStoreChannelTests`
* Optimize
`JdbcMessageStoreChannelTests.testSendAndActivateTransactionalSend()`
to rely on the short `1` millisecond TX timeout.
This safes for us 10 seconds of the tests executions.
* Configure all the embedded DB beans for the `ignore-failures="ALL"`
as a fallback option if polling endpoint is not a cause of the concurrent
resource access.
**Cherry-pick to 5.0.x**
JIRA: https://jira.spring.io/browse/INT-4438
The `SmartLifecycleRoleController` is based on the `MultiValueMap`
which used internally a `List` for the values.
With such an architecture we can add the same value several times.
On the other hand we are iterating over `Lifecycle`s in the role and
build a `Map` for their running status.
In this case when `NamesComponent`s return the same name the Java
`Collectors.toMap()` fails with a duplicate key error.
In any cases it would be better do not allow to add the same lifecylce
several time to the role or different with the same name.
* Add search logic to the `addLifecycleToRole()` to fail fast with the
`IllegalArgumentException` because a lifecycle with the same name is
already present in the role
**Cherry-pick to 5.0.x**
* Remove redundant `this.initialized = false` from the
`AbstractPollingEndpoint.doStop()`
Add `allEndpointsRunning()` verification to the `EndpointRoleParserTests`
Polishing
