JIRA: https://jira.spring.io/browse/INT-4290
See CVE-2017-4995
To disallow deserialization of unknown classes,
the `JacksonJsonUtils#messagingAwareMapper()` can now be supplied
with the `trustedPackages`.
The default list is:
```
java.util
java.lang
org.springframework.messaging.support
org.springframework.integration.support
org.springframework.integration.message
org.springframework.integration.store
```
Can be configured with `*` (asterisk) with meaning trust all
**Cherry-pick to 4.3.x**
Polishing according PR comments
JIRA: https://jira.spring.io/browse/INT-4267Fixes: spring-projects/spring-integration#2110
The documentation clearly point that we can simply use JSON (de)serialization
with the `RedisMessageStore`, but actually it isn't so easy
* Fix `MessageGroupMetadata`, `MessageHolder`, `MessageMetadata` for Jackson
deserialization compatibility
* Add `MessageHeaders`-based ctor to the `AdviceMessage`
* Add `MessageHeadersJacksonSerializer` to serialize `MessageHeaders`
to the `HashMap` for easier deserialization afterwards
* Add deserializer implementations for all `Message` types
* Add convenient `JsonObjectMapperProvider#jacksonMessageAwareMapper()`
factory method to build `ObjectMapper` supplied with mentioned above
(de)serializers
**Cherry-pick to 4.3.10 without `MessageHolder` and `MessageMetadata`**
Address PR comments and document the feature
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4258
Since Redis `ZADD` command doesn't perform incrementation by default,
it should be consistent in the `RedisStoreWritingMessageHandler` as well
* Make a default behavior of the `RedisStoreWritingMessageHandler`
regarding `INCR` option as `false`
* Provide some performance refactoring for the
`RedisStoreWritingMessageHandler` moving the code to Java 8 style
and using `FunctionExpression`
* Fix tests according to the changed default behavior
* Add `setZsetIncrementExpression` and String-based setters for other
expressions on the `RedisStoreWritingMessageHandler`
* Add `zset-increment-expression` option for XML configuration
* Add `What's New` note on the matter
Polishing
This commit harmonizes `LockRegistry#obtain` logic in Zookeeper and JDBC
implementations with Redis implementation
by using Java 8 `ConcurrentMap#computeIfAbsent`
* Remove `synchronized (this.locks)` from the `expireUnusedOlderThan`
implementations because `iterator()` is thread-safe from `ConcurrentHashMap`
* Fix deprecation in the `IntegrationGraphControllerRegistrar`
* Revert Spring Security version to `4.2.2`, since `5.0 B-S` is broken
JIRA: https://jira.spring.io/browse/INT-4248
To avoid unexpected double locking behavior in the cluster,
remove the local cache functionality.
Now with the new `clientId` property, the `expire` for the record in
store is always update on each lock operation
JIRA: https://jira.spring.io/browse/INT-4244
The `JpaOutboundGateway` returns a whole `Message` for reply copying request headers.
Building the reply a `AbstractMessageProducingHandler` uses `MessageBuilder` to create a new `Message` with all request headers because of `shouldCopyRequestHeaders() == true`
* To avoid some overhead with new `Message` and `MessageBuilder` for nothing, just return a JPA result as is and let super class to build the Message
* Optimize all reply producers to return `MessageBuilder` to the super `handleMessage()` process
* Return the whole `gatherResult` `Message<?>` from the `ScatterGatherHandler` to have ability to transfer reply headers as well
* Mark `DelayHandler` as `shouldCopyRequestHeaders() return false` since it doesn't modify Message and returns it as is after delay
JIRA: https://jira.spring.io/browse/INT-4243
Remove newline char from key exchange.
Fix CR and Add Windows Delete Diagnostics
Fix For Windows
More diagnostics.
It appears the new server doesn't close the file when the session is closed.
In the streaming test, consume the stream before closing the session.
Polishing - PR Comments
* More polishing according PR comments
JIRA: https://jira.spring.io/browse/INT-4219
- Use `AdditionalAnswers.returnFirstArgument` instead of lambda to return first arg
- Enable CheckStyle `AvoidStaticImport` for tests
- Fix static import violations in tests
JIRA: https://jira.spring.io/browse/INT-4182
To avoid extra allocation for the `SimpleDateFormat` use a new Java 8 `DateTimeFormatter` based on new `Temporal` abstraction
Leave test-cases as is to save some time for other tasks
Fix RedisLockRegistry for the proper formatter and ZoneId
JIRA: https://jira.spring.io/browse/INT-4176
The `DelayerHandlerRescheduleIntegrationTests` doesn't close `context` in the end of test.
That causes clashes for `beanFactoryResolver` in other tests.
When those tests are fail, we come into condition when new processes can't be started/stopped because of effect of non-stopped context
* Fix all the `DelayerHandlerRescheduleIntegrationTests` to stop `context` in the end of test
* Remove unused files in the `gemfire/store` test package
* Add current date value to the `CacheServerProcess` name to avoid clashes with other ran on the same host
* Add NPE check for `region` in the `GemfireMetadataStoreTests`
* Extract `@BeforeClass/@AfterClass` for the `CacheWritingMessageHandlerTests` to start only one Gemfire cache per class
**Cherry-pick to 4.3.x & 4.2.x**
https://sonar.spring.io/component_issues?id=org.springframework.integration%3Aspring-integration%3Amaster#resolved=false|types=BUG
In `IntegrationFlowRegistration` double check locking is ok for `inputChannel`
because we're assigning an existing object, but `MessagingTemplate` constructs
a new object for which double check locking doesn't work.
In any case, for both these items, the chance of concurrent access is extremely low
and is idempotent anyway, so remove double check locking.
Several inner classes can be static.
Other minor fixes.
JIRA: https://jira.spring.io/browse/INT-3770,
https://jira.spring.io/browse/INT-4107
Having `TransactionHandleMessageAdvice` we can start TX from any `MessageHandler.handleMessage()`
* Add `<transactional>` alongside with the `<request-handler-advice-chain>` for those components which produce reply
* Merge `<transactional>` and `<request-handler-advice-chain>` configuration to a single `ManagedList`
* Rework JPA `<transactional>` in favor of common solution
* Some polishing and refactoring
AbstractPollingEndpoint: avoid `new ArrayList` if we don't have `receiveOnlyAdvice`s
Polishing - PR Comments and Closeable Warnings
Eclipse emits bogus warnings with exceptions in lambdas.
Even though the lambda might run on another thread, elipse thinks it could
cause the context to not be closed.
SPR-14854: MessageChannel is now a @FunctionalInterface
* Additional Lambda polishing and some code style fixes
JIRA: https://jira.spring.io/browse/INT-2460,
https://jira.spring.io/browse/INT-4122
Since the main purpose of the `MessageStore` to persist message for durability and only,
it doesn't make sense to modify `Message` for additional headers like `SAVED` and `CREATED_DATE`.
Such a logic should be a part of metadata stored together with the message.
And it is provided by the out-of-the-box `MessageStore` implementation.
In addition we free ourselves from the reflection operations to retain `ID` and `TIMESTAMP` headers when we add `SAVED` and `CREATED_DATE`
* Control "already saved" logic in the `JdbcMessageStore`s via `DuplicateKeyException` on the `INSERT`.
This is much effective then additional `SELECT` in case of `SAVED` before
* Control "already saved" logic in the `AbstractConfigurableMongoDbMessageStore` via `DuplicateKeyException` on the `INSERT`.
Since `MongoDbMessageStore` doesn't provide extra `messageId` field, perform extra `SELECT` before store document.
Anyway the `MongoDbMessageStore` isn't recommended for use.
We may consider to deprecate it
* Control "already saved" logic in the `AbstractKeyValueMessageStore` via `putIfAbsent` operation
With this fix we persist message in the store as is without any modifications when we perform standard serialization procedure.
Any custom serializers should consider to use `MutableMessageBuilder` if there is a requirement to retain `ID` and `TIMESTAMP`
Rework `MongoDbMetadataStore.putIfAbsent()` to normal `findAndModify()` with particular `$setOnInsert`.
Technically the MongoDB query looks like:
```
db.collection.findAndModify({
query: { _id: $key },
update: {
$setOnInsert: { value: $value } // perform modification only on upsert
},
new: false, // don't return new doc if one is upserted
upsert: true // insert the document if it does not exist
})
```
Move single import to the appropriate JavaDoc
Polishing after rebase
DEBUG messages in `doStoreIfAbsent()` implementations
* To keep track of the extra message information in the `MessageStore`, without `Message` modification, introduce `MessageMetadata` and `MessageHolder`
* Add `MessageStore#getMessageMetadata()`
* Modify MongoDb `MessageStore` to add extra `timestamp` for individual message
* Fix `ConcurrentAggregatorTests` race condition.
Since currently the default release strategy is `SimpleSequenceSizeReleaseStrategy` which is just based on the `MessageGroup` size, there is no guaranty which messages will complete the group in concurrent environment.
The test is really based on the `SequenceAwareMessageGroup` logic to discard the message with the same `correlationId`
Fix `@Copyright` format
Move cast to `MessageHolder` after `Assert.isInstanceOf(MessageHolder.class, messageHolder)`
Retain backward compatibility in the `AbstractKeyValueMessageStore`
Polishing
JIRA: https://jira.spring.io/browse/INT-832
When we scan directory recursively for files (e.g. `WatchServiceDirectoryScanner`), it can be useful to get access to the relative path from the `Message`, e.g. on the `FileWritingMessageHandler` side to restore the original structure
* Add a `FileHeaders.FILENAME` into the outbound `Message` from the `FileReadingMessageSource`
* The result of that header is like a removal of leading `this.directory.getAbsolutePath()` in the target `File.getAbsolutePath()`.
In case of not recursion we get just only regular file name.
* Introduce `FileHeaders.RELATIVE_PATH`
* Populated `FileHeaders.RELATIVE_PATH`, `FileHeaders.FILENAME`, `FileHeaders.ORIGINAL_FILE` in the `FileReadingMessageSource`
* File `FileTailingMessageProducerSupport` to populate `FileHeaders` properly
* Introduce ctor for the `LastModifiedFileListFilter` for better Java configuration experience
* Add docs for changes
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4126
Looks like `WATCH` doesn't provide good isolation when the call is performed concurrently at the same time from different Threads.
* Change locking algorithm as it is recommended by Redis `SET` command: http://redis.io/commands/set:
```
SET resource-name anystring NX EX max-lock-time
```
* Add `@Repeat(10)` to the `AggregatorWithRedisLocksTests#testDistributedAggregator()` since that helped to reproduce the issue
**Cherry-pick to 4.3.x & 4.2.x**
* Upgrade to Checkstyle `7.1`
* Relax `RequireThis` rule a bit. Right now it does the effort only in case of overlapping. See https://github.com/checkstyle/checkstyle/issues/2362 for more info
* Enable some annotation rules and provide fixes for violations
* Enable `tabs indents` rule. This was the biggest fix in this PR
* Resolve `TODO` in the `MessagingMethodInvokerHelper` and fix tests to meet `IllegalStateException` now
Remain Checkstyle version `6.16.1` and reinstate `RequireThis` rule
The latest Checkstyle has a bug with local scope variables if they have the same names as property.
Revert some literals splitting
Fix some line length exceeding and code style
JIRA: https://jira.spring.io/browse/INT-4105
Fixes GH-1888
The lock in Redis can be expired in between `obtain()` calls.
So, even if we return a new lock instance, the old one must clear properly.
* Add `lock.unlock()` to the `obtain()` if the case of expiration in Redis.
* Add `warn` for the exception on the `lock.unlock()`. We don't care about error here and just proceed to a new instance.
**Chery-pick to 4.3.x**
Remove the lock reference from `weakThreadLocks` as well
Add `registry.setUseWeakReferences(true);` to test-case
* Add assert for `lock.unlock()` in case of expiration
* Add `UUID.randomUUID()` for the registry key to avoid cross-talking during concurrent builds, e.g. on the CI server
https://build.spring.io/browse/INT-MASTER-352
This is actually a fix for the
JIRA: https://jira.spring.io/browse/INT-4083
* Make most functional interfaces as `@FunctionalInterface`
* Convert some abstract classes to `@FunctionalInterface` with `default` methods
* Apply Lambda style implementation in some places
* Remove `Function` in favor of similar in Java 8
* Remove redundant code from `DefaultAmqpHeaderMapper` since we are already on Spring AMQP-2.0
* Add several ctors to the `ExpressionEvaluatingMessageListProcessor`
* Populate explicit `Boolean.class` `expectedType` from the `ExpressionEvaluatingReleaseStrategy`
JIRA: https://jira.spring.io/browse/INT-3913
* Remove deprecated classes and methods/constructors, deprecated XML attributes
* Remove `TcpConnectionEventListeningMessageProducer` and rework tests logic to the `ApplicationEventListeningMessageProducer`
* Fix several typos
* Remove/rework deprecated entities mentioning
* Fix several typos in log messages. And some test on the matter as well
* Add comment to `AbstractPersistentAcceptOnceFileListFilter.rollback()` to clarify the reason of `rollingBack` variable
* Make `RemoteFileTemplate.StreamHolder` as `static` to avoid extra internal variable to outer class instance
* Replace `MessagingException` with `AbstractInboundFileSynchronizingMessageSource` in `init()` method of some components. It isn't Messaging yet in that phase
* Fix `SubscribableRedisChannel.MessageListenerDelegate` to handle `Object` not `String`, because with the `serializer` injection there is no guaranty that incoming is always `String`
* Move `JSch.setLogger(new JschLogger());` in the `DefaultSftpSessionFactory` to `static` block. It really should be done only once
* Remove `Assert.isTrue(this.port >= 0)` from the `DefaultSftpSessionFactory`. The subsequant `initJschSession()` convert it to default `22` port
* Change in the `JschProxyFactoryBean` `UnsupportedOperationException` to `IllegalArgumentException`. Wrong enum is wrong argument. That isn't a problem of operation
* Simplify `stop()` in the `CuratorFrameworkFactoryBean` and mark it as a `this.running = false`. Otherwise it wasn't able to be restarted
* Expose `leaderEventPublisher` in the `LeaderInitiatorFactoryBean` and fix `stop(Runnable callback)` with propagation `callback` to delegate.
Fix `SubscribableRedisChannelTests` for new `handleMessage(Object)` signature
JIRA: https://jira.spring.io/browse/INT-1463
An existing `QueueChannel#getRemainingCapacity()` is based on the `BlockingQueue#getRemainingCapacity()` which is always `Integer.MAX_VALUE`
and doesn't reflect reality for provided `capacity`
* Fix `PriorityChannel` to return `upperBound.availablePermits()` for `getRemainingCapacity()`
* Add test for `PriorityChannel` on the matter
* Also increase `reply-timeout` for `<int-redis:queue-outbound-gateway>` in `RedisQueueGatewayIntegrationTests-context.xml` from 1 sec to 10 secs
The `RedisMessageGroupStoreTests.testMessageGroupUpdatedDateChangesWithEachAddedMessage()` can fail
sporadically because there is no guaranty the several calls to the `System.getCurrentTimeMillis()` return the same value.
* Fix `AbstractKeyValueMessageStore` to reuse `MessageGroup.timestamp` for the `lastModified` in case of a new group
https://build.spring.io/browse/INT-MJATS41-629
* Revert to SF and SA BUILD-SNAPSHOTs
* Upgrade to Spring Security 4.1
* Fix `RedisQueueMessageDrivenEndpointTests.testInt3442ProperlyStop()` for `atLeastOnce()` verification
because in between stop and the last `rightPush` there might be one more `pop` and push again because of `listening = false` state
* There is still sporadic `OutboundGatewayFunctionTests.testLazyContainerWithDest()`, so revise all the JMS tests for:
- proper `ApplicationContext.stop()` in the end of test
- `@DirtiesContext` if `SpringJUnit4ClassRunner` is in use
- `TaskScheduler.destroy()` if that
The JMS fix might relate to the https://jira.spring.io/browse/INT-4030, therefore consider this fix for cherry-picking into `4.2.x` (excluding `build.gradle` changes)
JIRA: https://jira.spring.io/browse/INT-3387,
https://jira.spring.io/browse/INT-3806
* Introduce
```
MessageGroupStore
void addMessagesToGroup(Object groupId, Message<?>... messages);
```
And implement it in all stores.
* Use new `addMessagesToGroup` where it is reasonable, e.g. `DelayHandler`
* Optimize test-case to use a new store method (where it is possible)
* Fix timing delays in the `JdbcMessageStoreTests`
* Introduce `PersistentMessageGroup`
* Add `AbstractMessageGroupStore#proxyMessageGroupForLazyLoad` to wrap the raw `MessageGroup` to the `PersistentMessageGroup` for lazy-load
* Rework `MessageGroupMetadata` do not be `immutable` and allow to store/restore in the `AbstractKeyValueMessageStore` only the `MessageGroupMetadata`
* Refactor `ResequencingMessageHandler` and `SequenceSizeReleaseStrategy` a bit for better performance when interact with the `MessageGroup`
* Add `AbstractMessageGroupStore#setLazyLoadMessageGroups` to switch off the `lazy-load` behavior and restore the previous full `MessageGroup` logic
* Add `What's New` note and `message-store.adoc` paragraph for the lazy-load functionality
`GroupType.PERSISTENT` and not lazy by default
PR Comments
Fix `JdbcMessageStoreTests` timing issues
Address PR comments
* Add performance test to the `ConfigurableMongoDbMessageGroupStoreTests`
* Add JavaDocs for the `MessageGroupFactory` methods
* Add `log4j.properties` into the `test` MongoDB module for better traceability
* Fix `JdbcMessageStore#getOneMessageFromGroup()` over the `doPollForMessage()` delegation.
The `jdbcTemplate.queryForObject()` requires exactly one and only one raw in `resultSet`
* Add performance test results into the `message-store.adoc`
JIRA: https://jira.spring.io/browse/INT-3909
Un`@Ignore` the `ClientWebSocketContainerTests` to ensure that `Tomcat-8.0.33` has the fix for the closed WebSocket session race condition.
Fix `StompIntegrationTests` test data
Clean up queue channels before each test, because of some remained extra messages during the web-socket protocol interactions,
like socket close events or similar
Fix `RedisStoreInboundChannelAdapterIntegrationTests` race conditions
Since Jedis driver is async, we not always receive a reply from the `TX-sync` component immediately.
And loop checking the Redis for the `key` after some `Thread.sleep()`
checkstyle EmptyBlock
checkstyle fixRightCurly Script
checkstyle EmptyStatement
checkstyle RightCurly
checkstyle TailingWhite
checkstyle NeedBraces
Fix the line separator in the `fixRightCurly.gradle`
fixes
fixModifiers after fixFinal
Revert CachingSessionFactory
Class is spied in tests.
checkstyle - Import Rules
checkstyle InterfaceIsType
checkstyle InnerTypeLast
checkstyle OneStatementPerLine
CovariantEquals
OneTopLevelClass
* Revert `'\n'` -> `System.lineSeparator()` in the Gradle scripts to meet Git `autocrlf = true` on Windows
* Fix timing issue with the `LastModifiedFileListFilterTests`, when the `age = 1` might not be enough for the file object when we have some delay before checking
* `gradlew clean check -x test --parallel --continue` - to collect reports
* `gradlew fixThis --parallel` - to fix all possible vulnerabilities. With `-Dfile.encoding=UTF-8` on Windows
Since the `RequireThisCheck` doesn't see parents for anonymous classes (e.g. `Runnable` callback), its report doesn't contains the outer class name with `this.`,
therefore we still have to fix those cases manually.
Thanks to the wrong `replacer` just with `this.` we have uncompilable code enough easy to find problems.
Not so easy to fix for good readability though...
* Upgrade to Grade 2.12
* Upgrade to SonarQube native plugin
The fix contains at about 300 files. So, will be done on merge.
Fix `fixThis.gradle` according PR comments
Apply `fixThis` and also `fixModifiers` for test classes.
Fix some `this.` inner issues manually.
Make code polishing for long lines after `fixThis`
Fix conflicts and vulnerabilities after the rebase
Add `fixModifiers.gradle` to run after `check` task.
The `RedundantModifierChecker` doesn't catch all errors at once.
We need run `check -> fixModifiers` pair several times to reach `SUCCESSFUL`.
The `fixThis` has been ported from the SA, but without applying.
* Polishing for `fixModifiers.gradle`
* Fix all redundant modifier with the `fixModifiers.gradle`
NOTE: Since all these task modify files on Windows we have to run them with the `-Dfile.encoding=UTF-8`.
Otherwise they are read and write with the Windows default `cp1251` making them incompatible with Unix system.
JIRA: https://jira.spring.io/browse/INT-3939
Previously `RedisQueueInboundGateway` and `RedisQueueOutboundGateway` used different `RedisSerializer`s
for non-String objects.
* Change the default to the `JdkSerializationRedisSerializer` for the consistency in case of client/server scenarios.
The previous behavior with the `StringRedisSerializer` can be reinstated with `serializer` injection.
Doc Polishing.
JIRAs:
https://jira.spring.io/browse/INT-3830https://jira.spring.io/browse/INT-3523https://jira.spring.io/browse/INT-3846
* Fix the OOM condition, when we `release()` `UpperBound` independently of the previous `remove` result (https://jira.spring.io/browse/INT-3846)
* Fix "confuse" around `groupCapacity`, when we really didn't care about individual groups (https://jira.spring.io/browse/INT-3523)
* Add `upperBoundTimeout` to have a hook to wait some time for the empty slot in the store (https://jira.spring.io/browse/INT-3830)
* Fix some JavaDocs warnings
* Fix some typos
* Fix inconsistency in the `DelayHandler` around `removeMessageFromGroup` when `MS` is `SimpleMessageStore`
* Remove `UpperBound.release()` operation from `SimpleMessageStore.removeGroup()`.
The waiting process should worry about the new `UpperBound` instance.
Some other polishing
`tryAcquire` outside of the `lock`
Move `tryAcquire` within the `addMessageToGroup` outside of `lock`.
But do that only for groups which already exist.
For the new groups we have a fresh `UpperBound`, so no need to worry about dead lock and
we can obtain a permit immediately.
SimpleMessageGroup: BlockingQueue -> LinkedHashSet
`SimpleMessageStore`: use "unsynchonized" `SimpleMessageGroup`
Make some synchronization fixes according to the migration to the `LinkedHashSet`
Avoid extra `Collection`
`ResequencingMessageHandler`: compare `size()` of collections instead of `containsAll()`
Fix `ConcurrentModificationException` in the `AbstractKeyValueMessageStore`
Add `SimpleMessageStore.clearMessageGroup()`
Accept polishing and fix `RedisChannelMessageStoreTests`
`@Deprecated` `MessageGroupStore.removeMessageFromGroup()`
Fix some typos
Introduce `SimpleMessageGroupFactory`
Extract `MessageGroupFactory` and address PR comments
Polishing after rebase
JavaDocs and Reference Manual
Fix JavaDocs
JIRA: https://jira.spring.io/browse/INT-3924
* The `RedisQueueInboundGateway` and `RedisQueueMessageDrivenEndpoint` used `MICROSECONDS` for the wait for stop condition.
- Change to `MILLISECONDS`
* Both of them may pick up data from the Redis List even after the `stop()`.
- Add `rightPush()` to the `RedisQueueInboundGateway` to emulate `rollback`.
- Even if we have `rollback` there we may "steal" messages after the `context.stop()` causing unexpected race condition.
So change the logic between `stopTimeout` and `receiveTimeout` to `max`, thus we wait for stop more than for `rightPop`.
* Some attributes have been missed for the XML configuration.
- add `stop-timeout` and `recovery-interval` attributes to the XSD and parsers code.
- adjust `RedisQueueGatewayIntegrationTests` to use those new attributes.
* Some tests needed polishing:
- Rework `RedisStoreOutboundChannelAdapterIntegrationTests` to be based on `@ContextConfiguration` for better performance
- `RedisChannelParserTests` used to `mock` channels for the same Redis topic. Change of of them to different name to avoid `Dispatcher has no subscribers`
from the `testPubSubChannelUsage()` when the second `SubscribableRedisChannel` doesn't have subscribers.
* Polishing for `redis.adoc`
Address PR comments
* Rework logic to the `Math.min(this.stopTimeout, this.receiveTimeout)` on the adapter/gateway `stop()`
* Fix JavaDocs and `redis.adoc` to describe `stopTimeout` properly.
* Rework `RedisQueueGatewayIntegrationTests` to be based on the `randomUUID` for the queue name to send and receive.
Address PR comments.
