JIRA: https://jira.spring.io/browse/INT-4095
Limit the number of remote files fetched on each poll (when it is necessary to fetch files).
Polishing - PR Comments
Polishing - Decouple MaxFetchSize from Poller
Polishing - PR Comments
Schemas and Docs
More Polishing
* Polishing according PR comments
JIRA: https://jira.spring.io/browse/INT-4099
The `IntegrationMessageHeaderAccessor.CLOSEABLE_RESOURCE` value has a typo
* Fix the typo in the `IntegrationMessageHeaderAccessor.CLOSEABLE_RESOURCE` constant value
* Fix the same type in the (s)ftp.adoc
JIRA: https://jira.spring.io/browse/INT-3941
Previously, like the SF `<jms:/>` namespace, Spring Integration used a default bean
name `connectionFactory` for the JMS Connection Factory.
Spring Boot auto-configures a bean name `jmsConnectionFactory`.
Since it is still common to use Spring Integration XML configuration with Spring Boot,
it is useful to align the two so that an SI app can use the auto configured bean.
Docs polishing
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4082
Starting with Spring Data 1.9 the `MappingContextIsNewStrategyFactory` relies on a newly introduced `PersistentEntities` and doesn't register entities lazily any more.
Such a change finishes with the `Unsupported entity` exception when an auditing is switched on (`<mongo:auditing/>`)
for the `MongoDbMessageStore` and `AbstractConfigurableMongoDbMessageStore` internally created `MongoTemplate` and `MessageWrapper` and `MessageDocument` SI internal entities.
* Don't register `ApplicationContext` into internally created `MongoTemplate`s since to avoid entity events emitting for `MessageWrapper` and `MessageDocument`
* Pull `MongoDbMessageBytesConverter` to the top-level class to let customize `MappingMongoConverter` properly if there is need to audit `MessageDocument` anyway,
what can be possible via external injections into `AbstractConfigurableMongoDbMessageStore` implementation
* Fix `mongodb.adoc` for some typos
**Cherry-pick to 4.2.x**
JIRA: https://jira.spring.io/browse/INT-4064
We configure `IdempotentReceiver` via `<int:idempotent-receiver>` component or `@IdempotentReceiver` annotation.
In case of regular Java config, e.g. direct `ConsumerEndpointFactoryBean` usage or Java DSL,
it isn't possible to configure `idempotentReceiverInterceptor` enough easy
* Introduce `if...else` logic into `ConsumerEndpointFactoryBean` to proxy `MessageHandler`,
if `adviceChain` contains an newly-introduced `HandleMessageAdvice`.
And do that independently if `MessageHandler` is `AbstractReplyProducingMessageHandler`
* Make `idempotentReceiverInterceptor extends HandleMessageAdvice`
* Skip `HandleMessageAdvice` in the `AbstractReplyProducingMessageHandler`
* Add advice applying logic into the `AbstractMethodAnnotationPostProcessor` as well
Introduce `HandleMessageAdvice` marker interceptor to cover the case when an `Advice` can be advices as well.
Remove unused `setAdviceChainIfPresent()` method in the `AbstractMethodAnnotationPostProcessor`
Document `HandleMessageAdvice`
Increase wait latch timeouts in the `LockRegistryLeaderInitiatorTests`
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4063
Since `DefaultXmlPayloadConverter` is used from many out-of-the-box components by default,
make it more flexible with the supported input types which can be converted into `Document`
Improve `xml.adoc` about this types and also fix a lot of typos there as well
Address PR comments
JIRA: https://jira.spring.io/browse/INT-4067
When `FileSplitter` is configured with `markers = true` and file is empty, an `iterator` for file throws `IOException: Stream closed`,
because we close the `buffer` just after the first `readLine()` attempt, but still return `true` from the first `hasNext()` call
where the `this.sof` and `this.eof` are `true` for markers.
Add logic to mark internal splitter `iterator` as `done` where we don't have content and still in `sof` state.
JIRA: https://jira.spring.io/browse/INT-4061
The `FTPClient.changeWorkingDirectory()` can be performed only after proper connection to the target FTP server
* Fix `ftp.adoc` to mention proper `postProcessClientBeforeConnect()` callback.
* Make other minor polishing for the `ftp.adoc`
JIRA: https://jira.spring.io/browse/INT-4058
If you hold the lock, you are the leader. This simple idea gets you
a long way if there is no "native" leader initiator. E.g. you
can use this with a RDBMS with JdbcLockRegistry.
Add some docs on leader election under "endpoints"
Make thread name for leader initiator unique
In case there are multiple instances in the same context we would
like to be able to spot them in the logs.
INT-4058: Polishing
* Code refactoring in the `LockRegistryLeaderInitiator`: SI use 120 line length
* Add `Assert.notNull()` for required properties
* Add `this.lock.unlock()` and `Thread.sleep(LockRegistryLeaderInitiator.this.busyWaitMillis)` into the `catch` block to let distributed elections to work.
Otherwise there is a big chance that we will acquire the lock and become a leader again just after `yield()`
* Change `LockContext.toString()` to use simple `String` concatenation which is optimized by compiler to the `StringBuilder`.
That let to have some better micro-performance compared with with extra `Formatter` object in case of `String.format()`
* Add `JdbcLockRegistryLeaderInitiatorTests`
* Fix `yield()` logic based on the `Future.cancel(true)`.
Since one `FutureTask.cancel(true)` makes it as `INTERRUPTED` any subsequent `yield()` does not make any effect, therefore our infinite selector loop isn't interrupted one more time.
* Reschedule the `LeaderSelector` in the `yield()` after cancel(true).
* Rework `catch` in the selector loop just to the `InterruptedException` and `return null;` to stop looping and let reschedule the selector.
* Add one more `onRevoked` logic into the `final` of the selector loop to notify that we have lost leadership during `stop()`
* Improve `JdbcLockRegistryLeaderInitiatorTests` to ensure that several `yield()` on the same initiator work well.
* Add `What's New` note.
JIRA: https://jira.spring.io/browse/INT-4048
The semantics of the locks are the same as for the default lock
registry (they have to be unlocked by the same thread that
locked them).
Updates from feedback
Add interface for JdbcClient so transactions can bind to proxy
Fix typo in exception message
Fix drop script
JdbcClient -> LockRepository
Make lock() contract more like native Lock
Make lockInterruptibly throw InterruptedException more
Delete all expired locks, not just the ones that we own
INT-4048: Polishing
* `DefaultLockRepository`: replace `prefix` only once in the `afterPropertiesSet()`
* Add JavaDocs to the `DefaultLockRepository`
* `JdbcLockRegistry`: add `Thread.sleep(100)` to lock loops do not request DB so often like in case of clean `while(true)`
* `JdbcLockRegistry`: implement more robust `tryLock(long time, TimeUnit unit)`. The logic mostly copied from `RedisLockRegistry`
* Add `testExclusiveAccess()` to be sure that JDBC locks have exclusive access to the data sequence
Document `JdbcLockRegistry`
Upgrade to SF-4.3 GA
JIRA: https://jira.spring.io/browse/INT-4046
Since not all FTP servers provide proper `STAT` command implementation,
plus the `NLIST` doesn't work properly for directories cases, introduce the `FtpRemoteFileTemplate.ExistsMode`
to let:
* to perform `STAT` by default (previous) behavior;
* to switch to `NLIST` for `FtpRemoteFileTemplate` internal use;
* perform the full `NLIST` and `FTPClient.changeWorkingDirectory()` algorithm if needed.
* Improve (S)Ftp components to use proper `RemoteFileTemplate` for internal instantiation
* Introduce `FtpMessageHandler` to wrap `FtpRemoteFileTemplate` with the proper `NLIST` `ExistsMode`
* Cover `NLIST` switching from the `FtpOutboundChannelAdapterParser` and `FtpOutboundGatewayParser`
* Document the `FtpRemoteFileTemplate.ExistsMode`
* Fix typo in the recently introduced `RemoteFileOperations.getSession()` method name
* Add JavaDoc to `Session.exists()`
* Add `NLIST` support for the `FtpSession.exists()` to meet the API requirements
**Cherry-pick to 4.2.x and 4.1.x**
Addressing PR comments
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4038
General format for node name: bean.method#n
The #n is needed to disambiguate methods with the same name.
Polishing according PR comments
* Fix generic type for `Collections.unmodifiableMap` usage
* Change the gateway method logic in the `IntegrationGraphServer` to include the method signature in the node name to distinguish them as unique
* Move `MutableMessageBuilderFactoryTests` into the separate nested `mutable` package since `@IntegrationComponentScan` sees a new `@MessagingGateway` in the `IntegrationGraphServerTests`.
See https://jira.spring.io/browse/INT-4040
* Document the `@MessagingGateway` representation in the `graph.adoc`
JIRA: https://jira.spring.io/browse/INT-3997
* Created the `H2ChannelMessageStoreQueryProvider` class for the
`ChannelMessageStoreQueryProvider` interface
* The `LongRunningIntegrationTest` class needs the
`RUN_LONG_INTEGRATION_TESTS` variable defined in the 'system'
and set in true. It to let work the test methods for the
`H2TxTimeoutMessageStoreTests` class
* Updated reference documentation about `ChannelMessageStoreQueryProvider`
* Polishing
JIRA: https://jira.spring.io/browse/INT-4005
* The `<int:queue>` lets declare the `capacity` attribute together
with either `message-store` or `ref` attributes, which it is not
correct
* The `<int:queue>` lets declare the `capacity` attribute together
with either `message-store` or `ref` attributes, which it is not
correct
* The `priority-queue` lets declare the `capacity` attribute
together with the `message-store` attribute, which it is not
correct
* Reference documentation fixed for `channel.adoc` and `jdbc.adoc`,
doing mention about these restrictions
* Minor changes about conventions and formats applied.
Code polishing
JIRA: https://jira.spring.io/browse/INT-4015https://jira.spring.io/browse/INT-3854
Initial commit.
Reworked to emit an input stream and use the file splitter.
Add StreamTransformer.
Add CLOSABLE_RESOURCE header so we can close the session automatically.
Implement INT-3854, FTP, SFTP
(S)FTP Namespace Changes
Docs - also fixes a PDF overflow
Polishing - PR Comments
checkstyle fixes
Polishing - Add Namespace for StreamParser
Polishing - PR Comments
JIRA: https://jira.spring.io/browse/INT-4011
* The `jdbc.adoc` has an incorrect reference in a snippet code about
*Priority Channel*. The correct attribute value must be `channelStore`
JIRA: https://jira.spring.io/browse/INT-4003
* The `channel.adoc` has a wrong mention of `JdbcMessageStore`,
when the `JdbcChannelMessageStore` must be used
JIRA: https://jira.spring.io/browse/INT-3994
If a header mapper is injected into the mail receiver, a mapped
message is result instead of a message with the raw `MimeMessage`.
`MimeMessage` properties are mapped as discrete headers; in addition
the raw email headers are provided as a multivalue map in the headers.
Handle Multipart
Since a Multipart content has a reference to the original MimeMessage,
convert to a simple byte[] by default.
Add an option in case the user wants to map the message but still
interpret the Multipart intact.
Namespace and Docs
Polishing
Polishing - PR Comments
JIRA: https://jira.spring.io/browse/INT-3989,
https://jira.spring.io/browse/INT-3990,
https://jira.spring.io/browse/INT-3988
INT-3989: Add `FileReadingMessageSource.WatchServiceDirectoryScanner`
* Deprecate top-level `WatchServiceDirectoryScanner` because of inconsistency around `Lifecycle` and shared `directory` property
* Copy/paste its logic into the `FileReadingMessageSource.WatchServiceDirectoryScanner` to hide that inconsistency, but still get a gain from the `WatchService` benefits
* Add support for the `StandardWatchEventKinds.ENTRY_MODIFY` and `StandardWatchEventKinds.ENTRY_DELETE` events in the `FileReadingMessageSource.WatchServiceDirectoryScanner`
* Introduce `useWatchService` option to switch to the internal `FileReadingMessageSource.WatchServiceDirectoryScanner`
* Make `CompositeFileListFilter` also as `ResettableFileListFilter`
* Deprecate weird `FileReadingMessageSource.onSend()` method and remove its usage from tests
* Modify `WatchServiceDirectoryScannerTests` for the new logic
* Document changes
Add `MODIFY` and `DELETE` test coverage
Optimize the `filesFromEvents()` logic replacing item with the fresh event sources.
Remove the item from the result set in case of `DELETE` event, because file removal generates both `MODIFY` and `DELETE` events.
* Add `FileReadingMessageSource.setWatchEvents` to allow to listen to the specific events,
not only `CREATE` or all of them.
* Modify tests and docs to reflect the new API
Address PR comments
* Improve `FileReadingMessageSource.setWatchEvents()`
* Add `file.exists()` before adding file from event.
Add more DEBUG logs into the `WatchServiceDirectoryScanner`
With the fact of those logs provide more optimizations:
* Don't register the same directory for watching:
- use the `ConcurrentMap<Path, WatchKey> pathKeys` to track registrations
- any modification within the directory causes the `ENTRY_MODIFY` for the directory as well.
So, skip such an event exactly for the directory during `walkDirectory`
* Add debug logs in case of `ENTRY_DELETE`
JIRA: https://jira.spring.io/browse/INT-3387,
https://jira.spring.io/browse/INT-3806
* Introduce
```
MessageGroupStore
void addMessagesToGroup(Object groupId, Message<?>... messages);
```
And implement it in all stores.
* Use new `addMessagesToGroup` where it is reasonable, e.g. `DelayHandler`
* Optimize test-case to use a new store method (where it is possible)
* Fix timing delays in the `JdbcMessageStoreTests`
* Introduce `PersistentMessageGroup`
* Add `AbstractMessageGroupStore#proxyMessageGroupForLazyLoad` to wrap the raw `MessageGroup` to the `PersistentMessageGroup` for lazy-load
* Rework `MessageGroupMetadata` do not be `immutable` and allow to store/restore in the `AbstractKeyValueMessageStore` only the `MessageGroupMetadata`
* Refactor `ResequencingMessageHandler` and `SequenceSizeReleaseStrategy` a bit for better performance when interact with the `MessageGroup`
* Add `AbstractMessageGroupStore#setLazyLoadMessageGroups` to switch off the `lazy-load` behavior and restore the previous full `MessageGroup` logic
* Add `What's New` note and `message-store.adoc` paragraph for the lazy-load functionality
`GroupType.PERSISTENT` and not lazy by default
PR Comments
Fix `JdbcMessageStoreTests` timing issues
Address PR comments
* Add performance test to the `ConfigurableMongoDbMessageGroupStoreTests`
* Add JavaDocs for the `MessageGroupFactory` methods
* Add `log4j.properties` into the `test` MongoDB module for better traceability
* Fix `JdbcMessageStore#getOneMessageFromGroup()` over the `doPollForMessage()` delegation.
The `jdbcTemplate.queryForObject()` requires exactly one and only one raw in `resultSet`
* Add performance test results into the `message-store.adoc`
