* To align with the rest of Spring portfolio, based on Spring Framework 5.3.x generation,
it is better to upgrade to the Kotlin version supported by SF
* Fix `ClassUtils.isLambda()` to treat `$inlined$` classes as Kotlin lambdas and let
Java DSL to delegate to the `LambdaMessageProcessor` for better expected type handling.
Looks like Kotlin does not carry type info in its lambdas anymore (at least for functions).
* Add `-parameters` to the compiler args for possible method argument names discovery.
Essentially realign with the rest of portfolio
* Fix Kotlin tests for deprecated API
* GH-3554: Eval remote dir on each synchToLocal
Fixes https://github.com/spring-projects/spring-integration/issues/3554
The `remoteDirectoryExpression` was introduced into an
`AbstractInboundFileSynchronizer` to let end-user to evaluate a remote directory
on each poll (essentially on each `synchronizeToLocalDirectory()` call).
The fix for the https://jira.spring.io/browse/INT-4491 introduced a regression
when such an expression was evaluated only once when we call a `setRemoteDirectory()`.
So, an original purpose of this option was lost and we don't get an actual
remote dir on each poll
* Remove `evaluateRemoteDirectory()` method and its usage since it doesn't
reflect expectation of the remote dir expression property
* Reinstate the `remoteDirectoryExpression` evaluation in the
`synchronizeToLocalDirectory()`
* Propagate the result of that expression into further methods for copying
files
* Remove setting of the `remoteDirectory` variable into a global `EvaluationContext`
of the `AbstractInboundFileSynchronizer` instance since it is not thread-safe
* Instead create an `EvaluationContext` locally for each `synchronizeToLocalDirectory()`
call and set the `remoteDirectory` variable into this scoped instances
* Generate a local file name from the `localFilenameGeneratorExpression`
against locally created `EvaluationContext` with the mentioned `remoteDirectory` variable
* Cover the expected functionality with a unit-test
**Cherry-pick to `5.4.x` & `5.3.x`**
* * Fix `testRemoteDirectoryRefreshedOnEachSynchronization` according PR review
The `AbstractPersistentAcceptOnceFileListFilter`
extends now an `AbstractDirectoryAwareFileListFilter`
and makes all the implementors to implement its `isDirectory()` method.
This is a breaking change in the point release which makes implementations
outside of the core project not compilable
The regression after: https://github.com/spring-projects/spring-integration/issues/3488
Related fix in the Spring Cloud GCP: https://github.com/spring-cloud/spring-cloud-gcp/pull/2654
* Implement an `isDirectory()` in the `AbstractPersistentAcceptOnceFileListFilter`
to let other implementations do not worry during upgrade
**Cherry-pick to `5.4.x` & `5.3.x`**
* Introduce a `groupConditionSupplier` for MGS
* Add a `MessageGroup.condition` option
* Add a `MessageGroupStore.conditionSupplier` option
* Use it from the `SimpleMessageStore.addMessagesToGroup()` API
to populate `condition` (if any) into a `MessageGroup`
* Introduce a `GroupConditionProvider` contract to be implemented
on those `ReleaseStrategy` contracts which could be aware of group condition
* Populated a `GroupConditionProvider.getGroupConditionSupplier()`
into a `MessageGroupStore` from the `AbstractCorrelatingMessageHandler`
for end-user convenience
* Rework a `FileMarkerReleaseStrategy` to implement a `GroupConditionProvider`
to provide a function which produces a condition from `file_lineCount` header
of the `END` marker message
* Make the `FileMarkerReleaseStrategy` logic already based on the condition from a group
* Delegate `GroupConditionProvider` from the `FileAggregator`
* Add test for empty file aggregation
* * Implement `condition` in the `AbstractKeyValueMessageStore` and `MongoDbMessageStore`
* Test `condition` for `mongo-aggregator-config.xml` and `FileAggregatorTests` against GemFire
* * Implement `condition` in the `ConfigurableMongoDbMessageStore`
* * Implement `condition` in the `JdbcMessageStore`
* `FileAggregatorTests` against `JdbcMessageStore`
* Refactor `JdbcMessageStore` for better handling of message group metadata
* Remove unused `MARKED` column in the DDL in favor of newly introduced `CONDITION`
* * Add docs for message group condition
* * Move `conditionSupplier` option from MGS to AbstractCorrelatingMH
* Make it as a `BiFunction` to propagate existing condition alongside with the
message to consult
* Expose `groupConditionSupplier` in Java & XML DSLs
* * Fix language in docs
* INT-4116: Introduce FileAggregator
JIRA: https://jira.spring.io/browse/INT-4116
* Implement a `FileSplitter.FileMarker`-based aggregation strategies
and utilize them in a general `FileAggregator` component
* Make `HeaderAttributeCorrelationStrategy.attributeName` as `final`; add `Assert.notEmpty()`
* Fix `AggregatorFactoryBean` and `AggregatorSpec` to parse the provided processor
for possible `CorrelationStrategy` and/or `ReleaseStrategy`
* Introduce short-cut methods into Java & Kotlin DSL for an `aggregate()` configuration
* Introduce a `FileHeaders.LINE_COUNT` for header to be populated in the `FileSplitter`.
We need this info in the `FileAggregator` to avoid possible overhead with JSON deserialization
of the `FileSplitter.FileMarker` messages
* Test and document the feature
* Improve `FileSplitter` doc for code block switch (tabs)
* * Rework FileAggregator do not use Java Streams
Fixes https://github.com/spring-projects/spring-integration/issues/3507
The `OSDelegatingFileTailingMessageProducer` passing command string to `Runtime.getRuntime().exec()`
may cause problems if spaces (and other special characters) are used in the filename.
* Use an array for command and its options to let the target `Runtime` to parse and
execute it properly
**Cherry-pick to 5.4.x, 5.3.x & 5.2.x**
* Use a tmp file in the `FileTests.testFileReadingFlow()`
for writing content and then rename it to the target file.
It looks like we may already have a file just with a short content
and it is picked up by the watch service for processing in the flow
* Rework `AggregatorWithCustomReleaseStrategyTests` to JUnit 5 and remove
extra looping logic since it just does not add any extra coverage just
performance overhead via crating and destroying the same application ctx
* Rework `ChannelAdapterParserTests` to `@SpringJUnitConfig` managed test
* FileInbound DSL: Add `recursive` for convenience
Related to https://stackoverflow.com/questions/66171881/how-to-read-nested-txt-file-from-spring-integration-file
The `FileInboundChannelAdapterSpec` can be configured with an external `DirectoryScanner`,
but it sometimes becomes burden for end-users to extract a scanner bean and configure it
with all the required file filters
* Expose a `recursive(boolean)` option for better end-user experience
* Rework `FileTests` for JUnit 5
* Mention a new option in the docs
* * Restore accidentally removed code
* Restore special symbols in the `FileTests`
* Fix language in the docs according review
Resolves https://github.com/spring-projects/spring-integration/issues/3488
Resolves two problems:
- When changes are made deep in the directory tree, they were not detected because
the directory is in the metadata store and only passes the filter if a file
immediately under it is changed, changing the directory's timestamp.
This is solved by subclassing `AbstractDirectoryAwareFileListFilter`, allowing its
`alwaysAcceptDirectories` property to be set.
- Only the filename was used as a metadata key; causing problems if a file with the
same name appears multiple times in the tree.
This is solved with a new property on `AbstractDirectoryAwareFileListFilter` used by
the gateways to determine whether to filter the raw file names returned by the session
(previous behavior) or the full path relative to the root directory.
**cherry-pick to 5.4.x, 5.3.x**
* Some code style clean up
* GH-3370: Remove synchronized from RemoteFileUtils
Fixes https://github.com/spring-projects/spring-integration/issues/3370
The `synchronized` on the `RemoteFileUtils.makeDirectories()` makes an application too
slow, especially when we deal with different paths in different sessions
* Remove the `synchronized` from that method and rework `SftpSession.mkdir()`
to return `false` when "A file cannot be created if it already exists" exception
is thrown from the server.
Essentially make an `exists()` call to be sure that an exception is really related
to "file-already-exists" answer from the server
**Cherry-pick to 5.3.x, 5.2.x & 4.3.x**
* * Re-throw an exception in the `SftpSession.mkdir()`
when error code is not `4` or remote dir does not exist
* * Check `session.mkdir()` result in the
`RemoteFileUtils` to throw an `IOException` when `false`
* * Fix mock test to return `true` for `mkdir` instead of `null`
* Implement `Emission` handling in the `FluxMessageChannel` and `IntegrationReactiveUtils`
* Upgrade to Spring Kafka `2.6.0`
* Fix R2DBC components for deprecation in the Spring Data R2DBC
* Implement `StatementMapper.SelectSpec` for query expression
* Clean up for some sporadic test failures
Resolves https://github.com/spring-projects/spring-integration/issues/3315
`maxFetchSize` ignored with filters supporting single file filtering (default).
This breaks "fair" rotation with the `RotatingServerAdvice`.
Honor `maxFetchSize`, even with filters that support single file filtering.
**cherry-pick to 5.3.x, 5.2.x**
* Support Java 14
* Provide changes to avoid deprecated Java API
and have a compatibility back to Java 8
* Change affected test classes to JUnit 5 whenever it is possible
* Ignore/Disable some TCP/IP tests which don't pass on Java 14
* Fix (some) TCP tests on JRE 14
* Fix SSL Handshake test - client side handshake is successful with java 14
- change the badClient cert to a badServer cert to force an error on the client side
Co-authored-by: artembilan <raven666>
Co-authored-by: Gary Russell <grussell@pivotal.io>
Fixes https://github.com/spring-projects/spring-integration/issues/3271
When exception happens at `.withPayload(session.readRaw(remotePath))`
in the `AbstractRemoteFileStreamingMessageSource` we don't close session.
The resource leaking happens in the caching session factory
* Add `session.close();` into the `catch (IOException e) {`
in the `AbstractRemoteFileStreamingMessageSource.doReceive()`
to clean up resources properly
**Cherry-pick to 5.2.x, 5.1.x & 4.3.x**
* Use a SimpleJsonSerializer in the FileSplitter
* To avoid extra dependency for Jackson when we
serialize `FileSplitter.FileMarker` to JSON, use
a `SimpleJsonSerializer` instead.
* Fix `SimpleJsonSerializer` to escape a `\` symbol from property values
since it is used for quoting string values in the final JSON
* * Document the change
* Increase latch timeouts for `RedisLockRegistryLeaderInitiatorTests`
* Some RMI tests fail with different outcome because an RMI registry might be
available on default `1099` port.
Fix them to not rely on that port
* GH-2996: Add Resource for remote file transfer
Fixes https://github.com/spring-projects/spring-integration/issues/2996
* Fix `RemoteFileTemplate.send()` to support a `Resource` payload
for remote file transferring content
* Code style clean up for `RemoteFileTemplate`
* Remove `volatile` for configuration properties for better performance
* Change a `charset` to the `Charset` for only once conversion from string
during configuration phase
* Fix (S)FTP tests for new functionality
* Change affected tests to JUnit 5
* Document a new feature; mention all the supported types and `FileExistsMode` constants
* * Fix language in `whats-new.adoc`
Fixes https://github.com/spring-projects/spring-integration/issues/3169
All other `DefaultSessionFactoryLocator` contracts are based on the `Object`,
so this `addSessionFactor`y has to be on `Object` as well.
* Add `DefaultSessionFactoryLocator.addSessionFactory(Object key, SessionFactory<F> factory)`
* Deprecate existing one based on `String`
* Fix tests do no use a deprecated API
* Some other code style clean up in the affected classes
**Cherry-pick to 5.2.x**
Fixes https://github.com/spring-projects/spring-integration/issues/3132
It turns out that Checkstyle EmptyBlock doesn't complain about
empty default ctor.
Plus a new check for `super();` call treats it as a violation
* Remove `super();` from all the no-arg ctors
* Code style clean up in the affected classes according
IDEA suggestions
* Fix new Sonar smells
* INT-3494: Resolve dir for writing as a Resource
JIRA: https://jira.spring.io/browse/INT-3494
The expression for local directory can be resolved into a `Resource`
or resource location.
* Fix `ExpressionUtils.expressionToFile()` to support `Resource` and
also use `ResourceUtils.getFile(path)` when expression result is a string
* Modify tests to ensure that resource is resolved properly
* Upgrade affected tests to JUnit 5
* Mention an new functionality in docs
* * Improve Java doc for `ExpressionUtils.expressionToFile()`
* Finish the sentence in the `file.adoc`
Fixes https://github.com/spring-projects/spring-integration/issues/3105
The loop in the `AbstractRemoteFileStreamingMessageSource` doesn't check
the next polled file for filtering.
* Add `continue;` when we filter the current file and poll the next one.
So, we go over into the `while()` beginning
* Clean up the code style and cover more code path in test
Fixesspring-projects/spring-integration#3076
* Return `null` instead of throwing an exception
* Fix license header
* Add missing `@since`
* Avoid over-engineered test logic
* Fix code style
* Test non-existent file does not get created
* Fix: remove unused import
Fixes https://github.com/spring-projects/spring-integration/issues/3038
* Add more delegating setters into the `AbstractRemoteFileOutboundGateway`
for its `RemoteFileTemplate` property
* Restrict externally provided `RemoteFileTemplate` from modifications
in the `AbstractRemoteFileOutboundGateway`: those options must be
configured on that external `RemoteFileTemplate`
* Expose new options in the Java DSL specs
* GH-3043: Add FileHeaders.REMOTE_HOST header
Fixes https://github.com/spring-projects/spring-integration/issues/3043
* Populate a `FileHeaders.REMOTE_HOST` from the
`AbstractRemoteFileStreamingMessageSource` and "get"-based commands
in the `AbstractRemoteFileOutboundGateway`
* Extract the value from the a `Session.getHost()` contract
* The `AbstractInboundFileSynchronizingMessageSource` cannot be
addressed with this because the real message is already based on the
locally stored file
* Adjust some affected tests according our code style requirements
* * Add remote file info support into `AbstractInboundFileSynchronizingMessageSource`
* Introduce a `MetadataStore` functionality into the `AbstractInboundFileSynchronizer`
to gather a remote file info an save it in the URI style against local file
* Retrieve such an info in the `AbstractInboundFileSynchronizingMessageSource`
during local file polling
* Introduce `protocol()` contract for the `AbstractInboundFileSynchronizer`
to build a proper URI in the metadata for external readers to distinguish
remote files properly
* Document the feature
* * Fix some typos in Docs
* * Rename property and header constant to the `HOST_PORT` pair
* Fix typos in Docs
* Add `remote-file-metadata-store` and `metadata-store-prefix` into XSD
of (S)FTP Inbound Channel Adapters
* Add `remoteFileMetadataStore` and `metadataStorePrefix` options
into `RemoteFileInboundChannelAdapterSpec` for Java DSL
Fixes https://github.com/spring-projects/spring-integration/issues/2748
* Refactor more `MessageHandlingException`s to include `this` into an
exception message
* Revert using `MessagingException` in some places which really are not
about messaging.
This helps to wrap them into `MessageHandlingException` later in the
`MessageHandler` for the `BeanDefinition` reference
* Remove `volatile` from configuration properties in the affected
classes
* Remove already deprecated `JmsOutboundGateway.setPriority()`
* Add `resource` and `source` for `BeanDefinition` in the
`AbstractChannelAdapterParser` & `AbstractInboundGatewayParser`
* Document the feature
JIRA: https://jira.spring.io/browse/INT-2480
* Introduce `headers-function` option into the `aggregator` for merging
and computing headers for the output message based on the completed
group
* Implement a `DefaultAggregateHeadersFunction` and use it in the
`AbstractAggregatingMessageGroupProcessor` for default behavior with
possible injection for any other implementation
* Add `DelegatingMessageGroupProcessor` to wrap any other
`MessageGroupProcessor` implementations with possible usage of the
`headersFunction` if result is not a `Message` or `MessageBuilder`
* Make `AbstractCorrelatingMessageHandler.getOutputProcessor()` as
`public` rto give access to this option from the `AggregatorSpec` to
be able to inject a `headersFunction` in Java DSL configuration
* Add `AbstractIntegrationMessageBuilder.getHeader()` to get access to
some underlying header avoiding extra `Map` in case of `getHeaders()`
* Change a logic in the `AbstractMessageProducingHandler.produceOutput()`
to consult a `reply` for the `replyChannel` as well `routingSlip` header
if the `reply` is a `Message` or `MessageBuilder`
* Introduce a `AbstractMessageProducingHandler.messageBuilderForReply()`
and use it in `AbstractMessageSplitter` to avoid duplication
* Validate a new functionality in tests
* Fix `FileOutboundGatewayParserTests` to rely on the `TemporaryFolder`
to clean up test files after using
* JavaDocs for `DefaultAggregateHeadersFunction`
* Some `router.adoc` polishing
* Fix link to Reactor in the `router.adoc`
* Add docs for new `Function<MessageGroup, Map<String, Object>>` strategy
* Doc polishing.
* Fix `LongRunningTest` to rely on the `@EnabledIf` from Spring Test
Framework instead of deprecated now in JUnit 5
* Remove unnecessary stubbing in the `FileReadingMessageSourceTest`
* GH-2765: Add discardChannel for splitter
Fixes https://github.com/spring-projects/spring-integration/issues/2765
When encountering empty collections, splitter should be able to send
the result to a discard channel.
Currently, when encountering an empty collection,
the splitter ends the flow.
Some use-cases may rely on a custom split function which may returns
empty collections.
These use-cases should be able to define a discard channel
so they can proceed with a possible compensation flow.
* Add `discardChannel` option to the `AbstractMessageSplitter`
* Delegate `discardChannel` population from everywhere it is possible:
DSL, XML, `AbstractMessageSplitter` extension like `FileSplitter` etc.
* Fix `FileSplitterTests` for broken charset
* Document new feature; fix some typos and out-dated code sample
* * Fix `SplitterFactoryBean` for NPE on the `discardChannelName`
propagation
* * Check `this.discardChannel` first
* `Assert.state()` in `doInit()` for mutual exclusiveness
To avoid wrong encoding during reading and writing files by the
`updateCopyrights` Gradle task, it is better to have an explicit
`-Dfile.encoding=UTF-8` before starting JVM.
* Fix `AbstractFilePayloadTransformerTests` for non-standard symbols in
the test data
* fix broken test
`FileToStringTransformerTests.transform_withWrongEncoding_notMatching()`
* Do not fix urls for the `http://apache.org/xml/features` when we
configure XML components
* Do not fix urls for localhost and fake links in tests
* Fix all the dead links in docs
Resolves https://github.com/spring-projects/spring-integration/issues/2777
If the filter supports it, defer filtering until the last possible moment.
Then, the worst case scenario after a catastrophic failure (e.g. power loss),
would be that at most one file will be incorrectly filtered on restart.
Polishing and add more tests.
Polishing Javadocs
More Polishing
Final polishing
More polishing.
Polishing and docPolishing and docs.
* Fix typos in Docs
