* Fix NPEs in DSL Specs
The `BaseWsInboundGatewaySpec` and `TailAdapterSpec` don't override super methods
and when we call them we fail with NPE since `target` was not populated.
* Fix `BaseWsInboundGatewaySpec` and its inheritors to populate the `target`
from their ctors.
* Remove redundant methods from `BaseWsInboundGatewaySpec` hierarchy
* Propagate `AbstractWebServiceInboundGateway` properties directly to the target
from the `BaseWsInboundGatewaySpec` inheritors
* Override `MessageProducerSpec` methods in the `TailAdapterSpec`
to populate respective option into the `FileTailInboundChannelAdapterFactoryBean`
* Expose more missed options for producer endpoint in the `FileTailInboundChannelAdapterFactoryBean`
**Cherry-pick to `6.0.x` & `5.5.x`**
* * `acceptIfNotNull()` for `errorMessageStrategy` in the `FileTailInboundChannelAdapterFactoryBean`
* * Fix `Unmarshaller` population logic in the `MarshallingWsInboundGatewaySpec`
* GH-3557: Add maxDepth, dirPredicate to FileReadMS
Fixes https://github.com/spring-projects/spring-integration/issues/3557
* Expose a `watchMaxDepth` on the `FileReadingMessageSource` for its `Files.walkFileTree()` API usage
* Add `watchDirPredicate` option ot the `FileReadingMessageSource` to skip sub-tree for `Files.walkFileTree()`
scanning according to some condition against directory `Path`
* Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
---------
Co-authored-by: Gary Russell <grussell@vmware.com>
Fixes https://github.com/spring-projects/spring-integration/issues/3869
* Move `ContextHolderRequestHandlerAdvice` to the `core` module for more general purposes
* Add `ContextHolderRequestHandlerAdviceTests`
* Rework `DelegatingSessionFactoryTests` to rely on the `ContextHolderRequestHandlerAdvice`.
This allows us to remove unnecessary XML configuration for this test class
* Document the feature
* GH-8562: Fix streaming source for remote calls
Fixes https://github.com/spring-projects/spring-integration/issues/8562
The `AbstractRemoteFileStreamingMessageSource.doReceive()` takes files first from a `toBeReceived` queue.
When `AbstractRemoteFileStreamingMessageSource.remoteFileToMessage()` fails to fetch the file content
because of interim connection issue, we reset this file from a filter and rethrow an exception.
The next `receive()` call will just go ahead to the next entry in the `toBeReceived` queue, but the
file we have just failed for will be retried only on the next list call to the remove directory.
This essentially breaks a possible in-order target application logic.
* Introduce `AbstractRemoteFileStreamingMessageSource.strictOrder` option to clear the `toBeReceived` queue
when we fail in the `remoteFileToMessage()`, so the next `receive()`
call would re-fetch files from remote dir, because the filter has been reset for those files.
* Fix `AbstractFileInfo.toString()` to not perform remote calls when we just log this file.
For example, we reset the file for connection failure and log the message about it,
but it fails again because we request `size` of the file which may require a remote connection.
**Cherry-pick to `6.0.x` & `5.5.x`**
* * Revert `AbstractFileInfo` changes
* Override `toString()` in `SmbFileInfo` instead -
exactly the place where connection is used to obtain
file attributes like `size` or `lastModified`
Fixes https://github.com/spring-projects/spring-integration/issues/3686
* Add `src/idea` with respective editor config for IntelliJ IDEA.
Must be imported into an IDE
* Add `src/eclipse` with respective editor config for Eclipse/STS
* Reformat imports in source code according a new editor config
Starting with Spring Data 2022.0.0 there is not going to be GemFire (Geode) support
directly from Spring team
A source code of the `spring-integration-gemfire` is moving to Spring Integration Extensions
from where community may consider to pull it and support in their own manner
Fixes https://github.com/spring-projects/spring-integration/issues/3840
* Create a new abstraction `RedisTest` for Testcontainers-based tests
* Move existing common utility methods to the new interface
* Migrate all existing Redis tests to use this new `RedisTest` interface
* Migrate all existing Redis tests to JUnit5
* Make all existing Redis tests confirming to JUnit5 standards, such as
default methods and classes visibility instead of public
* Add a dependency for parametrized tests in JUnit5
* Improve assertions readability across tests, for instance:
assertThat(thing.size()).isEqualTo(2) -> assertThat(thing).hasSize(2)
* Add real assertions to
`RedisLockRegistryTests.twoRedisLockRegistryTest` (earlier it did not
have assertions at all)
* Reformat, rearrange and cleanup the code
* Fix a couple of small changes after the code review:
* Change base interface name to be consistent with other similar places
* Typo in javadocs
* Small tests readability improvement regarding assertions
* Add `opens java.util` to `spring-integration-redis`
to satisfy a `ReactiveRedisStreamMessageHandlerTests.testMessageWithListPayload()`
requirements
Fixes https://github.com/spring-projects/spring-integration/issues/3623
* `IntegrationFlow` refactoring
* Apply several code style improvements and good practices
* Code style: no empty lines for methods javadocs
* make deprecated implementation reuse actual one instead of the copy-paste approach
* add whats-new comments
* Fix whats-new page according to standards
* Upgrade Spring dependencies to the latest SNAPSHOTs
* Fix tests to verify against stack traces: the message
of the `NestedRuntimeException` does not include the nested exception information.
Related to https://github.com/spring-projects/spring-framework/issues/25162
* Fix `JdbcMessageStore` and `DefaultLockRepository` to rely on the `DataIntegrityViolationException`
instead of only its `DuplicateKeyException` extension.
This is the current behavior of the SQL errors translation
* Disable `WebFluxDslTests.testValidation()` - doesn't subscribe to the reply somehow...
* Refine `SimplePool.PoolSemaphore.reducePermits()`
* Add `--add-opens` to `asciidoctor` Gradle task to avoid compilation warning
* Upgrade to Kotlin `1.6.10` and modify its Gradle task respective
* Add empty impl for new `AnnotationMetadataAdapter.getDeclaredMethods()`
* Parse HTTP methods to new `String methodNames` property of the `RequestMapping`
since `HttpMethod` is not an `enum` anymore.
* Ignore JDBC tests which rely on a map property resolution: the `[]` placeholder
is not handled in the latest SF anymore.
* Fix `WebFluxInboundEndpoint` for deprecations
* Upgrade to Java 17, SF-6.0, Gradle 7.2
* Upgrade to Jakarta dependencies and respective namespaces
* Fix some tests for Java 17 compatibility
* Fix wrong Javadocs
* Add some missed Javadocs
* Fix more `jakarta` namespace
* Fix WS & XML modules to use Jakarta EE
* `--add-opens` in some modules for their reflection-based tests
* Disable Kafka tests which does not work on Windows; see Apache Kafka `3.0.1`
* Upgrade to JUnit `5.8.1`
* Migrate JMS tests to Artemis
* Remove RMI module as it was deprecated before
* Fix `pr-build-workflow.yml` for Java 17
* Fix JavaDocs warnings using `Xdoclint:syntax` per module, not in the top-level `api` task
* Move docs for version `6.0`
* To align with the rest of Spring portfolio, based on Spring Framework 5.3.x generation,
it is better to upgrade to the Kotlin version supported by SF
* Fix `ClassUtils.isLambda()` to treat `$inlined$` classes as Kotlin lambdas and let
Java DSL to delegate to the `LambdaMessageProcessor` for better expected type handling.
Looks like Kotlin does not carry type info in its lambdas anymore (at least for functions).
* Add `-parameters` to the compiler args for possible method argument names discovery.
Essentially realign with the rest of portfolio
* Fix Kotlin tests for deprecated API
* GH-3554: Eval remote dir on each synchToLocal
Fixes https://github.com/spring-projects/spring-integration/issues/3554
The `remoteDirectoryExpression` was introduced into an
`AbstractInboundFileSynchronizer` to let end-user to evaluate a remote directory
on each poll (essentially on each `synchronizeToLocalDirectory()` call).
The fix for the https://jira.spring.io/browse/INT-4491 introduced a regression
when such an expression was evaluated only once when we call a `setRemoteDirectory()`.
So, an original purpose of this option was lost and we don't get an actual
remote dir on each poll
* Remove `evaluateRemoteDirectory()` method and its usage since it doesn't
reflect expectation of the remote dir expression property
* Reinstate the `remoteDirectoryExpression` evaluation in the
`synchronizeToLocalDirectory()`
* Propagate the result of that expression into further methods for copying
files
* Remove setting of the `remoteDirectory` variable into a global `EvaluationContext`
of the `AbstractInboundFileSynchronizer` instance since it is not thread-safe
* Instead create an `EvaluationContext` locally for each `synchronizeToLocalDirectory()`
call and set the `remoteDirectory` variable into this scoped instances
* Generate a local file name from the `localFilenameGeneratorExpression`
against locally created `EvaluationContext` with the mentioned `remoteDirectory` variable
* Cover the expected functionality with a unit-test
**Cherry-pick to `5.4.x` & `5.3.x`**
* * Fix `testRemoteDirectoryRefreshedOnEachSynchronization` according PR review
The `AbstractPersistentAcceptOnceFileListFilter`
extends now an `AbstractDirectoryAwareFileListFilter`
and makes all the implementors to implement its `isDirectory()` method.
This is a breaking change in the point release which makes implementations
outside of the core project not compilable
The regression after: https://github.com/spring-projects/spring-integration/issues/3488
Related fix in the Spring Cloud GCP: https://github.com/spring-cloud/spring-cloud-gcp/pull/2654
* Implement an `isDirectory()` in the `AbstractPersistentAcceptOnceFileListFilter`
to let other implementations do not worry during upgrade
**Cherry-pick to `5.4.x` & `5.3.x`**
* Introduce a `groupConditionSupplier` for MGS
* Add a `MessageGroup.condition` option
* Add a `MessageGroupStore.conditionSupplier` option
* Use it from the `SimpleMessageStore.addMessagesToGroup()` API
to populate `condition` (if any) into a `MessageGroup`
* Introduce a `GroupConditionProvider` contract to be implemented
on those `ReleaseStrategy` contracts which could be aware of group condition
* Populated a `GroupConditionProvider.getGroupConditionSupplier()`
into a `MessageGroupStore` from the `AbstractCorrelatingMessageHandler`
for end-user convenience
* Rework a `FileMarkerReleaseStrategy` to implement a `GroupConditionProvider`
to provide a function which produces a condition from `file_lineCount` header
of the `END` marker message
* Make the `FileMarkerReleaseStrategy` logic already based on the condition from a group
* Delegate `GroupConditionProvider` from the `FileAggregator`
* Add test for empty file aggregation
* * Implement `condition` in the `AbstractKeyValueMessageStore` and `MongoDbMessageStore`
* Test `condition` for `mongo-aggregator-config.xml` and `FileAggregatorTests` against GemFire
* * Implement `condition` in the `ConfigurableMongoDbMessageStore`
* * Implement `condition` in the `JdbcMessageStore`
* `FileAggregatorTests` against `JdbcMessageStore`
* Refactor `JdbcMessageStore` for better handling of message group metadata
* Remove unused `MARKED` column in the DDL in favor of newly introduced `CONDITION`
* * Add docs for message group condition
* * Move `conditionSupplier` option from MGS to AbstractCorrelatingMH
* Make it as a `BiFunction` to propagate existing condition alongside with the
message to consult
* Expose `groupConditionSupplier` in Java & XML DSLs
* * Fix language in docs
* INT-4116: Introduce FileAggregator
JIRA: https://jira.spring.io/browse/INT-4116
* Implement a `FileSplitter.FileMarker`-based aggregation strategies
and utilize them in a general `FileAggregator` component
* Make `HeaderAttributeCorrelationStrategy.attributeName` as `final`; add `Assert.notEmpty()`
* Fix `AggregatorFactoryBean` and `AggregatorSpec` to parse the provided processor
for possible `CorrelationStrategy` and/or `ReleaseStrategy`
* Introduce short-cut methods into Java & Kotlin DSL for an `aggregate()` configuration
* Introduce a `FileHeaders.LINE_COUNT` for header to be populated in the `FileSplitter`.
We need this info in the `FileAggregator` to avoid possible overhead with JSON deserialization
of the `FileSplitter.FileMarker` messages
* Test and document the feature
* Improve `FileSplitter` doc for code block switch (tabs)
* * Rework FileAggregator do not use Java Streams
Fixes https://github.com/spring-projects/spring-integration/issues/3507
The `OSDelegatingFileTailingMessageProducer` passing command string to `Runtime.getRuntime().exec()`
may cause problems if spaces (and other special characters) are used in the filename.
* Use an array for command and its options to let the target `Runtime` to parse and
execute it properly
**Cherry-pick to 5.4.x, 5.3.x & 5.2.x**
* Use a tmp file in the `FileTests.testFileReadingFlow()`
for writing content and then rename it to the target file.
It looks like we may already have a file just with a short content
and it is picked up by the watch service for processing in the flow
* Rework `AggregatorWithCustomReleaseStrategyTests` to JUnit 5 and remove
extra looping logic since it just does not add any extra coverage just
performance overhead via crating and destroying the same application ctx
* Rework `ChannelAdapterParserTests` to `@SpringJUnitConfig` managed test
* FileInbound DSL: Add `recursive` for convenience
Related to https://stackoverflow.com/questions/66171881/how-to-read-nested-txt-file-from-spring-integration-file
The `FileInboundChannelAdapterSpec` can be configured with an external `DirectoryScanner`,
but it sometimes becomes burden for end-users to extract a scanner bean and configure it
with all the required file filters
* Expose a `recursive(boolean)` option for better end-user experience
* Rework `FileTests` for JUnit 5
* Mention a new option in the docs
* * Restore accidentally removed code
* Restore special symbols in the `FileTests`
* Fix language in the docs according review
Resolves https://github.com/spring-projects/spring-integration/issues/3488
Resolves two problems:
- When changes are made deep in the directory tree, they were not detected because
the directory is in the metadata store and only passes the filter if a file
immediately under it is changed, changing the directory's timestamp.
This is solved by subclassing `AbstractDirectoryAwareFileListFilter`, allowing its
`alwaysAcceptDirectories` property to be set.
- Only the filename was used as a metadata key; causing problems if a file with the
same name appears multiple times in the tree.
This is solved with a new property on `AbstractDirectoryAwareFileListFilter` used by
the gateways to determine whether to filter the raw file names returned by the session
(previous behavior) or the full path relative to the root directory.
**cherry-pick to 5.4.x, 5.3.x**
* Some code style clean up
* GH-3370: Remove synchronized from RemoteFileUtils
Fixes https://github.com/spring-projects/spring-integration/issues/3370
The `synchronized` on the `RemoteFileUtils.makeDirectories()` makes an application too
slow, especially when we deal with different paths in different sessions
* Remove the `synchronized` from that method and rework `SftpSession.mkdir()`
to return `false` when "A file cannot be created if it already exists" exception
is thrown from the server.
Essentially make an `exists()` call to be sure that an exception is really related
to "file-already-exists" answer from the server
**Cherry-pick to 5.3.x, 5.2.x & 4.3.x**
* * Re-throw an exception in the `SftpSession.mkdir()`
when error code is not `4` or remote dir does not exist
* * Check `session.mkdir()` result in the
`RemoteFileUtils` to throw an `IOException` when `false`
* * Fix mock test to return `true` for `mkdir` instead of `null`
* Implement `Emission` handling in the `FluxMessageChannel` and `IntegrationReactiveUtils`
* Upgrade to Spring Kafka `2.6.0`
* Fix R2DBC components for deprecation in the Spring Data R2DBC
* Implement `StatementMapper.SelectSpec` for query expression
* Clean up for some sporadic test failures
Resolves https://github.com/spring-projects/spring-integration/issues/3315
`maxFetchSize` ignored with filters supporting single file filtering (default).
This breaks "fair" rotation with the `RotatingServerAdvice`.
Honor `maxFetchSize`, even with filters that support single file filtering.
**cherry-pick to 5.3.x, 5.2.x**
* Support Java 14
* Provide changes to avoid deprecated Java API
and have a compatibility back to Java 8
* Change affected test classes to JUnit 5 whenever it is possible
* Ignore/Disable some TCP/IP tests which don't pass on Java 14
* Fix (some) TCP tests on JRE 14
* Fix SSL Handshake test - client side handshake is successful with java 14
- change the badClient cert to a badServer cert to force an error on the client side
Co-authored-by: artembilan <raven666>
Co-authored-by: Gary Russell <grussell@pivotal.io>
Fixes https://github.com/spring-projects/spring-integration/issues/3271
When exception happens at `.withPayload(session.readRaw(remotePath))`
in the `AbstractRemoteFileStreamingMessageSource` we don't close session.
The resource leaking happens in the caching session factory
* Add `session.close();` into the `catch (IOException e) {`
in the `AbstractRemoteFileStreamingMessageSource.doReceive()`
to clean up resources properly
**Cherry-pick to 5.2.x, 5.1.x & 4.3.x**
* Use a SimpleJsonSerializer in the FileSplitter
* To avoid extra dependency for Jackson when we
serialize `FileSplitter.FileMarker` to JSON, use
a `SimpleJsonSerializer` instead.
* Fix `SimpleJsonSerializer` to escape a `\` symbol from property values
since it is used for quoting string values in the final JSON
* * Document the change
* Increase latch timeouts for `RedisLockRegistryLeaderInitiatorTests`
* Some RMI tests fail with different outcome because an RMI registry might be
available on default `1099` port.
Fix them to not rely on that port
* GH-2996: Add Resource for remote file transfer
Fixes https://github.com/spring-projects/spring-integration/issues/2996
* Fix `RemoteFileTemplate.send()` to support a `Resource` payload
for remote file transferring content
* Code style clean up for `RemoteFileTemplate`
* Remove `volatile` for configuration properties for better performance
* Change a `charset` to the `Charset` for only once conversion from string
during configuration phase
* Fix (S)FTP tests for new functionality
* Change affected tests to JUnit 5
* Document a new feature; mention all the supported types and `FileExistsMode` constants
* * Fix language in `whats-new.adoc`
Fixes https://github.com/spring-projects/spring-integration/issues/3169
All other `DefaultSessionFactoryLocator` contracts are based on the `Object`,
so this `addSessionFactor`y has to be on `Object` as well.
* Add `DefaultSessionFactoryLocator.addSessionFactory(Object key, SessionFactory<F> factory)`
* Deprecate existing one based on `String`
* Fix tests do no use a deprecated API
* Some other code style clean up in the affected classes
**Cherry-pick to 5.2.x**
Fixes https://github.com/spring-projects/spring-integration/issues/3132
It turns out that Checkstyle EmptyBlock doesn't complain about
empty default ctor.
Plus a new check for `super();` call treats it as a violation
* Remove `super();` from all the no-arg ctors
* Code style clean up in the affected classes according
IDEA suggestions
* Fix new Sonar smells
* INT-3494: Resolve dir for writing as a Resource
JIRA: https://jira.spring.io/browse/INT-3494
The expression for local directory can be resolved into a `Resource`
or resource location.
* Fix `ExpressionUtils.expressionToFile()` to support `Resource` and
also use `ResourceUtils.getFile(path)` when expression result is a string
* Modify tests to ensure that resource is resolved properly
* Upgrade affected tests to JUnit 5
* Mention an new functionality in docs
* * Improve Java doc for `ExpressionUtils.expressionToFile()`
* Finish the sentence in the `file.adoc`
Fixes https://github.com/spring-projects/spring-integration/issues/3105
The loop in the `AbstractRemoteFileStreamingMessageSource` doesn't check
the next polled file for filtering.
* Add `continue;` when we filter the current file and poll the next one.
So, we go over into the `while()` beginning
* Clean up the code style and cover more code path in test
Fixesspring-projects/spring-integration#3076
* Return `null` instead of throwing an exception
* Fix license header
* Add missing `@since`
* Avoid over-engineered test logic
* Fix code style
* Test non-existent file does not get created
* Fix: remove unused import
Fixes https://github.com/spring-projects/spring-integration/issues/3038
* Add more delegating setters into the `AbstractRemoteFileOutboundGateway`
for its `RemoteFileTemplate` property
* Restrict externally provided `RemoteFileTemplate` from modifications
in the `AbstractRemoteFileOutboundGateway`: those options must be
configured on that external `RemoteFileTemplate`
* Expose new options in the Java DSL specs
* GH-3043: Add FileHeaders.REMOTE_HOST header
Fixes https://github.com/spring-projects/spring-integration/issues/3043
* Populate a `FileHeaders.REMOTE_HOST` from the
`AbstractRemoteFileStreamingMessageSource` and "get"-based commands
in the `AbstractRemoteFileOutboundGateway`
* Extract the value from the a `Session.getHost()` contract
* The `AbstractInboundFileSynchronizingMessageSource` cannot be
addressed with this because the real message is already based on the
locally stored file
* Adjust some affected tests according our code style requirements
* * Add remote file info support into `AbstractInboundFileSynchronizingMessageSource`
* Introduce a `MetadataStore` functionality into the `AbstractInboundFileSynchronizer`
to gather a remote file info an save it in the URI style against local file
* Retrieve such an info in the `AbstractInboundFileSynchronizingMessageSource`
during local file polling
* Introduce `protocol()` contract for the `AbstractInboundFileSynchronizer`
to build a proper URI in the metadata for external readers to distinguish
remote files properly
* Document the feature
* * Fix some typos in Docs
* * Rename property and header constant to the `HOST_PORT` pair
* Fix typos in Docs
* Add `remote-file-metadata-store` and `metadata-store-prefix` into XSD
of (S)FTP Inbound Channel Adapters
* Add `remoteFileMetadataStore` and `metadataStorePrefix` options
into `RemoteFileInboundChannelAdapterSpec` for Java DSL
Fixes https://github.com/spring-projects/spring-integration/issues/2748
* Refactor more `MessageHandlingException`s to include `this` into an
exception message
* Revert using `MessagingException` in some places which really are not
about messaging.
This helps to wrap them into `MessageHandlingException` later in the
`MessageHandler` for the `BeanDefinition` reference
* Remove `volatile` from configuration properties in the affected
classes
* Remove already deprecated `JmsOutboundGateway.setPriority()`
* Add `resource` and `source` for `BeanDefinition` in the
`AbstractChannelAdapterParser` & `AbstractInboundGatewayParser`
* Document the feature
