Related to https://build.spring.io/browse/INT-MAIN-84/
The `lock.unlock()` must be called in the `finally` block of the
nested `try..catch`, not in the outer which may just fail on `lock.lockInterruptibly()`
in which case there is just not going to be anything we can `unlock()` in the end
**Cherry-pick to `5.4.x` & `5.3.x`**
* GH-3554: Eval remote dir on each synchToLocal
Fixes https://github.com/spring-projects/spring-integration/issues/3554
The `remoteDirectoryExpression` was introduced into an
`AbstractInboundFileSynchronizer` to let end-user to evaluate a remote directory
on each poll (essentially on each `synchronizeToLocalDirectory()` call).
The fix for the https://jira.spring.io/browse/INT-4491 introduced a regression
when such an expression was evaluated only once when we call a `setRemoteDirectory()`.
So, an original purpose of this option was lost and we don't get an actual
remote dir on each poll
* Remove `evaluateRemoteDirectory()` method and its usage since it doesn't
reflect expectation of the remote dir expression property
* Reinstate the `remoteDirectoryExpression` evaluation in the
`synchronizeToLocalDirectory()`
* Propagate the result of that expression into further methods for copying
files
* Remove setting of the `remoteDirectory` variable into a global `EvaluationContext`
of the `AbstractInboundFileSynchronizer` instance since it is not thread-safe
* Instead create an `EvaluationContext` locally for each `synchronizeToLocalDirectory()`
call and set the `remoteDirectory` variable into this scoped instances
* Generate a local file name from the `localFilenameGeneratorExpression`
against locally created `EvaluationContext` with the mentioned `remoteDirectory` variable
* Cover the expected functionality with a unit-test
**Cherry-pick to `5.4.x` & `5.3.x`**
* * Fix `testRemoteDirectoryRefreshedOnEachSynchronization` according PR review
The `AbstractPersistentAcceptOnceFileListFilter`
extends now an `AbstractDirectoryAwareFileListFilter`
and makes all the implementors to implement its `isDirectory()` method.
This is a breaking change in the point release which makes implementations
outside of the core project not compilable
The regression after: https://github.com/spring-projects/spring-integration/issues/3488
Related fix in the Spring Cloud GCP: https://github.com/spring-cloud/spring-cloud-gcp/pull/2654
* Implement an `isDirectory()` in the `AbstractPersistentAcceptOnceFileListFilter`
to let other implementations do not worry during upgrade
**Cherry-pick to `5.4.x` & `5.3.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/3507
The `OSDelegatingFileTailingMessageProducer` passing command string to `Runtime.getRuntime().exec()`
may cause problems if spaces (and other special characters) are used in the filename.
* Use an array for command and its options to let the target `Runtime` to parse and
execute it properly
**Cherry-pick to 5.4.x, 5.3.x & 5.2.x**
# Conflicts:
# spring-integration-file/src/main/java/org/springframework/integration/file/tail/OSDelegatingFileTailingMessageProducer.java
Resolves https://github.com/spring-projects/spring-integration/issues/3488
Resolves two problems:
- When changes are made deep in the directory tree, they were not detected because
the directory is in the metadata store and only passes the filter if a file
immediately under it is changed, changing the directory's timestamp.
This is solved by subclassing `AbstractDirectoryAwareFileListFilter`, allowing its
`alwaysAcceptDirectories` property to be set.
- Only the filename was used as a metadata key; causing problems if a file with the
same name appears multiple times in the tree.
This is solved with a new property on `AbstractDirectoryAwareFileListFilter` used by
the gateways to determine whether to filter the raw file names returned by the session
(previous behavior) or the full path relative to the root directory.
**cherry-pick to 5.4.x, 5.3.x**
* Some code style clean up
# Conflicts:
# spring-integration-file/src/main/java/org/springframework/integration/file/remote/gateway/AbstractRemoteFileOutboundGateway.java
# src/reference/asciidoc/whats-new.adoc
* Rely on `MProducerSupport.active` for `Flux`
* Fix `MessageProducerSupport` to extract an `active` flag and set it before
`isRunning` - the `Flux` subscription relies on the `takeWhile()`
where in case of `autoStartup = false` we will never start consume because
it is set to `true` already after `doStart()`
* Refactor all the `MessageProducerSupport` implementation with similar
`active` state to use already one from the super class
**Cherry-pick to 5.3.x**
* * Remove `MessageProducerSupport.setActive()`
to not let to mutate it from the implementations
* Set `active` to `false` in the `destroy()`
* Clean up and fix typos in the affected `JmsMessageDrivenEndpoint`
* * Pull `active` flag down to the `AbstractEndpoint`
* Set `active = true` in the `start()` before calling `doStart()`
* Do same for `active = false` in the `stop()`
* Clean up `AbstractEndpoint` impls to not call `doStart/doStop` for nothing
* Refactor endpoints to rely on the `active` state from the `AbstractEndpoint`
not their own
Resolves https://github.com/spring-projects/spring-integration/issues/3395
Outbound remote file gateway parser requires `expression` even though some
commands don't need or use it.
* Propagate the empty string for `expression` attribute
* Fix default (`payload`) expression logic in the `AbstractRemoteFileOutboundGateway`
**Cherry-pick to `5.3.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/3373
The `AbstractInboundFileSynchronizer` doesn't consider that
`hostPort` from `Session` could be in an IPv6 syntax
* Parse the `hostPort` from `Session` in a manner that only the last
`:` is treated as a port delimiter
**Cherry-pick to 5.3.x & 5.2.x**
* GH-3370: Remove synchronized from RemoteFileUtils
Fixes https://github.com/spring-projects/spring-integration/issues/3370
The `synchronized` on the `RemoteFileUtils.makeDirectories()` makes an application too
slow, especially when we deal with different paths in different sessions
* Remove the `synchronized` from that method and rework `SftpSession.mkdir()`
to return `false` when "A file cannot be created if it already exists" exception
is thrown from the server.
Essentially make an `exists()` call to be sure that an exception is really related
to "file-already-exists" answer from the server
**Cherry-pick to 5.3.x, 5.2.x & 4.3.x**
* * Re-throw an exception in the `SftpSession.mkdir()`
when error code is not `4` or remote dir does not exist
* * Check `session.mkdir()` result in the
`RemoteFileUtils` to throw an `IOException` when `false`
* * Fix mock test to return `true` for `mkdir` instead of `null`
Close the pool so that any sessions returned after the factory is
`destroy()`ed are closed.
* Call `removeAllIdleItems()` in `close()`.
* Close sessions in `SftpStreamingMessageSourceTests`.
**cherry-pick to all supported branches**
Resolves https://github.com/spring-projects/spring-integration/issues/3315
`maxFetchSize` ignored with filters supporting single file filtering (default).
This breaks "fair" rotation with the `RotatingServerAdvice`.
Honor `maxFetchSize`, even with filters that support single file filtering.
**cherry-pick to 5.3.x, 5.2.x**
Fixes https://github.com/spring-projects/spring-integration/issues/3271
When exception happens at `.withPayload(session.readRaw(remotePath))`
in the `AbstractRemoteFileStreamingMessageSource` we don't close session.
The resource leaking happens in the caching session factory
* Add `session.close();` into the `catch (IOException e) {`
in the `AbstractRemoteFileStreamingMessageSource.doReceive()`
to clean up resources properly
**Cherry-pick to 5.2.x, 5.1.x & 4.3.x**
* Introduce a `ReceiveMessageAdvice`
* Deprecate an `AbstractMessageSourceAdvice` in favor of
`default` method in the `MessageSourceMutator`
* Move a `applyReceiveOnlyAdviceChain()` logic into the `AbstractPollingEndpoint`:
now both `PollingConsumer` and `SourcePollingChannelAdapter` can use
`ReceiveMessageAdvice`
* Introduce a `SimpleActiveIdleReceiveMessageAdvice` based already
on the `ReceiveMessageAdvice` and deprecate a `SimpleActiveIdleMessageSourceAdvice`
which is fully replaceable with newly introduced `SimpleActiveIdleReceiveMessageAdvice`
* Add `@SuppressWarnings("deprecation")` for those out-of-the-box `ReceiveMessageAdvice`
implementation which still use an `AbstractMessageSourceAdvice` for
backward compatibility
* Document a new feature and give the `MessageSourceMutator` a new meaning
* * Fix language in the `polling-consumer.adoc`
* Use a SimpleJsonSerializer in the FileSplitter
* To avoid extra dependency for Jackson when we
serialize `FileSplitter.FileMarker` to JSON, use
a `SimpleJsonSerializer` instead.
* Fix `SimpleJsonSerializer` to escape a `\` symbol from property values
since it is used for quoting string values in the final JSON
* * Document the change
Fixes: https://github.com/spring-projects/spring-integration/issues/3249
When the `CachingSessionFactory` is configured with small enough pool
and it is very likely that dead lock may happen when `RemoteFileTemplate.send()`
is used.
The problem happens when we reach the `RemoteFileTemplate.exists()` call
which is done from the internal method called from already pulled from cache
`Session`
* Fix `RemoteFileTemplate` to use a `session.exists()` instead on the provided
into the method `Session`
* Demonstrate the problem in the `SftpRemoteFileTemplateTests.testNoDeadLockOnSend()`
**Cherry-pick to 5.2.x, 5.1.x & 4.3.x**
* Increase latch timeouts for `RedisLockRegistryLeaderInitiatorTests`
* Some RMI tests fail with different outcome because an RMI registry might be
available on default `1099` port.
Fix them to not rely on that port
* GH-2996: Add Resource for remote file transfer
Fixes https://github.com/spring-projects/spring-integration/issues/2996
* Fix `RemoteFileTemplate.send()` to support a `Resource` payload
for remote file transferring content
* Code style clean up for `RemoteFileTemplate`
* Remove `volatile` for configuration properties for better performance
* Change a `charset` to the `Charset` for only once conversion from string
during configuration phase
* Fix (S)FTP tests for new functionality
* Change affected tests to JUnit 5
* Document a new feature; mention all the supported types and `FileExistsMode` constants
* * Fix language in `whats-new.adoc`
* GH-3155: Add support for Java DSL extensions
Fixes https://github.com/spring-projects/spring-integration/issues/3155
Provide an `IntegrationFlowExtension` for possible custom EI-operators
in the target project use-cases.
* * Move `IntegrationFlowExtension` tests ot its own test class
* Make all the `IntegrationComponentSpec` ctors as `protected` for possible custom extensions
* Make some `BaseIntegrationFlowDefinition` methods and properties as `protected` to get them
access from the `IntegrationFlowExtension` implementations
* Document the feature
* * Fix language and typos in docs
* * Add `protected` to one more `GatewayEndpointSpec` ctor
* Add JavaDocs to `GatewayEndpointSpec` methods
* * Add `protected` to one more `JmsPollableMessageChannelSpec` ctor
Fixes https://github.com/spring-projects/spring-integration/issues/3169
All other `DefaultSessionFactoryLocator` contracts are based on the `Object`,
so this `addSessionFactor`y has to be on `Object` as well.
* Add `DefaultSessionFactoryLocator.addSessionFactory(Object key, SessionFactory<F> factory)`
* Deprecate existing one based on `String`
* Fix tests do no use a deprecated API
* Some other code style clean up in the affected classes
**Cherry-pick to 5.2.x**
* Remove Boon dependency and its usage
* Remove overloaded methods from the `IntegrationFlowDefinition`
- we can simply rely now on the super class
* Remove (or rework) deprecated entities in the docs
* Fix tests for removed deprecated APIs
* Rework affected tests to JUnit 5
Fixes https://github.com/spring-projects/spring-integration/issues/3132
It turns out that Checkstyle EmptyBlock doesn't complain about
empty default ctor.
Plus a new check for `super();` call treats it as a violation
* Remove `super();` from all the no-arg ctors
* Code style clean up in the affected classes according
IDEA suggestions
* Fix new Sonar smells
* Add `IntegrationPattern` contract to implement on the target components
which represent particular EIP
* Add `IntegrationPatternType` with an internal `IntegrationPatternCategory`
to return from the component implementing `IntegrationPattern`
* Parse `IntegrationPatternType` in the `IntegrationNode` for potential
use on the UI for drawing a particular icon
* More pattern representations
* Clean up Checkstyle
* Fix JavaDocs
* Add `integrationPatternCategory` assertion into the `IntegrationGraphServerTests`
* Add more IntegrationPattern implementations
* Provide some delegation and overriding logic whenever we have
components wrapping
* Fix unused imports
* Add `inbound_gateway` pattern indicator
* Add conditional on `expectReply` to indicate a component as an
`IntegrationPatternType.outbound_channel_adapter` or
`IntegrationPatternType.outbound_gateway`
* Make some code clean up in affected classes
* Add a `gateway` type for `@MessagingGateway`
* Comment the reason for `outbound_gateway` type in the `AbstractReplyProducingMessageHandler`
* Bump `IntegrationGraphServer.GRAPH_VERSION`
* Add new attributes into graph sample in the `graph.adoc`
* Document an `IntegrationPattern`
* Apply changes for version 5.3
* Rebased into `5.3-WIP`
* Add a `whats-new.adoc` note about an `IntegrationPattern`
* Move `What's New` into `changes-5.1-5.2.adoc`
* Remove version from the `XSD` files:
1. Align with SF
2. The version for XSD doesn't matter for the current jar version,
since only the version is available in classpath is from the current
jar version
3. Modify `spring.schemas` to map all the possible versions for XSD
into the current one in a jar.
This way target applications can upgrade without changing the version
for XSD location and we don't need to require `versionless` variant
any more
4. The jar in classpath can handle only its own XSD, so independently
of version or no-version variants we still may fail because the current
jar may not support end-user code any more - need some fix over there
anyway.
* Remove `checkTestConfigs` Gradle task since we don't worry about XSD
version any more
* Modify `schemaZip` Gradle task to populate XSDs into a distribution
still with a version to avoid overriding on-line XSDs for version `1.0.x`.
We may consider not do that since SF doesn't and just bite a bullet
for always overriding those on-line XSDs to the actual one from the latest
release.
* INT-3494: Resolve dir for writing as a Resource
JIRA: https://jira.spring.io/browse/INT-3494
The expression for local directory can be resolved into a `Resource`
or resource location.
* Fix `ExpressionUtils.expressionToFile()` to support `Resource` and
also use `ResourceUtils.getFile(path)` when expression result is a string
* Modify tests to ensure that resource is resolved properly
* Upgrade affected tests to JUnit 5
* Mention an new functionality in docs
* * Improve Java doc for `ExpressionUtils.expressionToFile()`
* Finish the sentence in the `file.adoc`
Fixes https://github.com/spring-projects/spring-integration/issues/3105
The loop in the `AbstractRemoteFileStreamingMessageSource` doesn't check
the next polled file for filtering.
* Add `continue;` when we filter the current file and poll the next one.
So, we go over into the `while()` beginning
* Clean up the code style and cover more code path in test
Fixes https://github.com/spring-projects/spring-integration/issues/3026
**Cherry-pick to `5.1.x`**
* Populate proper `FileTransferringMessageHandler` impl from DSL spec
implementations.
This way we are able to use a provided `chmod` from Java DSL
* Added `FileTransferringMessageHandlerSpec` ctor TODO
* Update SftpTests
* Code cleanup; `@Ignore` `SftpTests.testSftpOutboundFlowWithChmod()`
since it doesn't work properly on Windows
Fixesspring-projects/spring-integration#3076
* Return `null` instead of throwing an exception
* Fix license header
* Add missing `@since`
* Avoid over-engineered test logic
* Fix code style
* Test non-existent file does not get created
* Fix: remove unused import
Fixes https://github.com/spring-projects/spring-integration/issues/3038
* Add more delegating setters into the `AbstractRemoteFileOutboundGateway`
for its `RemoteFileTemplate` property
* Restrict externally provided `RemoteFileTemplate` from modifications
in the `AbstractRemoteFileOutboundGateway`: those options must be
configured on that external `RemoteFileTemplate`
* Expose new options in the Java DSL specs
* GH-3043: Add FileHeaders.REMOTE_HOST header
Fixes https://github.com/spring-projects/spring-integration/issues/3043
* Populate a `FileHeaders.REMOTE_HOST` from the
`AbstractRemoteFileStreamingMessageSource` and "get"-based commands
in the `AbstractRemoteFileOutboundGateway`
* Extract the value from the a `Session.getHost()` contract
* The `AbstractInboundFileSynchronizingMessageSource` cannot be
addressed with this because the real message is already based on the
locally stored file
* Adjust some affected tests according our code style requirements
* * Add remote file info support into `AbstractInboundFileSynchronizingMessageSource`
* Introduce a `MetadataStore` functionality into the `AbstractInboundFileSynchronizer`
to gather a remote file info an save it in the URI style against local file
* Retrieve such an info in the `AbstractInboundFileSynchronizingMessageSource`
during local file polling
* Introduce `protocol()` contract for the `AbstractInboundFileSynchronizer`
to build a proper URI in the metadata for external readers to distinguish
remote files properly
* Document the feature
* * Fix some typos in Docs
* * Rename property and header constant to the `HOST_PORT` pair
* Fix typos in Docs
* Add `remote-file-metadata-store` and `metadata-store-prefix` into XSD
of (S)FTP Inbound Channel Adapters
* Add `remoteFileMetadataStore` and `metadataStorePrefix` options
into `RemoteFileInboundChannelAdapterSpec` for Java DSL
Related to https://github.com/spring-projects/spring-integration/issues/3027
* Remove `Assert.isTrue()` from the `onRotation()` to make the behavior
as void in case of non-standard `MessageSource` provided
* Add JavaDocs into `onRotation()` to describe the behavior and
possibility to override
Fixes https://github.com/spring-projects/spring-integration/issues/3027
This is a polishing for PR https://github.com/spring-projects/spring-integration/pull/3029
* Move `KeyDirectory` in to the `RotationPolicy` since it looks like fully
coupled concept with that abstraction and the class name is so bad for
the public API
* Remove `AbstractStandardRotationPolicy` in favor of a `StandardRotationPolicy`
extendability
* Change `ftp.adoc` and `sftp.adoc` to reflect a new API reality
* Mantion these changes in the `whats-new.adoc`
