Fixes https://github.com/spring-projects/spring-integration/issues/3656
* Move `JavaUtils` from `util` package to the root one to break any possible
tangling to/from other packages.
* Deprecate an existing `JavaUtils` for backward compatibility
* Remove `@IntegrationConverter` from the `JsonNodeWrapperToJsonNodeConverter`
and register it manually in the `ConverterRegistrar`.
The bean registration for the `JsonNodeWrapperToJsonNodeConverter` must be removed
in the next `6.0`
* Remove usage of `IntegrationContextUtils` from the `support` package
* GH-3647: Use remoteDirExpression in MV command
Fixes https://github.com/spring-projects/spring-integration/issues/3647
To simplify a source and renameTo remote file expressions, the `remoteDirectoryExpression`
is consulted now, when they are not full paths.
This is useful when we want just to rename a remote file in some dir
* * Add JavaDoc for `getDirectoryExpressionProcessor()`
* Fix language in docs
Resolves https://github.com/spring-projects/spring-integration/issues/3488
Resolves two problems:
- When changes are made deep in the directory tree, they were not detected because
the directory is in the metadata store and only passes the filter if a file
immediately under it is changed, changing the directory's timestamp.
This is solved by subclassing `AbstractDirectoryAwareFileListFilter`, allowing its
`alwaysAcceptDirectories` property to be set.
- Only the filename was used as a metadata key; causing problems if a file with the
same name appears multiple times in the tree.
This is solved with a new property on `AbstractDirectoryAwareFileListFilter` used by
the gateways to determine whether to filter the raw file names returned by the session
(previous behavior) or the full path relative to the root directory.
**cherry-pick to 5.4.x, 5.3.x**
* Some code style clean up
* More GH Actions goodies
* Add RabbitMQ, MongoDB & Redis services into GH actions
* Remove Travis
* * Add `distributionSha256Sum` into `gradle-wrapper.properties`
* Try `TERM: dumb` to see difference for Gradle output in GH action
* Add `-S` to see the reason of test failure
* * Try MQTT plugin for RabbitMQ image
* * Try `-i` for test failure in Gradle on Actions
* * Use `eclipse-mosquitto` image for MQTT on actions
* Fix `FtpServerOutboundTests` for the proper FTP file list
* Output file list for the failed session in the `RotatingServersTests`
* * Adjust SOUT in the test for Checkstyle
* * Make some change to `RotatingServersTests`.
Looks like the order for `@BeforeAll` methods is not determined
and we may start to interaction with FTP server which is not
started yet
* Fix SSL Handshake Test
- we already made this change for the NIO test, but the NET test is failing the same
way - the cert failure switched to the server so the expected exception was not
thrown on the client side. Due to JVM changes.
* * Fix SFTP tests for wrong host and auto-startup state
* Remove `-i` for Gradle in the GH Actions
* * Try `--no-parallel` for Gradle in the GH Actions
* * Try GH actions without Mosquitto
* * Try `cyrilix/rabbitmq-mqtt` for GH Actions
Co-authored-by: Gary Russell <grussell@pivotal.io>
Resolves https://github.com/spring-projects/spring-integration/issues/3395
Outbound remote file gateway parser requires `expression` even though some
commands don't need or use it.
* Propagate the empty string for `expression` attribute
* Fix default (`payload`) expression logic in the `AbstractRemoteFileOutboundGateway`
**Cherry-pick to `5.3.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/3373
The `AbstractInboundFileSynchronizer` doesn't consider that
`hostPort` from `Session` could be in an IPv6 syntax
* Parse the `hostPort` from `Session` in a manner that only the last
`:` is treated as a port delimiter
**Cherry-pick to 5.3.x & 5.2.x**
* GH-3370: Remove synchronized from RemoteFileUtils
Fixes https://github.com/spring-projects/spring-integration/issues/3370
The `synchronized` on the `RemoteFileUtils.makeDirectories()` makes an application too
slow, especially when we deal with different paths in different sessions
* Remove the `synchronized` from that method and rework `SftpSession.mkdir()`
to return `false` when "A file cannot be created if it already exists" exception
is thrown from the server.
Essentially make an `exists()` call to be sure that an exception is really related
to "file-already-exists" answer from the server
**Cherry-pick to 5.3.x, 5.2.x & 4.3.x**
* * Re-throw an exception in the `SftpSession.mkdir()`
when error code is not `4` or remote dir does not exist
* * Check `session.mkdir()` result in the
`RemoteFileUtils` to throw an `IOException` when `false`
* * Fix mock test to return `true` for `mkdir` instead of `null`
Close the pool so that any sessions returned after the factory is
`destroy()`ed are closed.
* Call `removeAllIdleItems()` in `close()`.
* Close sessions in `SftpStreamingMessageSourceTests`.
**cherry-pick to all supported branches**
Fixes: https://github.com/spring-projects/spring-integration/issues/3249
When the `CachingSessionFactory` is configured with small enough pool
and it is very likely that dead lock may happen when `RemoteFileTemplate.send()`
is used.
The problem happens when we reach the `RemoteFileTemplate.exists()` call
which is done from the internal method called from already pulled from cache
`Session`
* Fix `RemoteFileTemplate` to use a `session.exists()` instead on the provided
into the method `Session`
* Demonstrate the problem in the `SftpRemoteFileTemplateTests.testNoDeadLockOnSend()`
**Cherry-pick to 5.2.x, 5.1.x & 4.3.x**
* GH-3247: Fix `SftpSession.exists` for error code
Fixes https://github.com/spring-projects/spring-integration/issues/3247
When there is no path on the SFTP server, a `ChannelSftp.SSH_FX_NO_SUCH_FILE`
error is returned in the thrown `SftpException`.
* Fix `SftpSession.exists()` to check for the `SSH_FX_NO_SUCH_FILE` to
return `false` and re-throw an exception otherwise
* Add mock test for `SftpSession.exists()`
* Add `org.mockito.AdditionalMatchers` to `checkstyle.xml` exclusions
**Cherry-pick to 5.2.x & 5.1.x**
* * Add exists tests against Mina embedded server
* GH-2996: Add Resource for remote file transfer
Fixes https://github.com/spring-projects/spring-integration/issues/2996
* Fix `RemoteFileTemplate.send()` to support a `Resource` payload
for remote file transferring content
* Code style clean up for `RemoteFileTemplate`
* Remove `volatile` for configuration properties for better performance
* Change a `charset` to the `Charset` for only once conversion from string
during configuration phase
* Fix (S)FTP tests for new functionality
* Change affected tests to JUnit 5
* Document a new feature; mention all the supported types and `FileExistsMode` constants
* * Fix language in `whats-new.adoc`
* GH-3211: Add DefSftpSF.setKnownHosts(Resource)
Fixes https://github.com/spring-projects/spring-integration/issues/3211
* Add `DefaultSftpSessionFactory.setKnownHosts(Resource)` to
allow to configure externally any resource for file with known_hosts content
* Deprecate an existing method in favor of new one
* The new method makes it aligned with the `setPrivateKey(Resource)`
* Fix tests do not use a deprecated method any more
**Cherry-pick to 5.2.x**
* * Rename to `setKnownHostsResource()` to avoid XML parser confusion
* Change `sftp.adoc` to reflect a new property
* GH-3155: Add support for Java DSL extensions
Fixes https://github.com/spring-projects/spring-integration/issues/3155
Provide an `IntegrationFlowExtension` for possible custom EI-operators
in the target project use-cases.
* * Move `IntegrationFlowExtension` tests ot its own test class
* Make all the `IntegrationComponentSpec` ctors as `protected` for possible custom extensions
* Make some `BaseIntegrationFlowDefinition` methods and properties as `protected` to get them
access from the `IntegrationFlowExtension` implementations
* Document the feature
* * Fix language and typos in docs
* * Add `protected` to one more `GatewayEndpointSpec` ctor
* Add JavaDocs to `GatewayEndpointSpec` methods
* * Add `protected` to one more `JmsPollableMessageChannelSpec` ctor
Fixes https://github.com/spring-projects/spring-integration/issues/3132
It turns out that Checkstyle EmptyBlock doesn't complain about
empty default ctor.
Plus a new check for `super();` call treats it as a violation
* Remove `super();` from all the no-arg ctors
* Code style clean up in the affected classes according
IDEA suggestions
* Fix new Sonar smells
* Move `What's New` into `changes-5.1-5.2.adoc`
* Remove version from the `XSD` files:
1. Align with SF
2. The version for XSD doesn't matter for the current jar version,
since only the version is available in classpath is from the current
jar version
3. Modify `spring.schemas` to map all the possible versions for XSD
into the current one in a jar.
This way target applications can upgrade without changing the version
for XSD location and we don't need to require `versionless` variant
any more
4. The jar in classpath can handle only its own XSD, so independently
of version or no-version variants we still may fail because the current
jar may not support end-user code any more - need some fix over there
anyway.
* Remove `checkTestConfigs` Gradle task since we don't worry about XSD
version any more
* Modify `schemaZip` Gradle task to populate XSDs into a distribution
still with a version to avoid overriding on-line XSDs for version `1.0.x`.
We may consider not do that since SF doesn't and just bite a bullet
for always overriding those on-line XSDs to the actual one from the latest
release.
Fixes https://github.com/spring-projects/spring-integration/issues/3026
**Cherry-pick to `5.1.x`**
* Populate proper `FileTransferringMessageHandler` impl from DSL spec
implementations.
This way we are able to use a provided `chmod` from Java DSL
* Added `FileTransferringMessageHandlerSpec` ctor TODO
* Update SftpTests
* Code cleanup; `@Ignore` `SftpTests.testSftpOutboundFlowWithChmod()`
since it doesn't work properly on Windows
Related to: https://build.spring.io/browse/INT-MASTERSPRING40-826/
If we don't close application context on the embedded (S)FTP server,
we my still have some session opened in other tests, which would cause
some unexpected behavior like extra `SessionClosedEvent` in the
test for FTP events
Fixes https://github.com/spring-projects/spring-integration/issues/3038
* Add more delegating setters into the `AbstractRemoteFileOutboundGateway`
for its `RemoteFileTemplate` property
* Restrict externally provided `RemoteFileTemplate` from modifications
in the `AbstractRemoteFileOutboundGateway`: those options must be
configured on that external `RemoteFileTemplate`
* Expose new options in the Java DSL specs
* GH-3043: Add FileHeaders.REMOTE_HOST header
Fixes https://github.com/spring-projects/spring-integration/issues/3043
* Populate a `FileHeaders.REMOTE_HOST` from the
`AbstractRemoteFileStreamingMessageSource` and "get"-based commands
in the `AbstractRemoteFileOutboundGateway`
* Extract the value from the a `Session.getHost()` contract
* The `AbstractInboundFileSynchronizingMessageSource` cannot be
addressed with this because the real message is already based on the
locally stored file
* Adjust some affected tests according our code style requirements
* * Add remote file info support into `AbstractInboundFileSynchronizingMessageSource`
* Introduce a `MetadataStore` functionality into the `AbstractInboundFileSynchronizer`
to gather a remote file info an save it in the URI style against local file
* Retrieve such an info in the `AbstractInboundFileSynchronizingMessageSource`
during local file polling
* Introduce `protocol()` contract for the `AbstractInboundFileSynchronizer`
to build a proper URI in the metadata for external readers to distinguish
remote files properly
* Document the feature
* * Fix some typos in Docs
* * Rename property and header constant to the `HOST_PORT` pair
* Fix typos in Docs
* Add `remote-file-metadata-store` and `metadata-store-prefix` into XSD
of (S)FTP Inbound Channel Adapters
* Add `remoteFileMetadataStore` and `metadataStorePrefix` options
into `RemoteFileInboundChannelAdapterSpec` for Java DSL
- republish certain events as `ApplicationEvent`s.
* * Add ApacheMinaFtplet to provide the same functionality with FTP
* Fix typo
* * Polishing javadocs and event toString() methods
Fixes https://github.com/spring-projects/spring-integration/issues/2987
To resolve XSD files properly from the classpath, their HTTPS reference
must be present in the `spring.schemas` to avoid the Internet interaction
for resolving an XSD file
**Cherry-pick to 5.1.x, 5.0.x & 4.3.x**
* Add `io.spring.nohttp` Gradle plugin into the build script and allow
it only on CI server, since it is going to parse every file
* Fix tests do not use `http` any more
* Fix `test.html` in the `mail` modules tests to exclude a Namespace for
OExchange XRD protocol
* Fix some files for new line in the end of file according Checkstyle
violations
While investigating an SI build failure, I noticed that there is
a timeout available for when connecting channels.
This won't resolve the build issue (where the session closed) but
worth adding as a property.
* Optimize DefaultSftpSessionFactory
Related to https://build.spring.io/browse/INT-MASTERSPRING40-677/
Doesn't look like `DefaultSftpSessionFactory.getSession()` needs
locking around `sharedJschSession`
* change the logic in the `getSession()` to store a `sharedJschSession`
into the local variable and if it is `null` or not connected, create a
new `JSchSessionWrapper`, connect it and store to the `sharedJschSession`
back into the `sharedJschSession` property if `this.isSharedSession`.
This way we always deal with `sharedJschSession` anyway if it is valid
or create a new fresh one if that is invalid.
Without locking we always get an actual state of the `sharedJschSession`
and don't fall into the race condition when `sharedJschSession` is invalid,
but we can't connect to the SFTP channel from the `sftpSession.connect()`
* * Wrap `sharedJschSession` initialization to the lock
* * Store `sharedJschSession` back when it is really fresh and while the lock
https://build.spring.io/browse/INT-MASTER-1481
The `SftpSession.connect()` may lead to race condition when we try to
open the `channel`, but `JschSession` is closed already.
It may happen in cases when we have `DefaultSftpSessionFactory`
configured for the `isSharedSession` and that shared session may be
closed by another thread before we reach the `channel.connect()`,
because we already have left the `this.sharedSessionLock` blocking path
**Cherry-pick to 5.1.x**
* Do not fix urls for the `http://apache.org/xml/features` when we
configure XML components
* Do not fix urls for localhost and fake links in tests
* Fix all the dead links in docs
