Fixes: #8967
In the `fair` mode the `StandardRotationPolicy` re-configures an `AbstractFetchLimitingMessageSource`
for a new directory (and possible new `ConnectionFactory`) in the `beforeReceive()`.
However, with default `maxFetchSize` (or bigger than `1`), the `receive()`` would poll `toBeReceived` internal queue
for files cached from the previous polling cycle.
Since we rotate the source immediately to a new set of options, all those cached files don't make sense
or even can cause the problem on fetching their content in case of `AbstractRemoteFileStreamingMessageSource`
when we rotate to a new `ConnectionFactory`.
* Call `fetchLimitingMessageSource.setMaxFetchSize(1);` in the `StandardRotationPolicy.beforeReceive()`
when `fair && !this.initialized`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8786
If `FtpSession.readRaw()` fails, the next `FtpSession.finalizeRaw()` call
would lead to `FTPClient.completePendingCommand()` failure since
there is no command to finish.
* Fix `FtpSession.finalizeRaw()` to exit earlier positively in case of
`FTPReply.isNegativePermanent()` for the current reply code set by the failure
from a previous `FtpSession.readRaw()`
**Cherry-pick to `6.1.x`, `6.0.x` & `5.5.x`**
Related to: https://stackoverflow.com/questions/77268009/how-to-use-sftp-spring-integration-mput-with-sftpoutboundgateway-when-file-objec
In some scenarios when the flow starts with a file inbound channel adapter
and then an MPUT operation is performed for remote file outbound gateway,
the populated in the beginning `FileHeaders.FILENAME` is used from
the `DefaultFileNameGenerator` for all the files from local directory to
upload.
Such a behaviour leads only to the last file in the target remote
directory and only with the name from that header.
* Fix the `AbstractRemoteFileOutboundGateway` to remove a `FileHeaders.FILENAME` header
when message is build for specific item from MPUT request.
This way an original local file is used when we upload directory.
* GH-8745: Add RFT.shouldMarkSessionAsDirty()
Fixes https://github.com/spring-projects/spring-integration/issues/8745
Not all errors caught in the `RemoteFileTemplate.execute()`
are fatal to mark session as dirty and physically close the target session
in the cache
* Introduce a `RemoteFileTemplate.shouldMarkSessionAsDirty()`
to consult with an exception if it is really a fatal error to close
the session in the end.
* Override `shouldMarkSessionAsDirty()` in the `RemoteFileTemplate`
implementations to check statuses of respective protocol errors
**Cherry-pick to `6.1.x` & `6.0.x`**
* * Fix tests for pool interaction
* * Fix language in Javadocs
* Add more `not dirty` statuses to `SftpRemoteFileTemplate` & `SmbRemoteFileTemplate`
Fixes https://github.com/spring-projects/spring-integration/issues/8691
* Remove setAge with TimeUnit
Turned out we don't need anymore since we're using Duration for age in FtpLastModifiedFileListFilter and SftpLastModifiedFileListFilter.
* Add changes to the docs
* Introduce AbstractLastModifiedFileListFilter
* Some code readability improvements
* Make language in the docs more official
Currently, many timeouts in the project are like `-1` or other negative value
with a meaning to wait indefinitely.
According to distributed systems design and bad demo developing experience
it is not OK to block forever.
* Rework most of the timeouts in the framework to be `30` seconds.
Only one remained as `1` seconds is a `PollingConsumer` where it is
better to not block even for those 30 seconds when no messages in the queue,
but let the polling task be rescheduled.
* Remove the `MessagingGatewaySupport.replyTimeout` propagation down to the
`PollingConsumer` correlator where it was a `-1` before and blocked
the polling thread on the `Queue.poll()`.
This fixed the problem with a single thread in a pool for auto-configured `TaskScheduler`.
Now with 1 seconds wait time we are able to switch to other scheduled tasks
even with only 1 thread in the pool
* Upgrade dependencies whenever it is reasonable
* Fix deprecations and some new API paths
* Move docs to a new version with respective branching for the `changes-5.5-6.0.adoc`
Looks like sometime Spring Framework XML parser can properly determine the target factory method to chose,
but sometimes it does that in a wrong order.
* Add `type="java.lang.Class"` to the ctor args everywhere for `class="org.mockito.Mockito" factory-method="mock"`
`<bean>` definitions
Fixes https://github.com/spring-projects/spring-integration/issues/3686
* Add `src/idea` with respective editor config for IntelliJ IDEA.
Must be imported into an IDE
* Add `src/eclipse` with respective editor config for Eclipse/STS
* Reformat imports in source code according a new editor config
* Mostly changes are related to the `TaskScheduler` and `Trigger` APIs
* Migrate to `micrometer-tracing` dependency
* Rework `SocketTestUtils` to use a `InetAddress.getLocalHost()`
for more stability and performance on Windows
* Fix docs for new `PeriodicTrigger` API
Fixes https://github.com/spring-projects/spring-integration/issues/3623
* `IntegrationFlow` refactoring
* Apply several code style improvements and good practices
* Code style: no empty lines for methods javadocs
* make deprecated implementation reuse actual one instead of the copy-paste approach
* add whats-new comments
* Fix whats-new page according to standards
* Upgrade Spring dependencies to the latest SNAPSHOTs
* Fix tests to verify against stack traces: the message
of the `NestedRuntimeException` does not include the nested exception information.
Related to https://github.com/spring-projects/spring-framework/issues/25162
* Fix `JdbcMessageStore` and `DefaultLockRepository` to rely on the `DataIntegrityViolationException`
instead of only its `DuplicateKeyException` extension.
This is the current behavior of the SQL errors translation
* Disable `WebFluxDslTests.testValidation()` - doesn't subscribe to the reply somehow...
* Refine `SimplePool.PoolSemaphore.reducePermits()`
Fixes https://github.com/spring-projects/spring-integration/issues/3656
* Move `JavaUtils` from `util` package to the root one to break any possible
tangling to/from other packages.
* Deprecate an existing `JavaUtils` for backward compatibility
* Remove `@IntegrationConverter` from the `JsonNodeWrapperToJsonNodeConverter`
and register it manually in the `ConverterRegistrar`.
The bean registration for the `JsonNodeWrapperToJsonNodeConverter` must be removed
in the next `6.0`
* Remove usage of `IntegrationContextUtils` from the `support` package
* GH-3554: Eval remote dir on each synchToLocal
Fixes https://github.com/spring-projects/spring-integration/issues/3554
The `remoteDirectoryExpression` was introduced into an
`AbstractInboundFileSynchronizer` to let end-user to evaluate a remote directory
on each poll (essentially on each `synchronizeToLocalDirectory()` call).
The fix for the https://jira.spring.io/browse/INT-4491 introduced a regression
when such an expression was evaluated only once when we call a `setRemoteDirectory()`.
So, an original purpose of this option was lost and we don't get an actual
remote dir on each poll
* Remove `evaluateRemoteDirectory()` method and its usage since it doesn't
reflect expectation of the remote dir expression property
* Reinstate the `remoteDirectoryExpression` evaluation in the
`synchronizeToLocalDirectory()`
* Propagate the result of that expression into further methods for copying
files
* Remove setting of the `remoteDirectory` variable into a global `EvaluationContext`
of the `AbstractInboundFileSynchronizer` instance since it is not thread-safe
* Instead create an `EvaluationContext` locally for each `synchronizeToLocalDirectory()`
call and set the `remoteDirectory` variable into this scoped instances
* Generate a local file name from the `localFilenameGeneratorExpression`
against locally created `EvaluationContext` with the mentioned `remoteDirectory` variable
* Cover the expected functionality with a unit-test
**Cherry-pick to `5.4.x` & `5.3.x`**
* * Fix `testRemoteDirectoryRefreshedOnEachSynchronization` according PR review
Resolves https://github.com/spring-projects/spring-integration/issues/3488
Resolves two problems:
- When changes are made deep in the directory tree, they were not detected because
the directory is in the metadata store and only passes the filter if a file
immediately under it is changed, changing the directory's timestamp.
This is solved by subclassing `AbstractDirectoryAwareFileListFilter`, allowing its
`alwaysAcceptDirectories` property to be set.
- Only the filename was used as a metadata key; causing problems if a file with the
same name appears multiple times in the tree.
This is solved with a new property on `AbstractDirectoryAwareFileListFilter` used by
the gateways to determine whether to filter the raw file names returned by the session
(previous behavior) or the full path relative to the root directory.
**cherry-pick to 5.4.x, 5.3.x**
* Some code style clean up
We fail sporadically in some FTP tests against
embedded Mina FTP server.
Turns out there was not enough concurrent logins allowed
on the server instance.
* Change the `ConnectionConfig` for FTP server to allow `1024`
concurrent logins instead of `10` by default
Related to https://build.spring.io/browse/INT-MASTERSPRING40-1314/
When we initialize a `CachingSessionFactory`, we may fail connecting
ot FTP server, but we realize that only when we call `super.getSession()`
from spied factory beans in the test configuration
* Move `super.getSession()` as a first call of the overridden method
and don't change the state of monitored variables when an exception
happens during FTP connection
* More GH Actions goodies
* Add RabbitMQ, MongoDB & Redis services into GH actions
* Remove Travis
* * Add `distributionSha256Sum` into `gradle-wrapper.properties`
* Try `TERM: dumb` to see difference for Gradle output in GH action
* Add `-S` to see the reason of test failure
* * Try MQTT plugin for RabbitMQ image
* * Try `-i` for test failure in Gradle on Actions
* * Use `eclipse-mosquitto` image for MQTT on actions
* Fix `FtpServerOutboundTests` for the proper FTP file list
* Output file list for the failed session in the `RotatingServersTests`
* * Adjust SOUT in the test for Checkstyle
* * Make some change to `RotatingServersTests`.
Looks like the order for `@BeforeAll` methods is not determined
and we may start to interaction with FTP server which is not
started yet
* Fix SSL Handshake Test
- we already made this change for the NIO test, but the NET test is failing the same
way - the cert failure switched to the server so the expected exception was not
thrown on the client side. Due to JVM changes.
* * Fix SFTP tests for wrong host and auto-startup state
* Remove `-i` for Gradle in the GH Actions
* * Try `--no-parallel` for Gradle in the GH Actions
* * Try GH actions without Mosquitto
* * Try `cyrilix/rabbitmq-mqtt` for GH Actions
Co-authored-by: Gary Russell <grussell@pivotal.io>
* Fix unused import in the `IntegrationRequestMappingHandlerMapping`
* Fix deprecations from Reactor
* Fix race condition in the `AbstractCorrelatingMessageHandlerTests`:
the discard message is sent much earlier than group is removed from the store.
Iterate group count call until it pass or 10 seconds timeout
* Remove list size assert in the `FtpServerOutboundTests`: looks like
it is not updated properly even if we have an expected content in the collection
* Increase timeout to assert remote files removal in the `FtpRemoteFileTemplateTests`
- Simplify MBeans - instead of wrapping to expose lifecycle methods,
implement `ManageableLifecycle`. Register an additional MBean for
polled endpoints to control the lifecycle.
* Polishing
- Move `QueueChannel` `@ManagedAttribute`s to `QueueChannelOperations`
- Make all `AbstractEndpoints` `IntegrationManagedResource`s and remove `ManagedEndpoint`
to allow exposure of any `@Managed*` methods (including those on `Pausable`)
- Revert to `Lifecycle` for classes that are not related to endpoints
- Remove legacy metrics from docs
The `ArrayList` cannot be used in the test-case for checking
its size because the property is not `volatile`: even if
content of the list is what we expect (test logs confirms that),
the non-`volatile` property doesn't give us an actual value
* Use `CopyOnWriteArrayList` for this concurrent test to
be sure that internal `array.length` reflects the reality
after each interaction with the list
Resolves https://github.com/spring-projects/spring-integration/issues/3315
`maxFetchSize` ignored with filters supporting single file filtering (default).
This breaks "fair" rotation with the `RotatingServerAdvice`.
Honor `maxFetchSize`, even with filters that support single file filtering.
**cherry-pick to 5.3.x, 5.2.x**
* GH-2996: Add Resource for remote file transfer
Fixes https://github.com/spring-projects/spring-integration/issues/2996
* Fix `RemoteFileTemplate.send()` to support a `Resource` payload
for remote file transferring content
* Code style clean up for `RemoteFileTemplate`
* Remove `volatile` for configuration properties for better performance
* Change a `charset` to the `Charset` for only once conversion from string
during configuration phase
* Fix (S)FTP tests for new functionality
* Change affected tests to JUnit 5
* Document a new feature; mention all the supported types and `FileExistsMode` constants
* * Fix language in `whats-new.adoc`
Fixes https://github.com/spring-projects/spring-integration/issues/3168
* Call `this.client.noop()` instead of `this.client.isConnected()` to really
check that client has a live connection with the server before calling a `this.client.logout()`
* Wrap `this.client.logout()` into a `try..catch` to be sure that we wil call a
`this.client.disconnect()` even if `logout()` fails for some reason.
* Change `WARN` logs about `Session.close()` into a `DEBUG` level -
when we have a problem with closing session because of server disconnect reason
we have no any control to do with a situation
**Cherry-pick to 5.2.x**
* GH-3155: Add support for Java DSL extensions
Fixes https://github.com/spring-projects/spring-integration/issues/3155
Provide an `IntegrationFlowExtension` for possible custom EI-operators
in the target project use-cases.
* * Move `IntegrationFlowExtension` tests ot its own test class
* Make all the `IntegrationComponentSpec` ctors as `protected` for possible custom extensions
* Make some `BaseIntegrationFlowDefinition` methods and properties as `protected` to get them
access from the `IntegrationFlowExtension` implementations
* Document the feature
* * Fix language and typos in docs
* * Add `protected` to one more `GatewayEndpointSpec` ctor
* Add JavaDocs to `GatewayEndpointSpec` methods
* * Add `protected` to one more `JmsPollableMessageChannelSpec` ctor
Fixes https://github.com/spring-projects/spring-integration/issues/3132
It turns out that Checkstyle EmptyBlock doesn't complain about
empty default ctor.
Plus a new check for `super();` call treats it as a violation
* Remove `super();` from all the no-arg ctors
* Code style clean up in the affected classes according
IDEA suggestions
* Fix new Sonar smells
* Move `What's New` into `changes-5.1-5.2.adoc`
* Remove version from the `XSD` files:
1. Align with SF
2. The version for XSD doesn't matter for the current jar version,
since only the version is available in classpath is from the current
jar version
3. Modify `spring.schemas` to map all the possible versions for XSD
into the current one in a jar.
This way target applications can upgrade without changing the version
for XSD location and we don't need to require `versionless` variant
any more
4. The jar in classpath can handle only its own XSD, so independently
of version or no-version variants we still may fail because the current
jar may not support end-user code any more - need some fix over there
anyway.
* Remove `checkTestConfigs` Gradle task since we don't worry about XSD
version any more
* Modify `schemaZip` Gradle task to populate XSDs into a distribution
still with a version to avoid overriding on-line XSDs for version `1.0.x`.
We may consider not do that since SF doesn't and just bite a bullet
for always overriding those on-line XSDs to the actual one from the latest
release.
* GH-3090: Add `logout() to `FtpSession.close()`
Fixes https://github.com/spring-projects/spring-integration/issues/3090
Without `logout()` the FTP session is not closed at all,
but just the connection is closed.
Some FTP servers close those sessions eventually anyway, but some just
leak with resources.
**Cherry-pick to 5.1.x & 4.3.x**
* * Migrate `SessionFactoryTests` to JUnit 5
Fixes https://github.com/spring-projects/spring-integration/issues/3026
**Cherry-pick to `5.1.x`**
* Populate proper `FileTransferringMessageHandler` impl from DSL spec
implementations.
This way we are able to use a provided `chmod` from Java DSL
* Added `FileTransferringMessageHandlerSpec` ctor TODO
* Update SftpTests
* Code cleanup; `@Ignore` `SftpTests.testSftpOutboundFlowWithChmod()`
since it doesn't work properly on Windows
Related to: https://build.spring.io/browse/INT-MASTERSPRING40-826/
If we don't close application context on the embedded (S)FTP server,
we my still have some session opened in other tests, which would cause
some unexpected behavior like extra `SessionClosedEvent` in the
test for FTP events
