Fixes https://github.com/spring-projects/spring-integration/issues/3026
**Cherry-pick to `5.1.x`**
* Populate proper `FileTransferringMessageHandler` impl from DSL spec
implementations.
This way we are able to use a provided `chmod` from Java DSL
* Added `FileTransferringMessageHandlerSpec` ctor TODO
* Update SftpTests
* Code cleanup; `@Ignore` `SftpTests.testSftpOutboundFlowWithChmod()`
since it doesn't work properly on Windows
Related to: https://build.spring.io/browse/INT-MASTERSPRING40-826/
If we don't close application context on the embedded (S)FTP server,
we my still have some session opened in other tests, which would cause
some unexpected behavior like extra `SessionClosedEvent` in the
test for FTP events
* GH-3043: Add FileHeaders.REMOTE_HOST header
Fixes https://github.com/spring-projects/spring-integration/issues/3043
* Populate a `FileHeaders.REMOTE_HOST` from the
`AbstractRemoteFileStreamingMessageSource` and "get"-based commands
in the `AbstractRemoteFileOutboundGateway`
* Extract the value from the a `Session.getHost()` contract
* The `AbstractInboundFileSynchronizingMessageSource` cannot be
addressed with this because the real message is already based on the
locally stored file
* Adjust some affected tests according our code style requirements
* * Add remote file info support into `AbstractInboundFileSynchronizingMessageSource`
* Introduce a `MetadataStore` functionality into the `AbstractInboundFileSynchronizer`
to gather a remote file info an save it in the URI style against local file
* Retrieve such an info in the `AbstractInboundFileSynchronizingMessageSource`
during local file polling
* Introduce `protocol()` contract for the `AbstractInboundFileSynchronizer`
to build a proper URI in the metadata for external readers to distinguish
remote files properly
* Document the feature
* * Fix some typos in Docs
* * Rename property and header constant to the `HOST_PORT` pair
* Fix typos in Docs
* Add `remote-file-metadata-store` and `metadata-store-prefix` into XSD
of (S)FTP Inbound Channel Adapters
* Add `remoteFileMetadataStore` and `metadataStorePrefix` options
into `RemoteFileInboundChannelAdapterSpec` for Java DSL
- republish certain events as `ApplicationEvent`s.
* * Add ApacheMinaFtplet to provide the same functionality with FTP
* Fix typo
* * Polishing javadocs and event toString() methods
While investigating an SI build failure, I noticed that there is
a timeout available for when connecting channels.
This won't resolve the build issue (where the session closed) but
worth adding as a property.
Resolves https://github.com/spring-projects/spring-integration/issues/2777
If the filter supports it, defer filtering until the last possible moment.
Then, the worst case scenario after a catastrophic failure (e.g. power loss),
would be that at most one file will be incorrectly filtered on restart.
Polishing and add more tests.
Polishing Javadocs
More Polishing
Final polishing
More polishing.
Polishing and docPolishing and docs.
* Fix typos in Docs
* Defer Messaging annotations process
The `AbstractMethodAnnotationPostProcessor` and its implementations
have a `beanFactory.getBean()` call for the `@Bean` methods with
Messaging annotations.
This is done, actually, from the
`MessagingAnnotationPostProcessor.postProcessAfterInitialization()`
which might be still too early in some scenarios, like Spring Cloud Feign
with its child application contexts being initialized from the
`FeignClientFactoryBean`, causing a `BeanCurrentlyInCreationException`
See https://stackoverflow.com/questions/54887963/beancurrentlyincreationexception-when-using-spring-integration-with-spring-cloud
* Implement a `SmartInitializingSingleton` for the `MessagingAnnotationPostProcessor`
and gather `Runnable` wrappers for newly introduced `postProcessMethodAndRegisterEndpointIfAny()`
to be called later in the `afterSingletonsInstantiated()` when context is
still in the initialization phase.
All runtime-registered beans are going to be processed normally from the
regular `postProcessAfterInitialization()`
**Cherry-pick to 5.1.x**
* * Fix unused imports in the `MessagingAnnotationsWithBeanAnnotationTests`
* * Fix `IntegrationEndpointsInitializer` in the testing framework to handle
all the possible `AbstractEndpoint` beans registration.
See its JavaDocs for more info
* Fix `AbstractCorrelatingMessageHandlerParser` and
`AbstractConsumerEndpointParser` to use bean names for `outputChannel`
and `discardChannel` instead of bean references.
Since `MessagingAnnotationPostProcessor` now registers endpoints and
beans for channels much later, than parsers, we can't rely on bean
references any more there.
* Fixes for failing tests which expected `outputChannel/discardChannel`
bean references, when it is already just their names for late binding.
* Apply some code style polishing for the affected classes.
* Add `@Nullable` for `MessageSelector` parameter in the `QueueChannel.purge()`
Mostly thanks to IDEA's plugin: https://plugins.jetbrains.com/plugin/10345-assertions2assertj
There is still a lot of work to do when complex and composite matchers are used.
* Add `awaitility` dependency and deprecate `EventuallyMatcher` in favor
of `awaitility`
* Remove Hamcrest from dependencies and disable JUnit & Hamcrest
static imports to encourage to use only AssertJ
* Migrate JUnit assumptions in rules to AssertJ's assumptions
* Deprecate some custom matchers in favor of existing in Hamcrest
after upgrading the last to version `2.1`
* Replace `ExpectedException` rules with `assertThatThrownBy()`
* Mention `MessagePredicate` in the `testing.adoc`
* Spring AMQP 2.1 B-S and fix imports for moved classes
* Reactor GA
* SF-5.1 B-S and exclude `org.springframework` from all the
Spring Security dependencies
* the latest Hibernate and JPA and fix the test query for the
appropriate actual syntax
* latest SSHD and add required dependency for `sshd-sftp`
* latest Groovy and extra dependency for the `groovy-dateutil`
* latest Smack for XMPP and appropriate code fixes for compatibility
* Fix `StompIntegrationTests` to use `AFTER_EACH_TEST_METHOD`:
when session is disconnected on the server, we can't interact with it
anymore
* INT-4381: MessageSources refactoring
JIRA: https://jira.spring.io/browse/INT-4381
* Make all the out-of-the-box `MessageSource` implementations based
on the `AbstractMessageSource`
* Fix `JdbcPollingChannelAdapterIntegrationTests` for sporadic failure
because of `fixed-rate` for the poller
* Fix HeaderEnricherTests race condition
The `errorChannel()` expect an error in the `testErrorChannel` after
`1000` ms, but at the same time the `poller` in configured for the
`3000` ms.
* Increase all the timeouts for replies
* Decrease a `fixed-delay` on the `poller`
* Some other code style polishing for the `HeaderEnricherTests`
JIRA: https://jira.spring.io/browse/INT-4498
* Use `FileCopyUtils` instead of `StreamUtils` in test-cases when we deal
with `InputStream` s and files resources.
* Fix reflection-based tests with AOP to use an `ArrayList` access
instead of `LinkedList` since an optimization in SF:
6d6aa72e8f
**Cherry-pick to 5.0.x and 4.3.x**
Windows is too scrupulous for non closed file resources.
Our `RemoteFileTestSupport` recreates a test directory with files
for each test.
When we don't close some file handler, we are not able to delete the
directory and we fail with subsequent tests.
* Close `InputStream` for each polled file in new tests in the
`SftpStreamingMessageSourceTests`
* Close `CLOSEABLE_RESOURCE` in new tests in the
`FtpStreamingMessageSourceTests`
* Rework `FtpStreamingMessageSourceTests.testAllContents()` do not
poll all the files on each polling cycle - this causes a race condition
when we don't close `CLOSEABLE_RESOURCE` yet in the `StreamTransformer`,
but try to proceed with recreation a test directory structure
**Cherry-pick to 5.0.x**
JIRA: https://jira.spring.io/browse/INT-4495
Previously, `maxFetch` was applied before directories were removed from
the fetch list.
Remove the directories before filtering and applying `maxFetch`.
* Polishing - PR Comments
* More polishing
* More polishing.
* Check for empty array.
* Remove test main method.
JIRA: https://jira.spring.io/browse/INT-4390
Phase I: flow context/registration cycle
- extract interfaces
- rename implementations to `Standard...`
Phase II - dsl<->dsl.channel tangles
Move the channel specs to dsl.
Fix missing refactorings
(Not related to DSL) - rename core `event` package to `events`
- avoid collision with event module package
- fix tangle caused by `MessageGroupExpiredEvent`.
Phase III - dsl<->config.dsl
- move classes from config.dsl to dsl
(Not related to DSL) - fix many tangles caused by graph being a sub-package of support.management
- make `graph` a top-level package
(Not related to DSL) - move `IntegrationManagementConfigurer` from
`management` to `config` - tangle between core and management
* Polishing Copyrights, diamonds, some JavaDocs and What's New
* INT-4376: Upgage to Log4J 2
JIRA: https://jira.spring.io/browse/INT-4376
* Deprecate Log4J 1.x components in favor of newly added
a `Log4j2LevelAdjuster` JUnit `@Rule`
* Update all the logging configuration to Log4J 2
* Polishing after testing
JIRA: https://jira.spring.io/browse/INT-4283
Filter that only passes files where a corresponding "marker" file is also present.
Implementations for file and(S)FTP.
* Polishing - PR Comments
* Missed one
JIRA: https://jira.spring.io/browse/INT-4245
* Move `ReplyProducingMessageHandlerWrapper` outside of
`ServiceActivatorAnnotationPostProcessor`.
* Use it from the `ServiceActivatorFactoryBean` as well to fix
the missed `Lifecycle` control
* Increase wait timeout in the `ConnectionFactoryTests`
* Mark endpoint as `auto-startup="false"` in the SFTP parser tests
* Remove redundant `org.gradle.daemon=true` from the `gradle.properties`
since it is like that by default for a while already
JIRA: https://jira.spring.io/browse/INT-4086
The remote files may be carried with sub-path, not just file name.
The same sub-dirs logic can be achieve with the `localFilenameGeneratorExpression`
* Add the logic into `AbstractInboundFileSynchronizer` to build sub-dirs for local files
* Switch on `WatchService` for the internal `FileReadingMessageSource` in the `AbstractInboundFileSynchronizingMessageSource` to react for all the changes in the `localDirectory`
Introduce `RecursiveDirectoryScanner` and use it for `AbstractInboundFileSynchronizingMessageSource` by default
* Add `maxDepth` and `fileVisitOptions` options to the `RecursiveDirectoryScanner`
* Polishing JavaDocs, tests and docs
JIRA: https://jira.spring.io/browse/INT-4115
Apply `(S)FtpPersistentAcceptOnceFileListFilter` for the `(S)FtpInboundFileSynchronizer` by default to avoid cases to sync the same remote files to the local directory again.
Especially when `localFileName` strategy is applied and we end up with new local files, but with the same remote content
Accept `(S)FtpPersistentAcceptOnceFileListFilter` for streaming adapters
Make `doSetFilter()` as `protected final`
Fix tests after rebase
Fix "What's New" after rebase
Compose `PersistentAcceptOnceFileListFilter` together with the regex or pattern filters
Document such a behavior
Address PR comments for formatting and typo
JIRA: https://jira.spring.io/browse/INT-4018
* Resolve some TODOs as expected
* Remove some as logically or physically with useless effort
* Leave others for future considerations
JIRA: https://jira.spring.io/browse/INT-4243
Remove newline char from key exchange.
Fix CR and Add Windows Delete Diagnostics
Fix For Windows
More diagnostics.
It appears the new server doesn't close the file when the session is closed.
In the streaming test, consume the stream before closing the session.
Polishing - PR Comments
* More polishing according PR comments
JIRA: https://jira.spring.io/browse/INT-4219
- Use `AdditionalAnswers.returnFirstArgument` instead of lambda to return first arg
- Enable CheckStyle `AvoidStaticImport` for tests
- Fix static import violations in tests
JIRA: https://jira.spring.io/browse/INT-4232
Currently to implement the synchronization logic when remote file is modified there is only one way: remove local file and provide a `localFilter` which can be clear from the removed file as well.
The Reference Manual claims that `FileSystemPersistentAcceptOnceFileListFilter` can let us pick up a fresh version of the remote file if we configure `preserveTimestamp = true`, but actually `AbstractInboundFileSynchronizer` just bypass that logic because it check a local file for existence.
* Modify `AbstractInboundFileSynchronizer` to compare `lastModified` of the remote file with local version if `preserveTimestamp == true`
* Also replace `AcceptOnceFileListFilter` to the `FileSystemPersistentAcceptOnceFileListFilter` since the first one doesn't care about `lastModified`.
If `preserveTimestamp` isn't in use the `FileSystemPersistentAcceptOnceFileListFilter` works fully similar to the `AcceptOnceFileListFilter`
**Cherry-pick to 4.3.x except test-case**
Make `SftpStreamingMessageSourceTests` compatible with Windows
Looks like SFTP embedded server provides different permissions for files on different OSs
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4233
INT-4233: Add (S)FTP FileInfo Header (Streaming)
JIRA: https://jira.spring.io/browse/INT-4233
Add the complete file info as JSON (when Jackson or Boon available) to the message headers
when streaming inbound.
Provide a mechanism to configure Boon to provide similar output to Jackson.
Also allow subclasses to provide their own object mapper.
Also clean up after the AMQP DSL tests, and don't use a queue `foo`. I often have such a queue
with content; this caused tests to fail.
Use SimpleJsonSerializer
Doc Polishing
Polishing - PR Comments
More Polishing
* Polishing according latest PR comments
JIRA: https://jira.spring.io/browse/INT-4231
For `FileWritingMessageHandler` and (S)FTP outbound gateways, support
`FileExistsMode.REPLACE_IF_MODIFIED` to allow overwriting an existing file if
the source file modified time is different to the existing file.
Polishing
JIRA: https://jira.spring.io/browse/INT-4230
The `get()` method only preserves the timestamp when an LS operation is done within it.
For MGET ops, we don't perform another LS, so the timestamp was not updated.
Add code to the recursive and non-recursive MGET methods to preserve the timestamp if so configured.
JIRA: https://jira.spring.io/browse/INT-4206
* Fix unnecessary dependency resolution in BOM module
* Fix `MessagingMethodInvokerHelper` to handle `$MockitoMock$` generated classed which isn't CGLib `Proxies` any more
* Provide fixes for test classes according upgrade to Mockito `2.5`
* Fix Ceckstyle do not allow static imports for deprecated Mockito classes
JIRA: https://jira.spring.io/browse/INT-4175
Fixes GH-1980 (https://github.com/spring-projects/spring-integration/issues/1980)
When the `isSharedSession` is used for the `DefaultSftpSessionFactory`,
there is some race condition window when we can call the target `this.jschSession.connect()` several times and end up with the session is already connected.
Wrap `sftpSession.connect()` to the `this.sharedSessionLock.readLock().lock()` when `isSharedSession` to protect from that race condition.
Note: there is no test for this change because it is pretty tricky to build barriers between threads for this kind of race conditions.
Especially after introduction `this.sharedSessionLock.readLock().lock()` around guilty code
**Cherry-pick to 4.3.x & 4.2.x**
Move `sharedJschSession` connect logic to the existing locking block
Fix mock test to reflect the current ConnectionFactory state
JIRA: https://jira.spring.io/browse/INT-4155
- Add streaming support
- Rework tests to use dynamic flow registrations due to the test FTP server is now a test support class.
Javadoc fixes
Checkstyle fix
Polishing - PR Comments
Polishing and Port SFTP
JIRA: https://jira.spring.io/browse/INT-3770,
https://jira.spring.io/browse/INT-4107
Having `TransactionHandleMessageAdvice` we can start TX from any `MessageHandler.handleMessage()`
* Add `<transactional>` alongside with the `<request-handler-advice-chain>` for those components which produce reply
* Merge `<transactional>` and `<request-handler-advice-chain>` configuration to a single `ManagedList`
* Rework JPA `<transactional>` in favor of common solution
* Some polishing and refactoring
AbstractPollingEndpoint: avoid `new ArrayList` if we don't have `receiveOnlyAdvice`s
AMQP Lambdas
Event Lambdas
Feed/File Lambdas
Gemfile/Groovy Lambdas
* Ensure that JavaDocs are checked via ` check.dependsOn javadoc`
* Add `@return` tag to the `MessageBuilder.readOnlyHeaders()`
JIRA: https://jira.spring.io/browse/INT-4095
Limit the number of remote files fetched on each poll (when it is necessary to fetch files).
Polishing - PR Comments
Polishing - Decouple MaxFetchSize from Poller
Polishing - PR Comments
Schemas and Docs
More Polishing
* Polishing according PR comments
JIRA: https://jira.spring.io/browse/INT-4021
Previously the `SimpleMessageStore` unconditionally followed with the super class options
and provided the `lazy-load` functionality by default.
* Since `persistent` and `lazy-load` logic does not make sense for the `in-memory` store, disable it in the `SimpleMessageStore`
* Fir `AggregatorTests` for better coverage.
* NOTE: The same message can't be persisted in the Persistent `MessageStore`. The store key is fully based on the `messageId`.
And also we provide the header which indicates that the messages has been stored before. See `JdbcMessageStore.addMessage()` for example:
```
if (message.getHeaders().containsKey(SAVED_KEY)) {
Message<T> saved = (Message<T>) getMessage(message.getHeaders().getId());
if (saved != null) {
if (saved.equals(message)) {
return message;
} // We need to save it under its own id
}
}
```
* Fix (S)FTP Streaming tests to use `AbstractPersistentAcceptOnceFileListFilter` instead of raw `AcceptOnceFileListFilter`,
which relies on the object identity, but neither `FtpFile`, nor `ChannelSftp.LsEntry` provides good `equals()` and `hashCode()` implementations.
* Make `StompIntegrationTests` as `DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD` because the sporadic failure on the Travis around wrong value from the queue isn't clear (yet).
Looks like some interim event is generated by the Websocket Container on the Tomcat.
JIRA: https://jira.spring.io/browse/INT-4015https://jira.spring.io/browse/INT-3854
Initial commit.
Reworked to emit an input stream and use the file splitter.
Add StreamTransformer.
Add CLOSABLE_RESOURCE header so we can close the session automatically.
Implement INT-3854, FTP, SFTP
(S)FTP Namespace Changes
Docs - also fixes a PDF overflow
Polishing - PR Comments
checkstyle fixes
Polishing - Add Namespace for StreamParser
Polishing - PR Comments
* Upgrade to SA and SF snapshots
* Address TODOs regarding those upgrades
* Resolve some other TODOs
* Replace `PropertyPlaceholderConfigurer` beans with the `<context:property-placeholder>`
* Upgrade to Spring Social Twitter 1.1.2 and resolve deprecations via mocks (https://github.com/spring-projects/spring-social-twitter/issues/91)
* Upgrade to Curator `3.1.0` and resolve deprecation in the `ZookeeperMetadataStore`
