JIRA: https://jira.spring.io/browse/INT-4283
Filter that only passes files where a corresponding "marker" file is also present.
Implementations for file and(S)FTP.
* Polishing - PR Comments
* Missed one
JIRA: https://jira.spring.io/browse/INT-4060
* Add `NLST` command to the `AbstractRemoteFileOutboundGateway` to perform
`listNames` on the target session.
Useful in case of server doesn't allow to perform `LS` or the names set is
sufficient for application requirements
* Add `workingDirExpression` to the `FtpOutboundGateway` to allow to perform
`FtpClient.changeWorkingDirectory()` based on the current request message
* Change `slf4j-log4j12` to the `testCompile` -
the FTP tests fail in the IDE with `ClassNotFoundException`
Address PR comments
* Add `nlst` to XSD config
* Reinstate `ls -1` test-case for the `FtpServerOutboundTests`
* wrap more commands to the `doInWorkingDirectory()`
Fix `MV` command in the `AbstractRemoteFileOutboundGateway`
* Implement `RemoteFileOperations#invoke(OperationsCallback<F, T>)`
for thread-bound `session`s
* Use a new `invoke()` for `put()` and `mPut()` commands in the `AbstractRemoteFileOutboundGateway`
* Add delegation for the `put()` and `mPut()` commands in the `FtpOutboundGateway`
* Add DSL support for the `workingDirExpression` and add `whats-new.adoc` note
Document changes
Address some PR comments:
* Add `invokeScope` variable to `execute` to track `ThreadLocal` session or not
* Check for `null` in the `getSession()` and fallback to regular
`sessionFactory.getSession()`.
Most likely the `invoke()` is called from other thread
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4086
The remote files may be carried with sub-path, not just file name.
The same sub-dirs logic can be achieve with the `localFilenameGeneratorExpression`
* Add the logic into `AbstractInboundFileSynchronizer` to build sub-dirs for local files
* Switch on `WatchService` for the internal `FileReadingMessageSource` in the `AbstractInboundFileSynchronizingMessageSource` to react for all the changes in the `localDirectory`
Introduce `RecursiveDirectoryScanner` and use it for `AbstractInboundFileSynchronizingMessageSource` by default
* Add `maxDepth` and `fileVisitOptions` options to the `RecursiveDirectoryScanner`
* Polishing JavaDocs, tests and docs
JIRA: https://jira.spring.io/browse/INT-4115
Apply `(S)FtpPersistentAcceptOnceFileListFilter` for the `(S)FtpInboundFileSynchronizer` by default to avoid cases to sync the same remote files to the local directory again.
Especially when `localFileName` strategy is applied and we end up with new local files, but with the same remote content
Accept `(S)FtpPersistentAcceptOnceFileListFilter` for streaming adapters
Make `doSetFilter()` as `protected final`
Fix tests after rebase
Fix "What's New" after rebase
Compose `PersistentAcceptOnceFileListFilter` together with the regex or pattern filters
Document such a behavior
Address PR comments for formatting and typo
JIRA: https://jira.spring.io/browse/INT-4018
* Resolve some TODOs as expected
* Remove some as logically or physically with useless effort
* Leave others for future considerations
JIRA: https://jira.spring.io/browse/INT-4219
- Use `AdditionalAnswers.returnFirstArgument` instead of lambda to return first arg
- Enable CheckStyle `AvoidStaticImport` for tests
- Fix static import violations in tests
JIRA: https://jira.spring.io/browse/INT-4232
Currently to implement the synchronization logic when remote file is modified there is only one way: remove local file and provide a `localFilter` which can be clear from the removed file as well.
The Reference Manual claims that `FileSystemPersistentAcceptOnceFileListFilter` can let us pick up a fresh version of the remote file if we configure `preserveTimestamp = true`, but actually `AbstractInboundFileSynchronizer` just bypass that logic because it check a local file for existence.
* Modify `AbstractInboundFileSynchronizer` to compare `lastModified` of the remote file with local version if `preserveTimestamp == true`
* Also replace `AcceptOnceFileListFilter` to the `FileSystemPersistentAcceptOnceFileListFilter` since the first one doesn't care about `lastModified`.
If `preserveTimestamp` isn't in use the `FileSystemPersistentAcceptOnceFileListFilter` works fully similar to the `AcceptOnceFileListFilter`
**Cherry-pick to 4.3.x except test-case**
Make `SftpStreamingMessageSourceTests` compatible with Windows
Looks like SFTP embedded server provides different permissions for files on different OSs
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4233
INT-4233: Add (S)FTP FileInfo Header (Streaming)
JIRA: https://jira.spring.io/browse/INT-4233
Add the complete file info as JSON (when Jackson or Boon available) to the message headers
when streaming inbound.
Provide a mechanism to configure Boon to provide similar output to Jackson.
Also allow subclasses to provide their own object mapper.
Also clean up after the AMQP DSL tests, and don't use a queue `foo`. I often have such a queue
with content; this caused tests to fail.
Use SimpleJsonSerializer
Doc Polishing
Polishing - PR Comments
More Polishing
* Polishing according latest PR comments
JIRA: https://jira.spring.io/browse/INT-4231
For `FileWritingMessageHandler` and (S)FTP outbound gateways, support
`FileExistsMode.REPLACE_IF_MODIFIED` to allow overwriting an existing file if
the source file modified time is different to the existing file.
Polishing
JIRA: https://jira.spring.io/browse/INT-4230
The `get()` method only preserves the timestamp when an LS operation is done within it.
For MGET ops, we don't perform another LS, so the timestamp was not updated.
Add code to the recursive and non-recursive MGET methods to preserve the timestamp if so configured.
JIRA: https://jira.spring.io/browse/INT-4221
* Upgrade to those versions of Spring project dependencies which potentially will provide similar fix
**Cherry-pick to 4.3.x**
JIRA: https://jira.spring.io/browse/INT-4206
* Fix unnecessary dependency resolution in BOM module
* Fix `MessagingMethodInvokerHelper` to handle `$MockitoMock$` generated classed which isn't CGLib `Proxies` any more
* Provide fixes for test classes according upgrade to Mockito `2.5`
* Fix Ceckstyle do not allow static imports for deprecated Mockito classes
JIRA: https://jira.spring.io/browse/INT-4171
Since the solution requires reflection on `sun` classes, document only.
Also fix some PDF text overflows in (S)FTP.
JIRA: https://jira.spring.io/browse/INT-4155
- Add streaming support
- Rework tests to use dynamic flow registrations due to the test FTP server is now a test support class.
Javadoc fixes
Checkstyle fix
Polishing - PR Comments
Polishing and Port SFTP
JIRA: https://jira.spring.io/browse/INT-3770,
https://jira.spring.io/browse/INT-4107
Having `TransactionHandleMessageAdvice` we can start TX from any `MessageHandler.handleMessage()`
* Add `<transactional>` alongside with the `<request-handler-advice-chain>` for those components which produce reply
* Merge `<transactional>` and `<request-handler-advice-chain>` configuration to a single `ManagedList`
* Rework JPA `<transactional>` in favor of common solution
* Some polishing and refactoring
AbstractPollingEndpoint: avoid `new ArrayList` if we don't have `receiveOnlyAdvice`s
JIRA: https://jira.spring.io/browse/INT-832
When we scan directory recursively for files (e.g. `WatchServiceDirectoryScanner`), it can be useful to get access to the relative path from the `Message`, e.g. on the `FileWritingMessageHandler` side to restore the original structure
* Add a `FileHeaders.FILENAME` into the outbound `Message` from the `FileReadingMessageSource`
* The result of that header is like a removal of leading `this.directory.getAbsolutePath()` in the target `File.getAbsolutePath()`.
In case of not recursion we get just only regular file name.
* Introduce `FileHeaders.RELATIVE_PATH`
* Populated `FileHeaders.RELATIVE_PATH`, `FileHeaders.FILENAME`, `FileHeaders.ORIGINAL_FILE` in the `FileReadingMessageSource`
* File `FileTailingMessageProducerSupport` to populate `FileHeaders` properly
* Introduce ctor for the `LastModifiedFileListFilter` for better Java configuration experience
* Add docs for changes
Doc Polishing
AMQP Lambdas
Event Lambdas
Feed/File Lambdas
Gemfile/Groovy Lambdas
* Ensure that JavaDocs are checked via ` check.dependsOn javadoc`
* Add `@return` tag to the `MessageBuilder.readOnlyHeaders()`
* Upgrade to Checkstyle `7.1`
* Relax `RequireThis` rule a bit. Right now it does the effort only in case of overlapping. See https://github.com/checkstyle/checkstyle/issues/2362 for more info
* Enable some annotation rules and provide fixes for violations
* Enable `tabs indents` rule. This was the biggest fix in this PR
* Resolve `TODO` in the `MessagingMethodInvokerHelper` and fix tests to meet `IllegalStateException` now
Remain Checkstyle version `6.16.1` and reinstate `RequireThis` rule
The latest Checkstyle has a bug with local scope variables if they have the same names as property.
Revert some literals splitting
Fix some line length exceeding and code style
* Make most functional interfaces as `@FunctionalInterface`
* Convert some abstract classes to `@FunctionalInterface` with `default` methods
* Apply Lambda style implementation in some places
* Remove `Function` in favor of similar in Java 8
* Remove redundant code from `DefaultAmqpHeaderMapper` since we are already on Spring AMQP-2.0
* Add several ctors to the `ExpressionEvaluatingMessageListProcessor`
* Populate explicit `Boolean.class` `expectedType` from the `ExpressionEvaluatingReleaseStrategy`
JIRA: https://jira.spring.io/browse/INT-4095
Limit the number of remote files fetched on each poll (when it is necessary to fetch files).
Polishing - PR Comments
Polishing - Decouple MaxFetchSize from Poller
Polishing - PR Comments
Schemas and Docs
More Polishing
* Polishing according PR comments
* Fix several typos in log messages. And some test on the matter as well
* Add comment to `AbstractPersistentAcceptOnceFileListFilter.rollback()` to clarify the reason of `rollingBack` variable
* Make `RemoteFileTemplate.StreamHolder` as `static` to avoid extra internal variable to outer class instance
* Replace `MessagingException` with `AbstractInboundFileSynchronizingMessageSource` in `init()` method of some components. It isn't Messaging yet in that phase
* Fix `SubscribableRedisChannel.MessageListenerDelegate` to handle `Object` not `String`, because with the `serializer` injection there is no guaranty that incoming is always `String`
* Move `JSch.setLogger(new JschLogger());` in the `DefaultSftpSessionFactory` to `static` block. It really should be done only once
* Remove `Assert.isTrue(this.port >= 0)` from the `DefaultSftpSessionFactory`. The subsequant `initJschSession()` convert it to default `22` port
* Change in the `JschProxyFactoryBean` `UnsupportedOperationException` to `IllegalArgumentException`. Wrong enum is wrong argument. That isn't a problem of operation
* Simplify `stop()` in the `CuratorFrameworkFactoryBean` and mark it as a `this.running = false`. Otherwise it wasn't able to be restarted
* Expose `leaderEventPublisher` in the `LeaderInitiatorFactoryBean` and fix `stop(Runnable callback)` with propagation `callback` to delegate.
Fix `SubscribableRedisChannelTests` for new `handleMessage(Object)` signature
JIRA: https://jira.spring.io/browse/INT-4046
Since not all FTP servers provide proper `STAT` command implementation,
plus the `NLIST` doesn't work properly for directories cases, introduce the `FtpRemoteFileTemplate.ExistsMode`
to let:
* to perform `STAT` by default (previous) behavior;
* to switch to `NLIST` for `FtpRemoteFileTemplate` internal use;
* perform the full `NLIST` and `FTPClient.changeWorkingDirectory()` algorithm if needed.
* Improve (S)Ftp components to use proper `RemoteFileTemplate` for internal instantiation
* Introduce `FtpMessageHandler` to wrap `FtpRemoteFileTemplate` with the proper `NLIST` `ExistsMode`
* Cover `NLIST` switching from the `FtpOutboundChannelAdapterParser` and `FtpOutboundGatewayParser`
* Document the `FtpRemoteFileTemplate.ExistsMode`
* Fix typo in the recently introduced `RemoteFileOperations.getSession()` method name
* Add JavaDoc to `Session.exists()`
* Add `NLIST` support for the `FtpSession.exists()` to meet the API requirements
**Cherry-pick to 4.2.x and 4.1.x**
Addressing PR comments
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4021
Previously the `SimpleMessageStore` unconditionally followed with the super class options
and provided the `lazy-load` functionality by default.
* Since `persistent` and `lazy-load` logic does not make sense for the `in-memory` store, disable it in the `SimpleMessageStore`
* Fir `AggregatorTests` for better coverage.
* NOTE: The same message can't be persisted in the Persistent `MessageStore`. The store key is fully based on the `messageId`.
And also we provide the header which indicates that the messages has been stored before. See `JdbcMessageStore.addMessage()` for example:
```
if (message.getHeaders().containsKey(SAVED_KEY)) {
Message<T> saved = (Message<T>) getMessage(message.getHeaders().getId());
if (saved != null) {
if (saved.equals(message)) {
return message;
} // We need to save it under its own id
}
}
```
* Fix (S)FTP Streaming tests to use `AbstractPersistentAcceptOnceFileListFilter` instead of raw `AcceptOnceFileListFilter`,
which relies on the object identity, but neither `FtpFile`, nor `ChannelSftp.LsEntry` provides good `equals()` and `hashCode()` implementations.
* Make `StompIntegrationTests` as `DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD` because the sporadic failure on the Travis around wrong value from the queue isn't clear (yet).
Looks like some interim event is generated by the Websocket Container on the Tomcat.
JIRA: https://jira.spring.io/browse/INT-4015https://jira.spring.io/browse/INT-3854
Initial commit.
Reworked to emit an input stream and use the file splitter.
Add StreamTransformer.
Add CLOSABLE_RESOURCE header so we can close the session automatically.
Implement INT-3854, FTP, SFTP
(S)FTP Namespace Changes
Docs - also fixes a PDF overflow
Polishing - PR Comments
checkstyle fixes
Polishing - Add Namespace for StreamParser
Polishing - PR Comments
checkstyle EmptyBlock
checkstyle fixRightCurly Script
checkstyle EmptyStatement
checkstyle RightCurly
checkstyle TailingWhite
checkstyle NeedBraces
Fix the line separator in the `fixRightCurly.gradle`
JIRA: https://jira.spring.io/browse/INT-3591
Modify `TestFtpServer` and `FtpServerOutboundTests` for file with leading whitespace
to confirm the `FtpServerFactory` has been fixed in the `commons-net-3.4`
* `gradlew clean check -x test --parallel --continue` - to collect reports
* `gradlew fixThis --parallel` - to fix all possible vulnerabilities. With `-Dfile.encoding=UTF-8` on Windows
Since the `RequireThisCheck` doesn't see parents for anonymous classes (e.g. `Runnable` callback), its report doesn't contains the outer class name with `this.`,
therefore we still have to fix those cases manually.
Thanks to the wrong `replacer` just with `this.` we have uncompilable code enough easy to find problems.
Not so easy to fix for good readability though...
* Upgrade to Grade 2.12
* Upgrade to SonarQube native plugin
The fix contains at about 300 files. So, will be done on merge.
Fix `fixThis.gradle` according PR comments
Apply `fixThis` and also `fixModifiers` for test classes.
Fix some `this.` inner issues manually.
Make code polishing for long lines after `fixThis`
Fix conflicts and vulnerabilities after the rebase
Even it isn't harmful, the one useless class in the `.file.test` package looks confusing.
Move it out to the `test` scope and change the dependant projects respectively.
JIRA: https://jira.spring.io/browse/INT-3919
Since `FtpClient` supports `null` for the `LS` command, treating it as a current `working directory`,
there is no reason to forbid `null` from the FTP adapters end-user perspective.
* Allow `null` for the `FtpSession` `list()` and `listNames()` methods
* Allow `null` in the `remote-directory` for the `<int-ftp:inbound-channel-adapter>`
* Allow `null` in the `expression` for the `FtpOutboundGateway`
Polishing - send error if error on async output
Cover `onFailure()` from `onSuccess()` with the `errorChannel`
Address PR Comments
* Get rid of `null` population for the `remoteDirectoryExpression` in the `AbstractPollingInboundChannelAdapterParser`
* Populate `new LiteralExpression(null)` from the `FtpInboundFileSynchronizer` ctor
* Introduce `buildRemotePath(parent, child)` function in the `AbstractRemoteFileOutboundGateway` with the `null` logic for `parent`
* Rework `mGetWithoutRecursion()` to use `LS` command and allow `null` for the dir.
* Fix tests according the new `mGetWithoutRecursion()` logic
Polishing
JIRA: https://jira.spring.io/browse/INT-3926
* Add `<poller>` for all `<outbound-adapter(gateway)>`
* Add/modify tests for `<queue>` as an input channel and particular `<poller>`
* Move XSD from the `xml` package to the `config` level to achieve the consistency throughout the project.
