https://build.spring.io/browse/INT-AT42SIO-365/
* Fix timing issue with the `LastModifiedFileListFilterTests`, when the `age = 1` might not be enough for the file object when we have some delay before checking
* Get rid of any `Thread.sleep()` in the test via mocking age on the file.
Cherry-picked from the 4ac3a79df7
JIRA: https://jira.spring.io/browse/INT-4146
Fixes GH-1942 (https://github.com/spring-projects/spring-integration/issues/1942)
Previously the `FileWritingMessageHandler` didn't wrap `new OutputStream()` to the `try...catch...finally` block, therefore any errors on that (e.g. `FileNotFoundException` because of permissions) cause an early exist without `inputStream.close()`
* Move a `new OutputStream()` to the `try...catch`
* Change `destinationDirectory.canWrite()` to the `Files.isWritable(destinationDirectory.toPath())` because the first one doesn't work on Windows
**Cherry-pick to 4.3.x and 4.2.x but without Files.isWritable() change**
Conflicts:
spring-integration-file/src/main/java/org/springframework/integration/file/FileWritingMessageHandler.java
Resolved.
JIRA: https://jira.spring.io/browse/INT-4142
INT-4015: Streaming Remote File Inbound Adapter
Initial commit.
Reworked to emit an input stream and use the file splitter.
Add StreamTransformer.
Add CLOSABLE_RESOURCE header so we can close the session automatically.
Implement INT-3854, FTP, SFTP
(S)FTP Namespace Changes
Docs - also fixes a PDF overflow
Polishing - PR Comments
checkstyle fixes
Polishing - Add Namespace for StreamParser
Polishing - PR Comments
Fix Streaming (S)FTP Tests
https://build.spring.io/browse/INT-B43-190/
Add `AcceptOneFileListFilter`s.
Polishing
polishing
Polishing
* Polish `.travis.yml` to avoid unnecessary work on Travis CI
JIRA: https://jira.spring.io/browse/INT-4067
When `FileSplitter` is configured with `markers = true` and file is empty, an `iterator` for file throws `IOException: Stream closed`,
because we close the `buffer` just after the first `readLine()` attempt, but still return `true` from the first `hasNext()` call
where the `this.sof` and `this.eof` are `true` for markers.
Add logic to mark internal splitter `iterator` as `done` where we don't have content and still in `sof` state.
JIRA: https://jira.spring.io/browse/INT-4053
The recent changes to the `FileSplitter` introduced (by IDE autocompletion mistake) hidden dependency of Reactor using its `Assert` class instead of Spring's one
**Cherry-pick to 4.2.x**
(cherry picked from commit eca8536)
JIRA: https://jira.spring.io/browse/INT-4046
Since not all FTP servers provide proper `STAT` command implementation,
plus the `NLIST` doesn't work properly for directories cases, introduce the `FtpRemoteFileTemplate.ExistsMode`
to let:
* to perform `STAT` by default (previous) behavior;
* to switch to `NLIST` for `FtpRemoteFileTemplate` internal use;
* perform the full `NLIST` and `FTPClient.changeWorkingDirectory()` algorithm if needed.
* Improve (S)Ftp components to use proper `RemoteFileTemplate` for internal instantiation
* Introduce `FtpMessageHandler` to wrap `FtpRemoteFileTemplate` with the proper `NLIST` `ExistsMode`
* Cover `NLIST` switching from the `FtpOutboundChannelAdapterParser` and `FtpOutboundGatewayParser`
* Document the `FtpRemoteFileTemplate.ExistsMode`
* Fix typo in the recently introduced `RemoteFileOperations.getSession()` method name
* Add JavaDoc to `Session.exists()`
* Add `NLIST` support for the `FtpSession.exists()` to meet the API requirements
**Cherry-pick to 4.2.x and 4.1.x**
Addressing PR comments
Doc Polishing
Conflicts:
spring-integration-file/src/main/java/org/springframework/integration/file/remote/AbstractRemoteFileStreamingMessageSource.java
spring-integration-file/src/main/java/org/springframework/integration/file/remote/RemoteFileOperations.java
spring-integration-file/src/main/java/org/springframework/integration/file/remote/RemoteFileTemplate.java
spring-integration-ftp/src/main/java/org/springframework/integration/ftp/session/FtpSession.java
spring-integration-ftp/src/test/java/org/springframework/integration/ftp/config/FtpOutboundChannelAdapterParserTests.java
spring-integration-sftp/src/main/java/org/springframework/integration/sftp/gateway/SftpOutboundGateway.java
spring-integration-sftp/src/main/java/org/springframework/integration/sftp/outbound/SftpMessageHandler.java
spring-integration-sftp/src/main/java/org/springframework/integration/sftp/outbound/package-info.java
JIRA: https://jira.spring.io/browse/INT-3900
The message-driven adapter should not unsubscribe if `cleanSession` is
`false`.
(cherry picked from commit 616fc79)
* `ClientWebSocketContainer`: add some synchronization to avoid race conditions: https://build.spring.io/browse/INT-B41-492
* `TomcatWebSocketTestServer`: convert to `0` port to rely on the OS resolution for `localPort`
* Add `LogAdjustingTestSupport` for STOMP test
* `SftpServerTests`: use `0` port to rely on the OS resolution for `localPort`
* `ImapMailReceiver`, `OutboundGatewayFunctionTests` (JMS), `CachingClientConnectionFactoryTests`,
`AsyncGatewayTests`, `AsyncMessagingTemplateTests`, `GatewayParserTests`, `PriorityChannelTests`, `AggregatorIntegrationTests`: increase timeout
* `EnableIntegrationTests`: use `LogAdjustingTestSupport`
* `FileOutboundChannelAdapterParserTests`: rework `Thread.sleep()` with `CountDownLatch`
* `ConnectionToConnectionTests`: increase timeout and count iteration. Previously with `1sec` we may lose some events. And we can't just rely on the `10sec`,
because the last iteration will be so long
* `TcpOutboundGatewayTests`: `500ms` is so big timeout to wait for the `Exception` that in the high load environment we can yield to other Thread so long.
Like in our case to `server` Thread to send the reply for us. Therefore decrease the Exception timeout to the `50ms` and increase server delay to `2sec`
* `StompInboundChannelAdapterWebSocketIntegrationTests`: remove `@Qualifier("taskScheduler")` as a potential candidate to test against latest SF changes.
We're fine with `SF-4.2.2` and it is just a test-case. So, I don't see reason to wait for their fix here.
STOMP: `session = null` in adapters for any transportError
Polishing
JIRA: https://jira.spring.io/browse/INT-3849
A file inbound adapter with just a `filter` attribute (no implied filter attributes)
did not get its filter injected.
The parser took an early exit from the filter configuration if none of the implied
filter attributes were specified.
Work around is to explicitly set `ignore-hidden="true"`.
JIRA: https://jira.spring.io/browse/INT-3827
Provide a hook to enable removing a file from an `AcceptOnceFileListFilter`,
for example after a message processing failure.
Make the `CompositeFileListFilter` a `ReversibleFileListFilter` so it can
delegate to any of its composed filters that are reversible.
INT-3827: Polishing - PR Comments
JIRA: https://jira.spring.io/browse/INT-3820
Some Proxy may be based on the specif non-user classes and advised with
the provided end-user interfaces.
The best sample is `@Repository` from Spring Data projects.
In this case the `MessagingMethodInvokerHelper` just missed the user interfaces
to consider for the candidate methods or thrown an `Exception` like `NoSuchMethodError`.
* Add `((Advised) targetObject).getProxiedInterfaces()` for the scanning algorithm
* Move the `UniqueMethodFilter` to the internal `Set<Method>` to allow iteration
and filtering for methods from the `targetClass` as well as from all those user interfaces
* Add Spring JPA repository test-case
* Polishing form some time-weak tests
Revert MessagingMethodInvokerHelper
INT-3820: Simple Proxy Interface Method Matching
No-risk solution for INT-3820.
Instead of considering all interfaces on the proxy, only look at proxy interfaces
for a single matching method name if we find no candidate or fallback methods.
Consider the complete solution for 4.3.
Add `Message<?>` method distinguishing
JIRA: https://jira.spring.io/browse/INT-3619
Previously the `FileReadingMessageSource` allowed overriding `filter` and `locker` properties
on the `scanner` through its own setters.
It caused confusing and unexpected behavior.
Provide refactoring to check that provided `filter` and `locker` are applied for an internal `scanner`
instance, not for the injected one.
PR Comments
JIRA: https://jira.spring.io/browse/INT-3786
* Increase `receiveTimeout` for several `QueueChannel` and `CountDownLatch` based tests
* Fix `FileSplitterTests` for Windows compatibility - `UTF-8` for bytes conversion
* Make some STOMP tests as `LongRunning`
The next fixing phase
JIRA: https://jira.spring.io/browse/INT-3773
When the `FileExistsMode` was `FAIL` or `IGNORE` and the remote file
did NOT exist, the file was not written.
JIRA: https://jira.spring.io/browse/INT-2507
When using a `HeadDirectoryScanner`, the use of any filter removed the
size limitation passed into the contstructor (or `queue-size` attribute
of a file inbound channel adapter.
The `HeadFilter` was overwritten.
Combine the head filter into a `CompositeFileListFilter` (creating one
if necessary).
JIRA: https://jira.spring.io/browse/INT-3749
Since some target application could inject there is `EvaluationContext` before, removal of
`setIntegrationEvaluationContext()` is breaking change.
This fix reverts all those setters.
JIRA: https://jira.spring.io/browse/INT-3749
Since `IntegrationEvaluationContextAware` isn't so "context-free" resource like `BeanFactory`
and `ApplicationContext`, but just a specific bean in the context, we can't follow with `BeanPostProcessor` logic - bad architecture by level of responsibility.
Therefore we should follow with standard Dependency Injection mechanism to retrieve `evaluationContext` for the particular component.
Since we can't rely on the `@Autowired` because SI can be used from the raw XML configuration,
we use utility method instead. The pattern to get the proper `integrationEvaluationContext` is:
```
@Override
protected void onInit() throws Exception {
super.onInit();
this.evaluationContext = ExpressionUtils.createStandardEvaluationContext(getBeanFactory());
}
```
JIRA: https://jira.spring.io/browse/INT-3745
Emit events whenever a message is received that can't be correlated to
a connection (or request in the case of an outbound gateway).
Also, when sending messages, wrap the exception in a `MessagingException` so that
the `TcpConnectionExceptionEvent` provides access to the failed message.
INT-3745: Polishing - PR Comments
Fix JavaDocs errors
JIRA: https://jira.spring.io/browse/INT-3737
Support returning an `InputStream` from a GET operation.
This can be used in conjuction with a `<file:splitter/>` to stream a text file.
The user is responsible for releasing the session when the download is complete.
The session object is stored in the message header and `RemoteFileUtils.closeSession()` should be called
to clean up and close the session.
INT-3737: Polishing and Docs
Fix `CachedSession.close()` bug
Preventing `Caused by: java.io.IOException: Previous raw read was not finalized`
when `Session` is returned to the pool, but `readingRaw` hasn't been finalized
because the real `close()` isn't invoked in case of normal return to pool.
JIRA: https://jira.spring.io/browse/INT-3740
For `File` and `String` payloads add `FileHeaders.ORIGINAL_FILE` and `FileHeaders.FILENAME` headers.
Reworked to create headers once only.
Fix typos and Java > 6 API usage
JIRA: https://jira.spring.io/browse/INT-3739
Since iterating splitters don't have a sequence size, it would be useful to
include a line count in the EOF marker, to facilitate some downstream process
knowing that all lines have been processed, in a multi-threaded situation.
JIRA: https://jira.spring.io/browse/INT-3727
Add a configuration option such that the `FileSplitter` can emit
Start/End messages when splitting.
Polishing according PR comments
Add toString to FileMarker
JIRA: https://jira.spring.io/browse/INT-3718
* If not set to `false` explicitly, add `IgnoreHiddenFileListFilter` as a default in `FileListFilterFactoryBean`
* Update XML Schema XSD and add `ignore-hidden` attribute
* Update + Add tests
* Add documentation
* Add default `IgnoreHiddenFileListFilter` also to `DefaultDirectoryScanner`
JIRA: https://jira.spring.io/browse/INT-3706
Previously, the `mode` attribute on the (S)FTP outbound gateways was ignored.
Take account of the mode on (M)GET and (M)PUT commands.
