JIRA: https://jira.spring.io/browse/INT-4067
When `FileSplitter` is configured with `markers = true` and file is empty, an `iterator` for file throws `IOException: Stream closed`,
because we close the `buffer` just after the first `readLine()` attempt, but still return `true` from the first `hasNext()` call
where the `this.sof` and `this.eof` are `true` for markers.
Add logic to mark internal splitter `iterator` as `done` where we don't have content and still in `sof` state.
JIRA: https://jira.spring.io/browse/INT-4046
Since not all FTP servers provide proper `STAT` command implementation,
plus the `NLIST` doesn't work properly for directories cases, introduce the `FtpRemoteFileTemplate.ExistsMode`
to let:
* to perform `STAT` by default (previous) behavior;
* to switch to `NLIST` for `FtpRemoteFileTemplate` internal use;
* perform the full `NLIST` and `FTPClient.changeWorkingDirectory()` algorithm if needed.
* Improve (S)Ftp components to use proper `RemoteFileTemplate` for internal instantiation
* Introduce `FtpMessageHandler` to wrap `FtpRemoteFileTemplate` with the proper `NLIST` `ExistsMode`
* Cover `NLIST` switching from the `FtpOutboundChannelAdapterParser` and `FtpOutboundGatewayParser`
* Document the `FtpRemoteFileTemplate.ExistsMode`
* Fix typo in the recently introduced `RemoteFileOperations.getSession()` method name
* Add JavaDoc to `Session.exists()`
* Add `NLIST` support for the `FtpSession.exists()` to meet the API requirements
**Cherry-pick to 4.2.x and 4.1.x**
Addressing PR comments
Doc Polishing
Conflicts:
spring-integration-file/src/main/java/org/springframework/integration/file/remote/AbstractRemoteFileStreamingMessageSource.java
spring-integration-file/src/main/java/org/springframework/integration/file/remote/RemoteFileOperations.java
spring-integration-file/src/main/java/org/springframework/integration/file/remote/RemoteFileTemplate.java
spring-integration-ftp/src/main/java/org/springframework/integration/ftp/session/FtpSession.java
spring-integration-ftp/src/test/java/org/springframework/integration/ftp/config/FtpOutboundChannelAdapterParserTests.java
spring-integration-sftp/src/main/java/org/springframework/integration/sftp/gateway/SftpOutboundGateway.java
spring-integration-sftp/src/main/java/org/springframework/integration/sftp/outbound/SftpMessageHandler.java
spring-integration-sftp/src/main/java/org/springframework/integration/sftp/outbound/package-info.java
Conflicts:
spring-integration-ftp/src/main/java/org/springframework/integration/ftp/gateway/FtpOutboundGateway.java
spring-integration-sftp/src/main/java/org/springframework/integration/sftp/gateway/SftpOutboundGateway.java
JIRA: https://jira.spring.io/browse/INT-3827
Provide a hook to enable removing a file from an `AcceptOnceFileListFilter`,
for example after a message processing failure.
Make the `CompositeFileListFilter` a `ReversibleFileListFilter` so it can
delegate to any of its composed filters that are reversible.
INT-3827: Polishing - PR Comments
(cherry picked from commit 0d721739e9)
Conflicts:
src/reference/asciidoc/ftp.adoc
src/reference/asciidoc/sftp.adoc
JIRA: https://jira.spring.io/browse/INT-3773
When the `FileExistsMode` was `FAIL` or `IGNORE` and the remote file
did NOT exist, the file was not written.
JIRA: https://jira.spring.io/browse/INT-3740
For `File` and `String` payloads add `FileHeaders.ORIGINAL_FILE` and `FileHeaders.FILENAME` headers.
Reworked to create headers once only.
Fix typos and Java > 6 API usage
JIRA: https://jira.spring.io/browse/INT-3739
Since iterating splitters don't have a sequence size, it would be useful to
include a line count in the EOF marker, to facilitate some downstream process
knowing that all lines have been processed, in a multi-threaded situation.
JIRA: https://jira.spring.io/browse/INT-3727
Add a configuration option such that the `FileSplitter` can emit
Start/End messages when splitting.
Polishing according PR comments
Add toString to FileMarker
(cherry picked from commit 51b52ea)
JIRA: https://jira.spring.io/browse/INT-3706
Previously, the `mode` attribute on the (S)FTP outbound gateways was ignored.
Take account of the mode on (M)GET and (M)PUT commands.
Conflicts:
src/reference/asciidoc/ftp.adoc
src/reference/asciidoc/sftp.adoc
JIRA: https://jira.spring.io/browse/INT-3620
Add `setShouldAppendNewLine(boolean shouldAppendNewLine)` method to the
`FileWritingMessageHandler` class. Added similar test methods to the
`FileWritingMessageHandlerTests` class.
INT-3620: `appendNewLine` just only for `FileExistsMode.APPEND`
Also add `appendNewLine` for `File` payload
INT-3620: `appendNewLine` for any `fileExistsMode`
JIRA: https://jira.spring.io/browse/INT-3606
Close the store when an (S)FTP synchronizing MessageSource is stopped.
Will need another commit on 4.2.WIP to change to `Lifecycle`.
Add dirty Flag to Properties MetadataStore
Avoid unnecessary persists.
Remove unnecessary implementation of `AbstractInboundFileSynchronizingMessageSource#getComponentType()`
JIRA: https://jira.spring.io/browse/INT-3584
INT-3584: Move `iterator` option to ctor
Polishing
- Javadocs
- Fix String payload (missing 'else')
- Support charsets other than the default
- throw MessagingException instead of RuntimeException
- Add more tests to cover all cases
INT-3584: Throw `MessageHandlingException`
JIRA: https://jira.spring.io/browse/INT-3572
Previously, a blocking queue was used to hold seen files; this was used to enable
FIFO when a max capacity is set.
When used with no capacity, a queue is not needed; also the `contains` operation
on a queue requires a linear search which does not scale well for a large number
of files.
Use a `HashSet` instead to significantly improve the `contains` performance.
Only maintain a queue if a max capacity is set.
Fix `AcceptOnceFileListFilterTests` to be compatible with Java < 8
This commit updates AbstractHeaderMapper in a number of ways:
* The 'userDefinedHeaderPrefix' property has been removed as some
advanced tests revealed that the feature is actually broken right
now and there is no easy way to fix it. A new richer hook point
that indicates if the property comes from the target object or the
standard MessageHeaders has been added.
* A HeaderMatcher interface has been introduced to replace the
lengthy checks in shouldMapHeader. Several implementations
of that interface are provided
* An additional pattern has been added that maps any header that is
*not* a standard header. It uses the standardHeaderPrefix property
for that purpose
* A number of protected method that were only used to create the
instance have been removed in favour of a non default constructor.
Subclasses should provide those *static* values in their own default
constructor.
* The list of transient headers can now be customized. Transient
headers are headers that should never be mapped. The standard
ErrorChannel and ReplyChannel headers have also been removed from
the default transient headers list as they no longer need to be
transient
header-mapper: Polishing
JIRA: https://jira.spring.io/browse/INT-3412
Initial commit - review only.
TODO:
- SFTP Tests
- Namespace/Adapter support for file append
- Docs
INT-3412 Polishing
- Addressed PR comments
- Completed SFTP implementation
- Added namespace/parser support for `FileExistsMode` (append, etc)
- Added SFTP Tests
- Created Embedded SFTP server for tests (similar to FTP)
- Converted tests that needed a real server to use the embedded server
INT-3412 Docs and Polish (PR Comments)
JIRA: https://jira.spring.io/browse/INT-3464
Previously, if an IO exeption occurred while synchronizing
files, and an AcceptOnceFileListFilter is being used, files that were
not transferred would not be fetched next time.
Add strategy `ReversibleFileListFilter` that can rollback previously
accepted files.
Implement this interface on `AcceptOnceFileListFilter` and
`AbstractPersistentAcceptOnceFileListFilter`.
Add test cases.
Polishing
JIRA: https://jira.spring.io/browse/INT-3465
Add support for adding/removing individual recipients to the RecipientListRouter
Modify documentation in what's new and spring-integration-4.1.xsd
Polishing
`AbstractRemoteFileOutboundGateway`: close `outputStream` before `file.delete()` to release exclusive file-lock
JIRA: https://jira.spring.io/browse/INT-2438
If a get operation fails in the outbound gateway,
clean up (remove) the temporary file if present.
Test Polishing
JIRA: https://jira.spring.io/browse/INT-3404
Previously, `AbstractInboundFileSynchronizingMessageSource` implemented both `MessageProducer` and `MessageSource`.
The `MessageProducer` is really redundant and confusing.
Also, it might produce some side-effects when using JavaConfig.
Provide some other simple but important fix for all `AbstractMessageSource` implementation to invoke `super.afterPropertiesSet()`
Polishing
Add mock bean factory to failing tests.
Revert state assertions in AbstractInboundFileSynchronizingMessageSource;
the fields are state, not arguments, at the time they are being
asserted.
JIRA: https://jira.spring.io/browse/INT-3374
Move `getMessageBuilderFactory` from `IntegrationContexUtils`
to `IntegrationUtils`.
Change DOS Newlines to Unix
`AbstractMessageGroupStore`.
JIRA: https://jira.spring.io/browse/INT-3349
Several FactoryBeans did not propagate the BeanFactory
to their created object(s). Beans that create messages
must have access to a bean factory to get the
message builder factory.
Fix the FactoryBeans and add a mock FB to all tests that
need one.
Add a runtime environment variable to make any infractions
fatal. This should be set to `true` on CI builds and on
framework developer environments.
JIRA: https://jira.spring.io/browse/INT-3319
The `MessageBuilderFactory` abstraction relies on the bean factory
being propagated to all classes that create messages.
Some classes were missed in the initial PR.
JIRA: https://jira.spring.io/browse/INT-3041
Add namespace support to simplify configuration of
a `RequestHandlerRetryAdvice.
INT-3041 Polishing; PR Comments
* Allow a retry-advice element within the request-handler-advice-chain
* Clean up schema (should not have allowed `synchronization-factory` on
the request-handler-advice-chain.
INT-3041 Polishing; PR Comment
This is still a work in process.
There are a bunch of TODOs in classes
that are not managed by Spring and so
need to have the MessageBuilderFactory
injected.
But I am looking for feedback on the
approach.
INT-3309 Resolve TODOs
Provide access to the MessageBuilderFactory in all classes.
INT-3309 Polishing + Tests
* Fallback to 'fromMessage()' if mutating and inbound message is not MutableMessage
* Add 'alwaysMutate' boolean to MutableMessageBuilderFactory - coerces 'fromMessage' calls to 'mutateMessage'
* Add tests
INT-3309 Polishing; PR Comments
Also add tests to parent/child contexts where the parent has the
default message builder and the child has a mutable message builder.
INT-3309 More Polish; PR Comments
Also fix removeHeader in MMB.
- ((CachingSessionFactory<?>.CachedSession) session).dirty();
+ ((CachingSessionFactory.CachedSession) session).dirty();
Failed to compile with Java6 (compiles ok with Java7 with
sourceCompatibility=1.6).
If an exception occurs on a session it should be physically closed
and not reused because its state is indeterminate and the next
operation might fail.
Note: The booleans within CachedSession do not need to be volatile
because it is a short-lived object only used by the current thread.
Also fixes the assertion message in RFT.get().
JIRA: https://jira.springsource.org/browse/INT-3266
INT-3266 Polishing - PR Comments
Remove need for SuppressWarnings.
Fix 2 test cases where the exception has an additional cause.
