JIRA: https://jira.spring.io/browse/INT-4324
The `preserve-timestamp` attribute for the
`<int-file:outbound-channel-adapter>` has been missed for parsing in
the `FileWritingMessageHandlerBeanDefinitionBuilder` and propagation
in the `FileWritingMessageHandlerFactoryBean`
**Cherry-pick to 4.3.x**
Fixspring-projects/spring-integration-java-dsl#152
* Add Logic to the `FileInboundChannelAdapterSpec` do not populate
`FileListFilterFactoryBean` to the `FileReadingMessageSource`
when the `scanner` is provided.
Otherwise call the `FileReadingMessageSource` and rely on its logic to
refuse mutually exclusive `scanner` and `filter/locker` options
JIRA: https://jira.spring.io/browse/INT-4308
* Document `JdbcMetadataStore`
* Remove unnecessary context DERBY config for the `PersistentAcceptOnceFileListFilterExternalStoreTests`
* Rework the `testFileSystemWithJdbcMetadataStore()` to use `EmbeddedDatabase` directly, without context
JIRA: https://jira.spring.io/browse/INT-4306
Fixed checkstyle errors
Fixed checkstyle errors
Fixed checkstyle errors and sql schema
Manually fixed derby sql schema
Fixing some issues
added region support
small refactorings
test checkstyle fix
integration tests and fixes
integration tests and fixes
JIRA: https://jira.spring.io/browse/INT-4305
The `File.renameTo()` operation may fail, therefore the content of the
local file isn't changed, but since we change `setLastModified()` anyway,
this file might be eligible for local polling.
So, we end up with the same content from local file in a new message,
meanwhile we expect a new content from the remote file
* Check the `File.renameTo()` result and attempt to `delete()`
for existing local file
* When file isn't renames remove the remote file from the `filter` to let
it be transferred one more time on the next poll.
The local file might be opened for processing, so this way we postpone a fresh
remote file for the future poll rounds
* Modify `copyFileToLocalDirectory()` to return `boolean` to reflect the fact of copy.
This way we check the real number of transferred files
**Cherry-pick to 4.3.x**
Do not transfer remote file if we can't remove local one
Polishing Log Messages
JIRA: https://jira.spring.io/browse/INT-4297
* Add `FileSplitter.setFirstLineAsHeader()` and extract the first line from
the context before `iterator`.
* Populate the extracted header to each subsequent line
* Don't use `Streams` to read first line
* Add XSD and Java DSL option for the `firstLineAsHeader`
* Document the feature
* Add Java DSL sample for the `Files.slitter()`
Add a note about more complex header enrichment
Doc Polishing and fix field name.
JIRA: https://jira.spring.io/browse/INT-4283
Filter that only passes files where a corresponding "marker" file is also present.
Implementations for file and(S)FTP.
* Polishing - PR Comments
* Missed one
JIRA: https://jira.spring.io/browse/INT-4269
* Check if `outputChannel` is `ReactiveStreamsSubscribableChannel`
to let back-pressure splitting
* Build `Flux` or `Iterator` depending in the `reactive` state
* Allow to get a size of the `iterator` if it is possible,
for example `XPathMessageSplitter`
Add tests
Fix raw type and unused import
* Add JavaDocs to the `AbstractMessageSplitter#obtainSizeIfPossible()`
* Add asserts for the `sequenceSize` populataiton in the `XPathMessageSplitter`
* Document `Stream` & `Flux` support in the splitter
Minor doc polishing
JIRA: https://jira.spring.io/browse/INT-4117
Add an option to set file permissions on the `FileWritingMessageHandler` when the file system supports it.
Polishing - PR Comments; more robust file system detection.
Add DSL support and fix NPE with missing timestamp header.
Fix javadoc
Fix javadoc
Some polishing:
* Add `chmod >= 0` assertion
* Make some assertion with string concatenation as `Supplier`-based to
deffer that string concatenation
https://build.spring.io/browse/INT-MASTER-675/
Since SF is still based on the `Bismuth-M1`,
we can't afford ourselves to go to `BUILD-SNAPSHOT` yet
Also fix `FileTests` do not poll the same file again for `splitter` case
JIRA: https://jira.spring.io/browse/INT-4260
MPEH: Populate the `ErrorMessage.originalMessage`,
if available and not the same as the `failedMessage`.
Polishing - rename exception to `MessagingExceptionWrapper`;
make `MPEH` a subclass of `ErrorMessagePublisher`
PR Comments
Polishing - PR Comments
Using a `LiteralExpression` in the `AbstractRemoteFileOutboundGateway` (when the
user provides a `File` for the directory) causes us to create a new `File` object
each time, instead of using the user-supplied `File`.
Use a `ValueExpression` instead.
Move the `FileWritingMessageHandler` logic to `ExpressionUtils` and use it from both places.
Also add a String setter variant for convenient Java Configuration.
JIRA: https://jira.spring.io/browse/INT-4086
The remote files may be carried with sub-path, not just file name.
The same sub-dirs logic can be achieve with the `localFilenameGeneratorExpression`
* Add the logic into `AbstractInboundFileSynchronizer` to build sub-dirs for local files
* Switch on `WatchService` for the internal `FileReadingMessageSource` in the `AbstractInboundFileSynchronizingMessageSource` to react for all the changes in the `localDirectory`
Introduce `RecursiveDirectoryScanner` and use it for `AbstractInboundFileSynchronizingMessageSource` by default
* Add `maxDepth` and `fileVisitOptions` options to the `RecursiveDirectoryScanner`
* Polishing JavaDocs, tests and docs
JIRA: https://jira.spring.io/browse/INT-4244
The `JpaOutboundGateway` returns a whole `Message` for reply copying request headers.
Building the reply a `AbstractMessageProducingHandler` uses `MessageBuilder` to create a new `Message` with all request headers because of `shouldCopyRequestHeaders() == true`
* To avoid some overhead with new `Message` and `MessageBuilder` for nothing, just return a JPA result as is and let super class to build the Message
* Optimize all reply producers to return `MessageBuilder` to the super `handleMessage()` process
* Return the whole `gatherResult` `Message<?>` from the `ScatterGatherHandler` to have ability to transfer reply headers as well
* Mark `DelayHandler` as `shouldCopyRequestHeaders() return false` since it doesn't modify Message and returns it as is after delay
JIRA: https://jira.spring.io/browse/INT-4243
Remove newline char from key exchange.
Fix CR and Add Windows Delete Diagnostics
Fix For Windows
More diagnostics.
It appears the new server doesn't close the file when the session is closed.
In the streaming test, consume the stream before closing the session.
Polishing - PR Comments
* More polishing according PR comments
JIRA: https://jira.spring.io/browse/INT-4237
It was possible to flush (close) the file while a write was in process; more likely when
`flushWhenIdle` is false.
This probably would not occur in the real world, just tests with short flush intervals,
but certainly possible.
There is already a lock used to prevent concurrent writes while appending; use the same
lock when flushing.
JIRA: https://jira.spring.io/browse/INT-4232
Currently to implement the synchronization logic when remote file is modified there is only one way: remove local file and provide a `localFilter` which can be clear from the removed file as well.
The Reference Manual claims that `FileSystemPersistentAcceptOnceFileListFilter` can let us pick up a fresh version of the remote file if we configure `preserveTimestamp = true`, but actually `AbstractInboundFileSynchronizer` just bypass that logic because it check a local file for existence.
* Modify `AbstractInboundFileSynchronizer` to compare `lastModified` of the remote file with local version if `preserveTimestamp == true`
* Also replace `AcceptOnceFileListFilter` to the `FileSystemPersistentAcceptOnceFileListFilter` since the first one doesn't care about `lastModified`.
If `preserveTimestamp` isn't in use the `FileSystemPersistentAcceptOnceFileListFilter` works fully similar to the `AcceptOnceFileListFilter`
**Cherry-pick to 4.3.x except test-case**
Make `SftpStreamingMessageSourceTests` compatible with Windows
Looks like SFTP embedded server provides different permissions for files on different OSs
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4233
INT-4233: Add (S)FTP FileInfo Header (Streaming)
JIRA: https://jira.spring.io/browse/INT-4233
Add the complete file info as JSON (when Jackson or Boon available) to the message headers
when streaming inbound.
Provide a mechanism to configure Boon to provide similar output to Jackson.
Also allow subclasses to provide their own object mapper.
Also clean up after the AMQP DSL tests, and don't use a queue `foo`. I often have such a queue
with content; this caused tests to fail.
Use SimpleJsonSerializer
Doc Polishing
Polishing - PR Comments
More Polishing
* Polishing according latest PR comments
JIRA: https://jira.spring.io/browse/INT-4231
For `FileWritingMessageHandler` and (S)FTP outbound gateways, support
`FileExistsMode.REPLACE_IF_MODIFIED` to allow overwriting an existing file if
the source file modified time is different to the existing file.
Polishing
JIRA: https://jira.spring.io/browse/INT-4216
Add test for running empty array through `ChainFileListFilter`
Assure no exceptions are throw when filter input is empty.
INT-4216: Add fix for ChainFileListFilter edge case
INT-4216: Fix checkstyle error on test class
Add copyright header to test file
* Polishing for `ChainFileListFilter` JavaDocs
* Move `CompositeFileListFilterTests` and `ChainFileListFilterIntegrationTests` to the `filters` package instead of `file` core
JIRA: https://jira.spring.io/browse/INT-4215
Due to type erasure we can't create an empty generic array from an empty list.
Don't attempt to call remaining filters when the list is empty.
JIRA: https://jira.spring.io/browse/INT-4212
Add `firstWrite` to predicate methods.
Add DSL support for `flushWhenIdle`
While these are breaking API changes on the interfaces, I feel this is
likely not going to affect many users and this is a major release after
all.
I have added a note to the 5.0 migration guide.
JIRA: https://jira.spring.io/browse/INT-4212
Add an option to flush after the `flushInterval`, regardless of intermediate writes.
Rename `lastFlush` to `firstWrite`
Polishing
**Cherry-pick to 4.3.x**
JIRA: https://jira.spring.io/browse/INT-4206
* Fix unnecessary dependency resolution in BOM module
* Fix `MessagingMethodInvokerHelper` to handle `$MockitoMock$` generated classed which isn't CGLib `Proxies` any more
* Provide fixes for test classes according upgrade to Mockito `2.5`
* Fix Ceckstyle do not allow static imports for deprecated Mockito classes
JIRA: https://jira.springsource.org/browse/INT-4188
* add `<idle-event-interval>` XSD element
* add `FileTailingIdleEvent`
* move `TaskScheduler` and `getRequiredTaskScheduler` to `FileTailingMessageProducerSupport`
* add `setIdleEventInterval`
use taskExecutor instead of creating one
stop the scheduled task in case of `doStop()`
other required changes
update Test and Reference
implement changes required
What do you think about this checking if file exist before sending event.
Mock is used for `file.exist()` plus other minor updates
Polishing
* Some typos in the code as well as in the docs
* Fix `FileTailingMessageProducerTests.testIdleEvent()` to verify that `FileTailingIdleEvent` isn't emitted when file does not exist
JIRA: https://jira.spring.io/browse/INT-4187
Add javadoc and test
- assertTrue on default adapter
- assertFalse check on nativeAdapter
- update XSD to add enable-status-reader
add `@auther`
add example in reference document
update adoc
* Fix minor typos in docs and allow to configure `enable-status-reader` with property placeholder
**Cherry-pick to 4.3.x**
Change all the `BUILD-SNAPSHOT`s to their latest Milestones
Fix compatibility with those Milestones
Upgrade to Spring AMQP 2.0.0.M1
Upgrade to Spring Data Kay
Gemfire now is based on the Apache Geode, so changed all the imports to proper new packages
MongoDB now is based on the Mongo 3 Driver, therefore many breaking changes. Mostly to the mapping part
Fix Checkstyle violation
Fix race condition in the `MongoDbInboundChannelAdapterIntegrationTests` around `QueueChannel` and tx commit
Add `-s` to Travis Gradle command to see stack trace about `MongoDbMetadataStoreTests` problem
Test only MongoDB module on Travis with -d
Add addon to Travis to pull MongoDB-3.0
Fix `MongoDbAvailableRule` for MongoDB 3.0 Driver style
Increase `serverSelectionTimeout` to `100` in the `MongoDbAvailableRule`.
Looks like `0` isn't good value to get immediate answer
JIRA: https://jira.spring.io/browse/INT-4155
- Add streaming support
- Rework tests to use dynamic flow registrations due to the test FTP server is now a test support class.
Javadoc fixes
Checkstyle fix
Polishing - PR Comments
Polishing and Port SFTP
JIRA: https://jira.spring.io/browse/INT-4154
* Move `Transformers` for files to the `Files` factory
* Remove `@Deprecated` methods
* Fix `FilesTests` for new packages and removed methods like `.handleWithAdapter()`
Provide JavaDocs for File DSL and implement some new methods like `chmod()` and `renameFunction()`
Fix race condition in the `ScriptsTests` when even small time window delay may lead us to one more message in the queue
JIRA: https://jira.spring.io/browse/INT-4147
By default `FileReadingMessageSource` is created without `filter`, at the same time internal `WatchServiceDirectoryScanner` is supplied with default `filter` by its `DefaultDirectoryScanner` super class.
A `DELETE` watch event condition around `FileReadingMessageSource.this.filter` is wrong, because it is `null` by default. Therefore `DELETE` events never succeed
* Modify `FileInboundTransactionTests` to accept `DELETE` watch event as well and verify that `ResettableFileListFilter.remove(File)` is performed
* Call newly created getter for `DefaultDirectoryScanner.filter` in the `WatchServiceDirectoryScanner` to assert against supplied `filter`
Polishing
**Cherry-pick to 4.3.x**
JIRA: https://jira.spring.io/browse/INT-832
When we scan directory recursively for files (e.g. `WatchServiceDirectoryScanner`), it can be useful to get access to the relative path from the `Message`, e.g. on the `FileWritingMessageHandler` side to restore the original structure
* Add a `FileHeaders.FILENAME` into the outbound `Message` from the `FileReadingMessageSource`
* The result of that header is like a removal of leading `this.directory.getAbsolutePath()` in the target `File.getAbsolutePath()`.
In case of not recursion we get just only regular file name.
* Introduce `FileHeaders.RELATIVE_PATH`
* Populated `FileHeaders.RELATIVE_PATH`, `FileHeaders.FILENAME`, `FileHeaders.ORIGINAL_FILE` in the `FileReadingMessageSource`
* File `FileTailingMessageProducerSupport` to populate `FileHeaders` properly
* Introduce ctor for the `LastModifiedFileListFilter` for better Java configuration experience
* Add docs for changes
Doc Polishing
AMQP Lambdas
Event Lambdas
Feed/File Lambdas
Gemfile/Groovy Lambdas
* Ensure that JavaDocs are checked via ` check.dependsOn javadoc`
* Add `@return` tag to the `MessageBuilder.readOnlyHeaders()`
* Upgrade to Checkstyle `7.1`
* Relax `RequireThis` rule a bit. Right now it does the effort only in case of overlapping. See https://github.com/checkstyle/checkstyle/issues/2362 for more info
* Enable some annotation rules and provide fixes for violations
* Enable `tabs indents` rule. This was the biggest fix in this PR
* Resolve `TODO` in the `MessagingMethodInvokerHelper` and fix tests to meet `IllegalStateException` now
Remain Checkstyle version `6.16.1` and reinstate `RequireThis` rule
The latest Checkstyle has a bug with local scope variables if they have the same names as property.
Revert some literals splitting
Fix some line length exceeding and code style
JIRA: https://jira.spring.io/browse/INT-4096
* Introduce `spring.integration.readOnly.headers` Integration property
* Introduce `IntegrationMessageHeaderAccessor.setReadOnlyHeaders()` and use it from the overridden `IntegrationMessageHeaderAccessor.isReadOnly()`
* Add `DefaultMessageBuilderFactory.setReadOnlyHeaders()` and delegate the value to the new `MessageBuilder.readOnlyHeaders()`
* Modify `spring.integration.properties` for the `contentType` as a `readOnly` header for testing
* Ensure that provided logic works via an appropriate modification to the `ObjectToJsonTransformerParserTests`
Increase receive timeouts in the `OutboundGatewayFunctionTests`
Fix `FileInboundTransactionTests` for slow `WatchService` issue on Linux/OS X
Also redo the `tmp` dir logic to the `TemporaryFolder` `@ClassRule`
Fix unused `import` in the `FileInboundTransactionTests`
Polishing
Conflicts:
spring-integration-core/src/main/java/org/springframework/integration/context/IntegrationProperties.java
spring-integration-core/src/main/resources/META-INF/spring.integration.default.properties
Some polishing and documentation
Localize `readOnly.headers` customization only in the `ObjectToJsonTransformerParserTests-context.xml`
Otherwise ti might affect many other tests and the search for the reason of failure would be enough complicated
Doc Polish
JIRA: https://jira.spring.io/browse/INT-4108
Some `Lifecycle.start()/stop()` usage doesn't ensure robustness for components causing unexpected and difficulty tracing issues
* Fix `Lifecycle.start()/stop()` for `FileReadingMessageSource`, `FileWritingMessageHandler`, `AbstractMqttMessageHandler`
* In the `DefaultHeaderChannelRegistry`, `LockRegistryLeaderInitiator`, `MqttPahoMessageHandler` rework logic for shared variables to avoid `NPE`
* Increase receive timeouts in the `PayloadDeserializingTransformerParserTests` and `UdpChannelAdapterTests`
* Prove with the `WatchServiceDirectoryScannerTests` changes that several invocation for `FileReadingMessageSource.start()` are idempotent
**Cherry-pick to 4.3.x**
JIRA: https://jira.spring.io/browse/INT-3913
* Remove deprecated classes and methods/constructors, deprecated XML attributes
* Remove `TcpConnectionEventListeningMessageProducer` and rework tests logic to the `ApplicationEventListeningMessageProducer`
* Fix several typos
* Remove/rework deprecated entities mentioning
JIRA: https://jira.spring.io/browse/INT-4095
Limit the number of remote files fetched on each poll (when it is necessary to fetch files).
Polishing - PR Comments
Polishing - Decouple MaxFetchSize from Poller
Polishing - PR Comments
Schemas and Docs
More Polishing
* Polishing according PR comments
JIRA: https://jira.spring.io/browse/INT-4067
When `FileSplitter` is configured with `markers = true` and file is empty, an `iterator` for file throws `IOException: Stream closed`,
because we close the `buffer` just after the first `readLine()` attempt, but still return `true` from the first `hasNext()` call
where the `this.sof` and `this.eof` are `true` for markers.
Add logic to mark internal splitter `iterator` as `done` where we don't have content and still in `sof` state.
