* Add `RotatingServerAdvice.StandardRotationPolicy.getCurrent()` method
for better end-user experience when this class is extended
* Add NPE protection into the MQTT Channel Adapters.
Some code style polishing for them
**Cherry-pick to 5.0.x**
If exception happens in the `callback.doWithInputStream(inputStream)`,
we don't close the `inputStream = session.readRaw(remotePath)`.
* Move the `InputStream.close()` to the `finally` block of the
`SessionCallback` action in the `RemoteFileTemplate.get()`
**Cherry-pick to 5.0.x and 4.3.x**
JIRA: https://jira.spring.io/browse/INT-4531
The `DelegatingSessionFactory` uses plain `Object` as key type
whereas the default `RotatingServerAdvice` does not.
* Change the type from `String` to `Object` in the `RotatingServerAdvice.KeyDirectory`
* Fix typos in the (s)ftp.adoc
**Cherry-pick to 5.0.x**
(cherry picked from commit 5bfd6971db)
JIRA: https://jira.spring.io/browse/INT-4498
* Use `FileCopyUtils` instead of `StreamUtils` in test-cases when we deal
with `InputStream` s and files resources.
* Fix reflection-based tests with AOP to use an `ArrayList` access
instead of `LinkedList` since an optimization in SF:
6d6aa72e8f
**Cherry-pick to 5.0.x and 4.3.x**
(cherry picked from commit a1f016a)
JIRA: https://jira.spring.io/browse/INT-4495
Previously, `maxFetch` was applied before directories were removed from
the fetch list.
Remove the directories before filtering and applying `maxFetch`.
* Polishing - PR Comments
* More polishing
* More polishing.
* Check for empty array.
* Remove test main method.
INT-4495: Fix SFTP tests
JIRA: https://jira.spring.io/browse/INT-4486
The `SmartLifecycle.stop(Runnable callback)` must always call the
`callback` in the end independently of the internal state
* Revise all the `SmartLifecycle` implementations for the proper
`callback` handling
**Cherry-pick to 5.0.x and 4.3.x**
JIRA: https://jira.spring.io/browse/INT-4351
The `WatchService` reacts to the events in the file system and keep
track ove the events until we poll them, e.g. via `listEligibleFiles()`
in the `WatchServiceDirectoryScanner`.
On the other hand the `SourcePollingChannelAdapter` calls the mentioned
`listEligibleFiles()` according its polling period.
At this moment the `FileListFilter` is applied to the polled files.
It may happen that `LastModifiedFileListFilter` can't accept too young
files yet and they are lost for the future consideration.
* To allow, for example, to retain young files by the
`LastModifiedFileListFilter` judgment for the future cycles add
`DiscardAwareFileListFilter` with the `DiscardCallback` support.
The `WatchServiceDirectoryScanner` now registers such a callback into
the filter and stores discarded files into the `filesToPoll` queue
for the future poll cycle.
**Cherry-pick to 5.0**
Fix compilation warnings
* Replace `DiscardCallback` with the plain `Consumer`
* Ensure uniqueness in the `WatchServiceDirectoryScanner` internal
queue via a `Set` implementation, since discard callback may be called
several times for the same file from the `CompositeFileListFilter`
according to its nature
* Add JavaDocs and Docs
Change `@since` to `5.0.5`
JIRA: https://jira.spring.io/browse/INT-4430
When Iterator-based `FileSplitter` splits the file, and an exception
throws in downstream flow (in the same thread), the exception propagates
to the caller leaving underlying file reader opened.
This commit changes `AbstractMessageSplitter` the way, that,
when any exception happens, if Iterator implements `java.io.Closeable`,
its `close()` method will be called before propagating exception.
Also `FileSplitter`'s underlying iterator implements `Closeable` now.
* Make `CloseableIterator` to follow `Closeable` contract.
Now `CloseableIterator.close()` declares `IOException` and can be used
as base interface for `FunctionIterator`.
* Adjust tests.
Adjust tests to reflect the fact that we call `close()` on the reader
one more time in the end of iterator.
**Cherry-pick to 5.0.x and 4.3.x**
(cherry picked from commit 7f25cba)
JIRA: https://jira.spring.io/browse/INT-4382
There is a deadlock in the FMWH due to lock ordering.
Writes try to lock the handler monitor while holding the file lock.
The `Flusher` tries to lock the file lock while holding th handler monitor.
- Change the `Flusher`, `stop` and `flushIfNeeded` methods to not call close
on the file state while holding the handler monitor.
__cherry pick to 4.3.x__ (fix <> for Java 6)
Polishing - PR Comments
* Optimize `flushIfNeeded(FlushPredicate)` to delegate only to the
`flushIfNeeded(MessageFlushPredicate, Message<?>)`
To avoid `Message` re-creation during `AbstractMessageSource.receive()`
logic, refactor `AbstractMessageSource` implementations
to return `AbstractIntegrationMessageBuilder`
* Add `AbstractIntegrationMessageBuilder<File> doReceive()` to the
`FileReadingMessageSource` to be called from the `AbstractInboundFileSynchronizingMessageSource`
to avoid message recreation in its `doReceive()`
* Some code style refactoring in the `AbstractMessageSource`
* INT-4376: Upgage to Log4J 2
JIRA: https://jira.spring.io/browse/INT-4376
* Deprecate Log4J 1.x components in favor of newly added
a `Log4j2LevelAdjuster` JUnit `@Rule`
* Update all the logging configuration to Log4J 2
* Polishing after testing
JIRA: https://jira.spring.io/browse/INT-4344
The `FtpOutboundGateway` provides ctors without expression for remote path.
In this case it is treated as a `working directory` but only for the
`LS`, `NLST`, `PUT` and `MPUT` commands.
* Add assertion in the `AbstractRemoteFileOutboundGateway` to discard
configuration for all other commands when `expression` is `null`
**Cherry-pick to 4.3.x**
Fixesspring-projects/spring-integration#2253
For backward compatibility and use-cases when processed files are stored
in sub-directories in the `localDirectory` revert the
`AbstractInboundFileSynchronizingMessageSource` logic back to the
`DefaultDirectoryScanner`.
The `RecursiveDirectoryScanner` can be injected via `scanner` option
JIRA: https://jira.spring.io/browse/INT-4356
**Cherry-pick to 4.3.x**
Address PR comments:
* Remove unused variable in the test case
* Add string-based SpEL setters to the `AbstractInboundFileSynchronizer`
* Fix `@since` in the `FileTransferringMessageHandler` to the proper version
JIRA: https://jira.springsource.org/browse/INT-4334
* Add a `DirectoryScanner` setter named `setScanner` in
`AbstractInboundFileSynchronizingMessageSource` to set the
`FileReadingMessageSource`'s scanner.
* Add tests to `AbstractRemoteFileSynchronizerTests`.
Polish.
Updates xsd, tests and documentation.
Fix xsd formatting.
Addresses review comments.
* Some code style polishing
* Add `validateOnlyOverlapping = false` for `RequireThis` Checkstyle rule
* Add `this.` to caught places
* Use `CustomizableThreadFactory` instead of raw interface in the
`UnicastSendingMessageHandler`
JIRA: https://jira.spring.io/browse/INT-4324
The `preserve-timestamp` attribute for the
`<int-file:outbound-channel-adapter>` has been missed for parsing in
the `FileWritingMessageHandlerBeanDefinitionBuilder` and propagation
in the `FileWritingMessageHandlerFactoryBean`
**Cherry-pick to 4.3.x**
Fixspring-projects/spring-integration-java-dsl#152
* Add Logic to the `FileInboundChannelAdapterSpec` do not populate
`FileListFilterFactoryBean` to the `FileReadingMessageSource`
when the `scanner` is provided.
Otherwise call the `FileReadingMessageSource` and rely on its logic to
refuse mutually exclusive `scanner` and `filter/locker` options
JIRA: https://jira.spring.io/browse/INT-4308
* Document `JdbcMetadataStore`
* Remove unnecessary context DERBY config for the `PersistentAcceptOnceFileListFilterExternalStoreTests`
* Rework the `testFileSystemWithJdbcMetadataStore()` to use `EmbeddedDatabase` directly, without context
JIRA: https://jira.spring.io/browse/INT-4306
Fixed checkstyle errors
Fixed checkstyle errors
Fixed checkstyle errors and sql schema
Manually fixed derby sql schema
Fixing some issues
added region support
small refactorings
test checkstyle fix
integration tests and fixes
integration tests and fixes
JIRA: https://jira.spring.io/browse/INT-4305
The `File.renameTo()` operation may fail, therefore the content of the
local file isn't changed, but since we change `setLastModified()` anyway,
this file might be eligible for local polling.
So, we end up with the same content from local file in a new message,
meanwhile we expect a new content from the remote file
* Check the `File.renameTo()` result and attempt to `delete()`
for existing local file
* When file isn't renames remove the remote file from the `filter` to let
it be transferred one more time on the next poll.
The local file might be opened for processing, so this way we postpone a fresh
remote file for the future poll rounds
* Modify `copyFileToLocalDirectory()` to return `boolean` to reflect the fact of copy.
This way we check the real number of transferred files
**Cherry-pick to 4.3.x**
Do not transfer remote file if we can't remove local one
Polishing Log Messages
JIRA: https://jira.spring.io/browse/INT-4297
* Add `FileSplitter.setFirstLineAsHeader()` and extract the first line from
the context before `iterator`.
* Populate the extracted header to each subsequent line
* Don't use `Streams` to read first line
* Add XSD and Java DSL option for the `firstLineAsHeader`
* Document the feature
* Add Java DSL sample for the `Files.slitter()`
Add a note about more complex header enrichment
Doc Polishing and fix field name.
JIRA: https://jira.spring.io/browse/INT-4303
Don't emit ignored files in the output message payload.
Also, always preserve the timestamp when using `FileExistsMode.REPLACE_IF_MODIFIED`.
Some code style polishing
JIRA: https://jira.spring.io/browse/INT-4271
The `Flusher.run()` is `synchronized` on the `FWMH`; a `stop()` while it's running will interrupt the task
which could cause a state to be removed without a flush actually happening (interrupt
on lock acquisition).
- synchronize `stop()` so it won't interrupt a running flusher
- detect an interrupt in the flusher and stop flushing
**Cherry-pick to 4.3.x**
JIRA: https://jira.spring.io/browse/INT-4296
The `FileState.close()` method correctly acquires the lock to prevent closing while
writing. However, with a short flush interval, the close could occur between state
creation and the write operation acquiring the lock.
Move the state creation to within the scope of the lock acquired by the writer.
JIRA: https://jira.spring.io/browse/INT-4283
Filter that only passes files where a corresponding "marker" file is also present.
Implementations for file and(S)FTP.
* Polishing - PR Comments
* Missed one
JIRA: https://jira.spring.io/browse/INT-4269
* Check if `outputChannel` is `ReactiveStreamsSubscribableChannel`
to let back-pressure splitting
* Build `Flux` or `Iterator` depending in the `reactive` state
* Allow to get a size of the `iterator` if it is possible,
for example `XPathMessageSplitter`
Add tests
Fix raw type and unused import
* Add JavaDocs to the `AbstractMessageSplitter#obtainSizeIfPossible()`
* Add asserts for the `sequenceSize` populataiton in the `XPathMessageSplitter`
* Document `Stream` & `Flux` support in the splitter
Minor doc polishing
JIRA: https://jira.spring.io/browse/INT-4060
* Add `NLST` command to the `AbstractRemoteFileOutboundGateway` to perform
`listNames` on the target session.
Useful in case of server doesn't allow to perform `LS` or the names set is
sufficient for application requirements
* Add `workingDirExpression` to the `FtpOutboundGateway` to allow to perform
`FtpClient.changeWorkingDirectory()` based on the current request message
* Change `slf4j-log4j12` to the `testCompile` -
the FTP tests fail in the IDE with `ClassNotFoundException`
Address PR comments
* Add `nlst` to XSD config
* Reinstate `ls -1` test-case for the `FtpServerOutboundTests`
* wrap more commands to the `doInWorkingDirectory()`
Fix `MV` command in the `AbstractRemoteFileOutboundGateway`
* Implement `RemoteFileOperations#invoke(OperationsCallback<F, T>)`
for thread-bound `session`s
* Use a new `invoke()` for `put()` and `mPut()` commands in the `AbstractRemoteFileOutboundGateway`
* Add delegation for the `put()` and `mPut()` commands in the `FtpOutboundGateway`
* Add DSL support for the `workingDirExpression` and add `whats-new.adoc` note
Document changes
Address some PR comments:
* Add `invokeScope` variable to `execute` to track `ThreadLocal` session or not
* Check for `null` in the `getSession()` and fallback to regular
`sessionFactory.getSession()`.
Most likely the `invoke()` is called from other thread
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4252
To allow to provide arbitrary bean names for the intermediate components
in the `IntegrationFlow` change `ComponentsRegistration` to return
a `Map<Object, String>` instead of raw `Collection<Object>`
* Use the value from that map in the `IntegrationFlowBeanPostProcessor` as a
fallback option before walking into bean name generation
* Provide `containerId` option for the AMQP DSL components
* Revert `@AfterClass` in the `AmqpTests`
* Expose `id()` for the `JmsDestinationAccessorSpec`
* Register `ListenerContainer` and `JmsTemplate`
as bean via `ComponentRegistration` in the particular JMS Java DSL components
* Fix `Jms` factory to populate the `JmsDefaultListenerContainerSpec`
for the `messageDrivenChannelAdapter()` without class specified
* Refactor AMQP DSL Inbound components to deal with the new ContainerSpec API
* Remove `containerId()` in favor of `id()` in the ContainerSpec
JIRA: https://jira.spring.io/browse/INT-4117
Add an option to set file permissions on the `FileWritingMessageHandler` when the file system supports it.
Polishing - PR Comments; more robust file system detection.
Add DSL support and fix NPE with missing timestamp header.
Fix javadoc
Fix javadoc
Some polishing:
* Add `chmod >= 0` assertion
* Make some assertion with string concatenation as `Supplier`-based to
deffer that string concatenation
https://build.spring.io/browse/INT-MASTER-675/
Since SF is still based on the `Bismuth-M1`,
we can't afford ourselves to go to `BUILD-SNAPSHOT` yet
Also fix `FileTests` do not poll the same file again for `splitter` case
JIRA: https://jira.spring.io/browse/INT-4260
MPEH: Populate the `ErrorMessage.originalMessage`,
if available and not the same as the `failedMessage`.
Polishing - rename exception to `MessagingExceptionWrapper`;
make `MPEH` a subclass of `ErrorMessagePublisher`
PR Comments
Polishing - PR Comments
Using a `LiteralExpression` in the `AbstractRemoteFileOutboundGateway` (when the
user provides a `File` for the directory) causes us to create a new `File` object
each time, instead of using the user-supplied `File`.
Use a `ValueExpression` instead.
Move the `FileWritingMessageHandler` logic to `ExpressionUtils` and use it from both places.
Also add a String setter variant for convenient Java Configuration.
JIRA: https://jira.spring.io/browse/INT-4086
The remote files may be carried with sub-path, not just file name.
The same sub-dirs logic can be achieve with the `localFilenameGeneratorExpression`
* Add the logic into `AbstractInboundFileSynchronizer` to build sub-dirs for local files
* Switch on `WatchService` for the internal `FileReadingMessageSource` in the `AbstractInboundFileSynchronizingMessageSource` to react for all the changes in the `localDirectory`
Introduce `RecursiveDirectoryScanner` and use it for `AbstractInboundFileSynchronizingMessageSource` by default
* Add `maxDepth` and `fileVisitOptions` options to the `RecursiveDirectoryScanner`
* Polishing JavaDocs, tests and docs
JIRA: https://jira.spring.io/browse/INT-4244
The `JpaOutboundGateway` returns a whole `Message` for reply copying request headers.
Building the reply a `AbstractMessageProducingHandler` uses `MessageBuilder` to create a new `Message` with all request headers because of `shouldCopyRequestHeaders() == true`
* To avoid some overhead with new `Message` and `MessageBuilder` for nothing, just return a JPA result as is and let super class to build the Message
* Optimize all reply producers to return `MessageBuilder` to the super `handleMessage()` process
* Return the whole `gatherResult` `Message<?>` from the `ScatterGatherHandler` to have ability to transfer reply headers as well
* Mark `DelayHandler` as `shouldCopyRequestHeaders() return false` since it doesn't modify Message and returns it as is after delay
