JIRA: https://jira.spring.io/browse/INT-4351
The `WatchService` reacts to the events in the file system and keep
track ove the events until we poll them, e.g. via `listEligibleFiles()`
in the `WatchServiceDirectoryScanner`.
On the other hand the `SourcePollingChannelAdapter` calls the mentioned
`listEligibleFiles()` according its polling period.
At this moment the `FileListFilter` is applied to the polled files.
It may happen that `LastModifiedFileListFilter` can't accept too young
files yet and they are lost for the future consideration.
* To allow, for example, to retain young files by the
`LastModifiedFileListFilter` judgment for the future cycles add
`DiscardAwareFileListFilter` with the `DiscardCallback` support.
The `WatchServiceDirectoryScanner` now registers such a callback into
the filter and stores discarded files into the `filesToPoll` queue
for the future poll cycle.
**Cherry-pick to 5.0**
Fix compilation warnings
* Replace `DiscardCallback` with the plain `Consumer`
* Ensure uniqueness in the `WatchServiceDirectoryScanner` internal
queue via a `Set` implementation, since discard callback may be called
several times for the same file from the `CompositeFileListFilter`
according to its nature
* Add JavaDocs and Docs
Change `@since` to `5.0.5`
JIRA: https://jira.spring.io/browse/INT-4390
Phase I: flow context/registration cycle
- extract interfaces
- rename implementations to `Standard...`
Phase II - dsl<->dsl.channel tangles
Move the channel specs to dsl.
Fix missing refactorings
(Not related to DSL) - rename core `event` package to `events`
- avoid collision with event module package
- fix tangle caused by `MessageGroupExpiredEvent`.
Phase III - dsl<->config.dsl
- move classes from config.dsl to dsl
(Not related to DSL) - fix many tangles caused by graph being a sub-package of support.management
- make `graph` a top-level package
(Not related to DSL) - move `IntegrationManagementConfigurer` from
`management` to `config` - tangle between core and management
* Polishing Copyrights, diamonds, some JavaDocs and What's New
JIRA: https://jira.spring.io/browse/INT-4430
When Iterator-based `FileSplitter` splits the file, and an exception
throws in downstream flow (in the same thread), the exception propagates
to the caller leaving underlying file reader opened.
This commit changes `AbstractMessageSplitter` the way, that,
when any exception happens, if Iterator implements `java.io.Closeable`,
its `close()` method will be called before propagating exception.
Also `FileSplitter`'s underlying iterator implements `Closeable` now.
* Make `CloseableIterator` to follow `Closeable` contract.
Now `CloseableIterator.close()` declares `IOException` and can be used
as base interface for `FunctionIterator`.
* Adjust tests.
Adjust tests to reflect the fact that we call `close()` on the reader
one more time in the end of iterator.
**Cherry-pick to 5.0.x and 4.3.x**
JIRA: https://jira.spring.io/browse/INT-4392
JIRA: https://jira.spring.io/browse/INT-4391
JIRA: https://jira.spring.io/browse/INT-4393
Resolve class and leaf package tangles in:
- core (except DSL)
- file
- http
- move `StaticMessageHeaderAccessor` to root `integration` package
- move acknowledgment related classes to a new `acks` package
- move `MMIH` to `handler.support`, alongside argument resolvers, also `HandlerMethodArgumentResolversHolder`
- move `ErrorMessagePublisher` to `core`
- move `PatternMatchUtils` to `support.utils`
- move `MessageSourceManagement` to `support.management`
- remove direct reference to `MicrometerMetricsCapter` from `IntegrationManagementSupport`
- add new class `MetricsCaptorLoader`
- move `OperationsCallback` to inner interface in `RemoteFileOperations`
- move `HttpContextUtils` to `config`
* Simplify MetricsCaptor loading.
* Polishing
JIRA: https://jira.spring.io/browse/INT-4382
There is a deadlock in the FMWH due to lock ordering.
Writes try to lock the handler monitor while holding the file lock.
The `Flusher` tries to lock the file lock while holding th handler monitor.
- Change the `Flusher`, `stop` and `flushIfNeeded` methods to not call close
on the file state while holding the handler monitor.
__cherry pick to 4.3.x__ (fix <> for Java 6)
Polishing - PR Comments
* Optimize `flushIfNeeded(FlushPredicate)` to delegate only to the
`flushIfNeeded(MessageFlushPredicate, Message<?>)`
To avoid `Message` re-creation during `AbstractMessageSource.receive()`
logic, refactor `AbstractMessageSource` implementations
to return `AbstractIntegrationMessageBuilder`
* Add `AbstractIntegrationMessageBuilder<File> doReceive()` to the
`FileReadingMessageSource` to be called from the `AbstractInboundFileSynchronizingMessageSource`
to avoid message recreation in its `doReceive()`
* Some code style refactoring in the `AbstractMessageSource`
* INT-4376: Upgage to Log4J 2
JIRA: https://jira.spring.io/browse/INT-4376
* Deprecate Log4J 1.x components in favor of newly added
a `Log4j2LevelAdjuster` JUnit `@Rule`
* Update all the logging configuration to Log4J 2
* Polishing after testing
JIRA: https://jira.spring.io/browse/INT-4344
The `FtpOutboundGateway` provides ctors without expression for remote path.
In this case it is treated as a `working directory` but only for the
`LS`, `NLST`, `PUT` and `MPUT` commands.
* Add assertion in the `AbstractRemoteFileOutboundGateway` to discard
configuration for all other commands when `expression` is `null`
**Cherry-pick to 4.3.x**
Fixesspring-projects/spring-integration#2253
For backward compatibility and use-cases when processed files are stored
in sub-directories in the `localDirectory` revert the
`AbstractInboundFileSynchronizingMessageSource` logic back to the
`DefaultDirectoryScanner`.
The `RecursiveDirectoryScanner` can be injected via `scanner` option
JIRA: https://jira.spring.io/browse/INT-4356
**Cherry-pick to 4.3.x**
Address PR comments:
* Remove unused variable in the test case
* Add string-based SpEL setters to the `AbstractInboundFileSynchronizer`
* Fix `@since` in the `FileTransferringMessageHandler` to the proper version
JIRA: https://jira.springsource.org/browse/INT-4334
* Add a `DirectoryScanner` setter named `setScanner` in
`AbstractInboundFileSynchronizingMessageSource` to set the
`FileReadingMessageSource`'s scanner.
* Add tests to `AbstractRemoteFileSynchronizerTests`.
Polish.
Updates xsd, tests and documentation.
Fix xsd formatting.
Addresses review comments.
* Some code style polishing
* Add `validateOnlyOverlapping = false` for `RequireThis` Checkstyle rule
* Add `this.` to caught places
* Use `CustomizableThreadFactory` instead of raw interface in the
`UnicastSendingMessageHandler`
JIRA: https://jira.spring.io/browse/INT-4324
The `preserve-timestamp` attribute for the
`<int-file:outbound-channel-adapter>` has been missed for parsing in
the `FileWritingMessageHandlerBeanDefinitionBuilder` and propagation
in the `FileWritingMessageHandlerFactoryBean`
**Cherry-pick to 4.3.x**
Fixspring-projects/spring-integration-java-dsl#152
* Add Logic to the `FileInboundChannelAdapterSpec` do not populate
`FileListFilterFactoryBean` to the `FileReadingMessageSource`
when the `scanner` is provided.
Otherwise call the `FileReadingMessageSource` and rely on its logic to
refuse mutually exclusive `scanner` and `filter/locker` options
JIRA: https://jira.spring.io/browse/INT-4308
* Document `JdbcMetadataStore`
* Remove unnecessary context DERBY config for the `PersistentAcceptOnceFileListFilterExternalStoreTests`
* Rework the `testFileSystemWithJdbcMetadataStore()` to use `EmbeddedDatabase` directly, without context
JIRA: https://jira.spring.io/browse/INT-4306
Fixed checkstyle errors
Fixed checkstyle errors
Fixed checkstyle errors and sql schema
Manually fixed derby sql schema
Fixing some issues
added region support
small refactorings
test checkstyle fix
integration tests and fixes
integration tests and fixes
JIRA: https://jira.spring.io/browse/INT-4305
The `File.renameTo()` operation may fail, therefore the content of the
local file isn't changed, but since we change `setLastModified()` anyway,
this file might be eligible for local polling.
So, we end up with the same content from local file in a new message,
meanwhile we expect a new content from the remote file
* Check the `File.renameTo()` result and attempt to `delete()`
for existing local file
* When file isn't renames remove the remote file from the `filter` to let
it be transferred one more time on the next poll.
The local file might be opened for processing, so this way we postpone a fresh
remote file for the future poll rounds
* Modify `copyFileToLocalDirectory()` to return `boolean` to reflect the fact of copy.
This way we check the real number of transferred files
**Cherry-pick to 4.3.x**
Do not transfer remote file if we can't remove local one
Polishing Log Messages
JIRA: https://jira.spring.io/browse/INT-4297
* Add `FileSplitter.setFirstLineAsHeader()` and extract the first line from
the context before `iterator`.
* Populate the extracted header to each subsequent line
* Don't use `Streams` to read first line
* Add XSD and Java DSL option for the `firstLineAsHeader`
* Document the feature
* Add Java DSL sample for the `Files.slitter()`
Add a note about more complex header enrichment
Doc Polishing and fix field name.
JIRA: https://jira.spring.io/browse/INT-4303
Don't emit ignored files in the output message payload.
Also, always preserve the timestamp when using `FileExistsMode.REPLACE_IF_MODIFIED`.
Some code style polishing
JIRA: https://jira.spring.io/browse/INT-4271
The `Flusher.run()` is `synchronized` on the `FWMH`; a `stop()` while it's running will interrupt the task
which could cause a state to be removed without a flush actually happening (interrupt
on lock acquisition).
- synchronize `stop()` so it won't interrupt a running flusher
- detect an interrupt in the flusher and stop flushing
**Cherry-pick to 4.3.x**
JIRA: https://jira.spring.io/browse/INT-4296
The `FileState.close()` method correctly acquires the lock to prevent closing while
writing. However, with a short flush interval, the close could occur between state
creation and the write operation acquiring the lock.
Move the state creation to within the scope of the lock acquired by the writer.
JIRA: https://jira.spring.io/browse/INT-4283
Filter that only passes files where a corresponding "marker" file is also present.
Implementations for file and(S)FTP.
* Polishing - PR Comments
* Missed one
JIRA: https://jira.spring.io/browse/INT-4269
* Check if `outputChannel` is `ReactiveStreamsSubscribableChannel`
to let back-pressure splitting
* Build `Flux` or `Iterator` depending in the `reactive` state
* Allow to get a size of the `iterator` if it is possible,
for example `XPathMessageSplitter`
Add tests
Fix raw type and unused import
* Add JavaDocs to the `AbstractMessageSplitter#obtainSizeIfPossible()`
* Add asserts for the `sequenceSize` populataiton in the `XPathMessageSplitter`
* Document `Stream` & `Flux` support in the splitter
Minor doc polishing
JIRA: https://jira.spring.io/browse/INT-4060
* Add `NLST` command to the `AbstractRemoteFileOutboundGateway` to perform
`listNames` on the target session.
Useful in case of server doesn't allow to perform `LS` or the names set is
sufficient for application requirements
* Add `workingDirExpression` to the `FtpOutboundGateway` to allow to perform
`FtpClient.changeWorkingDirectory()` based on the current request message
* Change `slf4j-log4j12` to the `testCompile` -
the FTP tests fail in the IDE with `ClassNotFoundException`
Address PR comments
* Add `nlst` to XSD config
* Reinstate `ls -1` test-case for the `FtpServerOutboundTests`
* wrap more commands to the `doInWorkingDirectory()`
Fix `MV` command in the `AbstractRemoteFileOutboundGateway`
* Implement `RemoteFileOperations#invoke(OperationsCallback<F, T>)`
for thread-bound `session`s
* Use a new `invoke()` for `put()` and `mPut()` commands in the `AbstractRemoteFileOutboundGateway`
* Add delegation for the `put()` and `mPut()` commands in the `FtpOutboundGateway`
* Add DSL support for the `workingDirExpression` and add `whats-new.adoc` note
Document changes
Address some PR comments:
* Add `invokeScope` variable to `execute` to track `ThreadLocal` session or not
* Check for `null` in the `getSession()` and fallback to regular
`sessionFactory.getSession()`.
Most likely the `invoke()` is called from other thread
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4252
To allow to provide arbitrary bean names for the intermediate components
in the `IntegrationFlow` change `ComponentsRegistration` to return
a `Map<Object, String>` instead of raw `Collection<Object>`
* Use the value from that map in the `IntegrationFlowBeanPostProcessor` as a
fallback option before walking into bean name generation
* Provide `containerId` option for the AMQP DSL components
* Revert `@AfterClass` in the `AmqpTests`
* Expose `id()` for the `JmsDestinationAccessorSpec`
* Register `ListenerContainer` and `JmsTemplate`
as bean via `ComponentRegistration` in the particular JMS Java DSL components
* Fix `Jms` factory to populate the `JmsDefaultListenerContainerSpec`
for the `messageDrivenChannelAdapter()` without class specified
* Refactor AMQP DSL Inbound components to deal with the new ContainerSpec API
* Remove `containerId()` in favor of `id()` in the ContainerSpec
JIRA: https://jira.spring.io/browse/INT-4117
Add an option to set file permissions on the `FileWritingMessageHandler` when the file system supports it.
Polishing - PR Comments; more robust file system detection.
Add DSL support and fix NPE with missing timestamp header.
Fix javadoc
Fix javadoc
Some polishing:
* Add `chmod >= 0` assertion
* Make some assertion with string concatenation as `Supplier`-based to
deffer that string concatenation
https://build.spring.io/browse/INT-MASTER-675/
Since SF is still based on the `Bismuth-M1`,
we can't afford ourselves to go to `BUILD-SNAPSHOT` yet
Also fix `FileTests` do not poll the same file again for `splitter` case
JIRA: https://jira.spring.io/browse/INT-4260
MPEH: Populate the `ErrorMessage.originalMessage`,
if available and not the same as the `failedMessage`.
Polishing - rename exception to `MessagingExceptionWrapper`;
make `MPEH` a subclass of `ErrorMessagePublisher`
PR Comments
Polishing - PR Comments
Using a `LiteralExpression` in the `AbstractRemoteFileOutboundGateway` (when the
user provides a `File` for the directory) causes us to create a new `File` object
each time, instead of using the user-supplied `File`.
Use a `ValueExpression` instead.
Move the `FileWritingMessageHandler` logic to `ExpressionUtils` and use it from both places.
Also add a String setter variant for convenient Java Configuration.
JIRA: https://jira.spring.io/browse/INT-4086
The remote files may be carried with sub-path, not just file name.
The same sub-dirs logic can be achieve with the `localFilenameGeneratorExpression`
* Add the logic into `AbstractInboundFileSynchronizer` to build sub-dirs for local files
* Switch on `WatchService` for the internal `FileReadingMessageSource` in the `AbstractInboundFileSynchronizingMessageSource` to react for all the changes in the `localDirectory`
Introduce `RecursiveDirectoryScanner` and use it for `AbstractInboundFileSynchronizingMessageSource` by default
* Add `maxDepth` and `fileVisitOptions` options to the `RecursiveDirectoryScanner`
* Polishing JavaDocs, tests and docs
JIRA: https://jira.spring.io/browse/INT-4244
The `JpaOutboundGateway` returns a whole `Message` for reply copying request headers.
Building the reply a `AbstractMessageProducingHandler` uses `MessageBuilder` to create a new `Message` with all request headers because of `shouldCopyRequestHeaders() == true`
* To avoid some overhead with new `Message` and `MessageBuilder` for nothing, just return a JPA result as is and let super class to build the Message
* Optimize all reply producers to return `MessageBuilder` to the super `handleMessage()` process
* Return the whole `gatherResult` `Message<?>` from the `ScatterGatherHandler` to have ability to transfer reply headers as well
* Mark `DelayHandler` as `shouldCopyRequestHeaders() return false` since it doesn't modify Message and returns it as is after delay
JIRA: https://jira.spring.io/browse/INT-4115
Apply `(S)FtpPersistentAcceptOnceFileListFilter` for the `(S)FtpInboundFileSynchronizer` by default to avoid cases to sync the same remote files to the local directory again.
Especially when `localFileName` strategy is applied and we end up with new local files, but with the same remote content
Accept `(S)FtpPersistentAcceptOnceFileListFilter` for streaming adapters
Make `doSetFilter()` as `protected final`
Fix tests after rebase
Fix "What's New" after rebase
Compose `PersistentAcceptOnceFileListFilter` together with the regex or pattern filters
Document such a behavior
Address PR comments for formatting and typo
JIRA: https://jira.spring.io/browse/INT-4018
* Resolve some TODOs as expected
* Remove some as logically or physically with useless effort
* Leave others for future considerations
JIRA: https://jira.spring.io/browse/INT-4243
Remove newline char from key exchange.
Fix CR and Add Windows Delete Diagnostics
Fix For Windows
More diagnostics.
It appears the new server doesn't close the file when the session is closed.
In the streaming test, consume the stream before closing the session.
Polishing - PR Comments
* More polishing according PR comments
JIRA: https://jira.spring.io/browse/INT-4237
It was possible to flush (close) the file while a write was in process; more likely when
`flushWhenIdle` is false.
This probably would not occur in the real world, just tests with short flush intervals,
but certainly possible.
There is already a lock used to prevent concurrent writes while appending; use the same
lock when flushing.
JIRA: https://jira.spring.io/browse/INT-4232
Currently to implement the synchronization logic when remote file is modified there is only one way: remove local file and provide a `localFilter` which can be clear from the removed file as well.
The Reference Manual claims that `FileSystemPersistentAcceptOnceFileListFilter` can let us pick up a fresh version of the remote file if we configure `preserveTimestamp = true`, but actually `AbstractInboundFileSynchronizer` just bypass that logic because it check a local file for existence.
* Modify `AbstractInboundFileSynchronizer` to compare `lastModified` of the remote file with local version if `preserveTimestamp == true`
* Also replace `AcceptOnceFileListFilter` to the `FileSystemPersistentAcceptOnceFileListFilter` since the first one doesn't care about `lastModified`.
If `preserveTimestamp` isn't in use the `FileSystemPersistentAcceptOnceFileListFilter` works fully similar to the `AcceptOnceFileListFilter`
**Cherry-pick to 4.3.x except test-case**
Make `SftpStreamingMessageSourceTests` compatible with Windows
Looks like SFTP embedded server provides different permissions for files on different OSs
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4233
INT-4233: Add (S)FTP FileInfo Header (Streaming)
JIRA: https://jira.spring.io/browse/INT-4233
Add the complete file info as JSON (when Jackson or Boon available) to the message headers
when streaming inbound.
Provide a mechanism to configure Boon to provide similar output to Jackson.
Also allow subclasses to provide their own object mapper.
Also clean up after the AMQP DSL tests, and don't use a queue `foo`. I often have such a queue
with content; this caused tests to fail.
Use SimpleJsonSerializer
Doc Polishing
Polishing - PR Comments
More Polishing
* Polishing according latest PR comments
