JIRA: https://jira.spring.io/browse/INT-4243
Remove newline char from key exchange.
Fix CR and Add Windows Delete Diagnostics
Fix For Windows
More diagnostics.
It appears the new server doesn't close the file when the session is closed.
In the streaming test, consume the stream before closing the session.
Polishing - PR Comments
* More polishing according PR comments
Conflicts:
spring-integration-sftp/src/test/java/org/springframework/integration/sftp/SftpTestSupport.java
spring-integration-sftp/src/test/java/org/springframework/integration/sftp/dsl/SftpTests.java
spring-integration-sftp/src/test/java/org/springframework/integration/sftp/session/SftpRemoteFileTemplateTests.java
spring-integration-sftp/src/test/java/org/springframework/integration/sftp/session/SftpServerTests.java
spring-integration-sftp/src/test/java/org/springframework/integration/sftp/session/SftpSessionFactoryTests.java
JIRA: https://jira.spring.io/browse/INT-4237
It was possible to flush (close) the file while a write was in process; more likely when
`flushWhenIdle` is false.
This probably would not occur in the real world, just tests with short flush intervals,
but certainly possible.
There is already a lock used to prevent concurrent writes while appending; use the same
lock when flushing.
Conflicts:
spring-integration-file/src/main/java/org/springframework/integration/file/FileWritingMessageHandler.java
spring-integration-file/src/test/java/org/springframework/integration/file/FileWritingMessageHandlerTests.java
JIRA: https://jira.spring.io/browse/INT-4232
Currently to implement the synchronization logic when remote file is modified there is only one way: remove local file and provide a `localFilter` which can be clear from the removed file as well.
The Reference Manual claims that `FileSystemPersistentAcceptOnceFileListFilter` can let us pick up a fresh version of the remote file if we configure `preserveTimestamp = true`, but actually `AbstractInboundFileSynchronizer` just bypass that logic because it check a local file for existence.
* Modify `AbstractInboundFileSynchronizer` to compare `lastModified` of the remote file with local version if `preserveTimestamp == true`
* Also replace `AcceptOnceFileListFilter` to the `FileSystemPersistentAcceptOnceFileListFilter` since the first one doesn't care about `lastModified`.
If `preserveTimestamp` isn't in use the `FileSystemPersistentAcceptOnceFileListFilter` works fully similar to the `AcceptOnceFileListFilter`
**Cherry-pick to 4.3.x except test-case**
Make `SftpStreamingMessageSourceTests` compatible with Windows
Looks like SFTP embedded server provides different permissions for files on different OSs
Doc Polishing
Conflicts:
spring-integration-file/src/test/java/org/springframework/integration/file/remote/synchronizer/AbstractRemoteFileSynchronizerTests.java
spring-integration-ftp/src/test/java/org/springframework/integration/ftp/dsl/FtpTests.java
spring-integration-sftp/src/test/java/org/springframework/integration/sftp/inbound/SftpStreamingMessageSourceTests.java
src/reference/asciidoc/whats-new.adoc
Resolved.
JIRA: https://jira.spring.io/browse/INT-4230
The `get()` method only preserves the timestamp when an LS operation is done within it.
For MGET ops, we don't perform another LS, so the timestamp was not updated.
Add code to the recursive and non-recursive MGET methods to preserve the timestamp if so configured.
Conflicts:
spring-integration-file/src/main/java/org/springframework/integration/file/remote/gateway/AbstractRemoteFileOutboundGateway.java
spring-integration-ftp/src/test/java/org/springframework/integration/ftp/outbound/FtpServerOutboundTests-context.xml
spring-integration-sftp/src/test/java/org/springframework/integration/sftp/outbound/SftpServerOutboundTests-context.xml
JIRA: https://jira.spring.io/browse/INT-4221
* Upgrade to those versions of Spring project dependencies which potentially will provide similar fix
**Cherry-pick to 4.3.x**
JIRA: https://jira.spring.io/browse/INT-4216
Add test for running empty array through `ChainFileListFilter`
Assure no exceptions are throw when filter input is empty.
INT-4216: Add fix for ChainFileListFilter edge case
INT-4216: Fix checkstyle error on test class
Add copyright header to test file
* Polishing for `ChainFileListFilter` JavaDocs
* Move `CompositeFileListFilterTests` and `ChainFileListFilterIntegrationTests` to the `filters` package instead of `file` core
(cherry picked from commit 1f9d07a)
JIRA: https://jira.spring.io/browse/INT-4212
Add an option to flush after the `flushInterval`, regardless of intermediate writes.
Rename `lastFlush` to `firstWrite`
Polishing
**Cherry-pick to 4.3.x**
JIRA: https://jira.spring.io/browse/INT-4203
Add channel names for easier use in DSL.
Add documentation (boot) example.
Normalize Expression Setters; add Javadocs
Polishing - SPR-15091
Tiny code style polishing
Conflicts:
spring-integration-core/src/test/java/org/springframework/integration/dsl/flows/IntegrationFlowTests.java
src/reference/asciidoc/handler-advice.adoc
spring-integration-core/src/test/java/org/springframework/integration/handler/advice/ExpressionEvaluatingRequestHandlerAdviceTests.java
* Restore removed methods in the `ExpressionEvaluatingRequestHandlerAdvice` and deprecate them
* Remove methods in the `ExpressionEvaluatingRequestHandlerAdvice` introduced since `5.0`
JIRA: https://jira.spring.io/browse/INT-4187
Add javadoc and test
- assertTrue on default adapter
- assertFalse check on nativeAdapter
- update XSD to add enable-status-reader
add `@auther`
add example in reference document
update adoc
* Fix minor typos in docs and allow to configure `enable-status-reader` with property placeholder
**Cherry-pick to 4.3.x**
(cherry picked from commit 16c7535)
JIRA: https://jira.spring.io/browse/INT-4185
Even if `WatchEventType` doesn't expose Java 7 API, its instantiation leads to the `StandardWatchEventKinds` object which already isn't Java 6
* Move `this.watchEvents = new WatchEventType[] { WatchEventType.CREATE }` to the ctor and guard it with the `ClassUtils.isPresent("java.nio.file.WatchService")`
Assertions for `WatchService`-based setters
JIRA: https://jira.spring.io/browse/INT-4147
By default `FileReadingMessageSource` is created without `filter`, at the same time internal `WatchServiceDirectoryScanner` is supplied with default `filter` by its `DefaultDirectoryScanner` super class.
A `DELETE` watch event condition around `FileReadingMessageSource.this.filter` is wrong, because it is `null` by default. Therefore `DELETE` events never succeed
* Modify `FileInboundTransactionTests` to accept `DELETE` watch event as well and verify that `ResettableFileListFilter.remove(File)` is performed
* Call newly created getter for `DefaultDirectoryScanner.filter` in the `WatchServiceDirectoryScanner` to assert against supplied `filter`
Polishing
**Cherry-pick to 4.3.x**
Conflicts:
spring-integration-file/src/main/java/org/springframework/integration/file/DefaultDirectoryScanner.java
Resolved.
JIRA: https://jira.spring.io/browse/INT-4146
Fixes GH-1942 (https://github.com/spring-projects/spring-integration/issues/1942)
Previously the `FileWritingMessageHandler` didn't wrap `new OutputStream()` to the `try...catch...finally` block, therefore any errors on that (e.g. `FileNotFoundException` because of permissions) cause an early exist without `inputStream.close()`
* Move a `new OutputStream()` to the `try...catch`
* Change `destinationDirectory.canWrite()` to the `Files.isWritable(destinationDirectory.toPath())` because the first one doesn't work on Windows
**Cherry-pick to 4.3.x and 4.2.x but without Files.isWritable() change**
JIRA: https://jira.spring.io/browse/INT-4096
* Introduce `spring.integration.readOnly.headers` Integration property
* Introduce `IntegrationMessageHeaderAccessor.setReadOnlyHeaders()` and use it from the overridden `IntegrationMessageHeaderAccessor.isReadOnly()`
* Add `DefaultMessageBuilderFactory.setReadOnlyHeaders()` and delegate the value to the new `MessageBuilder.readOnlyHeaders()`
* Modify `spring.integration.properties` for the `contentType` as a `readOnly` header for testing
* Ensure that provided logic works via an appropriate modification to the `ObjectToJsonTransformerParserTests`
Increase receive timeouts in the `OutboundGatewayFunctionTests`
Fix `FileInboundTransactionTests` for slow `WatchService` issue on Linux/OS X
Also redo the `tmp` dir logic to the `TemporaryFolder` `@ClassRule`
Fix unused `import` in the `FileInboundTransactionTests`
Polishing
JIRA: https://jira.spring.io/browse/INT-4108
Some `Lifecycle.start()/stop()` usage doesn't ensure robustness for components causing unexpected and difficulty tracing issues
* Fix `Lifecycle.start()/stop()` for `FileReadingMessageSource`, `FileWritingMessageHandler`, `AbstractMqttMessageHandler`
* In the `DefaultHeaderChannelRegistry`, `LockRegistryLeaderInitiator`, `MqttPahoMessageHandler` rework logic for shared variables to avoid `NPE`
* Increase receive timeouts in the `PayloadDeserializingTransformerParserTests` and `UdpChannelAdapterTests`
* Prove with the `WatchServiceDirectoryScannerTests` changes that several invocation for `FileReadingMessageSource.start()` are idempotent
**Cherry-pick to 4.3.x**
* Fix several typos in log messages. And some test on the matter as well
* Add comment to `AbstractPersistentAcceptOnceFileListFilter.rollback()` to clarify the reason of `rollingBack` variable
* Make `RemoteFileTemplate.StreamHolder` as `static` to avoid extra internal variable to outer class instance
* Replace `MessagingException` with `AbstractInboundFileSynchronizingMessageSource` in `init()` method of some components. It isn't Messaging yet in that phase
* Fix `SubscribableRedisChannel.MessageListenerDelegate` to handle `Object` not `String`, because with the `serializer` injection there is no guaranty that incoming is always `String`
* Move `JSch.setLogger(new JschLogger());` in the `DefaultSftpSessionFactory` to `static` block. It really should be done only once
* Remove `Assert.isTrue(this.port >= 0)` from the `DefaultSftpSessionFactory`. The subsequant `initJschSession()` convert it to default `22` port
* Change in the `JschProxyFactoryBean` `UnsupportedOperationException` to `IllegalArgumentException`. Wrong enum is wrong argument. That isn't a problem of operation
* Simplify `stop()` in the `CuratorFrameworkFactoryBean` and mark it as a `this.running = false`. Otherwise it wasn't able to be restarted
* Expose `leaderEventPublisher` in the `LeaderInitiatorFactoryBean` and fix `stop(Runnable callback)` with propagation `callback` to delegate.
Fix `SubscribableRedisChannelTests` for new `handleMessage(Object)` signature
JIRA: https://jira.spring.io/browse/INT-4067
When `FileSplitter` is configured with `markers = true` and file is empty, an `iterator` for file throws `IOException: Stream closed`,
because we close the `buffer` just after the first `readLine()` attempt, but still return `true` from the first `hasNext()` call
where the `this.sof` and `this.eof` are `true` for markers.
Add logic to mark internal splitter `iterator` as `done` where we don't have content and still in `sof` state.
JIRA: https://jira.spring.io/browse/INT-4053
The recent changes to the `FileSplitter` introduced (by IDE autocompletion mistake) hidden dependency of Reactor using its `Assert` class instead of Spring's one
**Cherry-pick to 4.2.x**
JIRA: https://jira.spring.io/browse/INT-4046
Since not all FTP servers provide proper `STAT` command implementation,
plus the `NLIST` doesn't work properly for directories cases, introduce the `FtpRemoteFileTemplate.ExistsMode`
to let:
* to perform `STAT` by default (previous) behavior;
* to switch to `NLIST` for `FtpRemoteFileTemplate` internal use;
* perform the full `NLIST` and `FTPClient.changeWorkingDirectory()` algorithm if needed.
* Improve (S)Ftp components to use proper `RemoteFileTemplate` for internal instantiation
* Introduce `FtpMessageHandler` to wrap `FtpRemoteFileTemplate` with the proper `NLIST` `ExistsMode`
* Cover `NLIST` switching from the `FtpOutboundChannelAdapterParser` and `FtpOutboundGatewayParser`
* Document the `FtpRemoteFileTemplate.ExistsMode`
* Fix typo in the recently introduced `RemoteFileOperations.getSession()` method name
* Add JavaDoc to `Session.exists()`
* Add `NLIST` support for the `FtpSession.exists()` to meet the API requirements
**Cherry-pick to 4.2.x and 4.1.x**
Addressing PR comments
Doc Polishing
* To avoid a compiling noise like:
```
warning: No processor claimed any of these annotations: org.springframework.integration.support.management.IntegrationManagedResource,org.springframework.jmx.export.annotation.ManagedResource
```
add `-processing` option to the `compileTestJava.options.compilerArgs`
* Remove such an option from the `JPA` module as redundant
* Fix typos in the `spring-integration-file-4.3.xsd`
JIRA: https://jira.spring.io/browse/INT-4015https://jira.spring.io/browse/INT-3854
Initial commit.
Reworked to emit an input stream and use the file splitter.
Add StreamTransformer.
Add CLOSABLE_RESOURCE header so we can close the session automatically.
Implement INT-3854, FTP, SFTP
(S)FTP Namespace Changes
Docs - also fixes a PDF overflow
Polishing - PR Comments
checkstyle fixes
Polishing - Add Namespace for StreamParser
Polishing - PR Comments
* Upgrade to SA and SF snapshots
* Address TODOs regarding those upgrades
* Resolve some other TODOs
* Replace `PropertyPlaceholderConfigurer` beans with the `<context:property-placeholder>`
* Upgrade to Spring Social Twitter 1.1.2 and resolve deprecations via mocks (https://github.com/spring-projects/spring-social-twitter/issues/91)
* Upgrade to Curator `3.1.0` and resolve deprecation in the `ZookeeperMetadataStore`
JIRA: https://jira.spring.io/browse/INT-3989,
https://jira.spring.io/browse/INT-3990,
https://jira.spring.io/browse/INT-3988
INT-3989: Add `FileReadingMessageSource.WatchServiceDirectoryScanner`
* Deprecate top-level `WatchServiceDirectoryScanner` because of inconsistency around `Lifecycle` and shared `directory` property
* Copy/paste its logic into the `FileReadingMessageSource.WatchServiceDirectoryScanner` to hide that inconsistency, but still get a gain from the `WatchService` benefits
* Add support for the `StandardWatchEventKinds.ENTRY_MODIFY` and `StandardWatchEventKinds.ENTRY_DELETE` events in the `FileReadingMessageSource.WatchServiceDirectoryScanner`
* Introduce `useWatchService` option to switch to the internal `FileReadingMessageSource.WatchServiceDirectoryScanner`
* Make `CompositeFileListFilter` also as `ResettableFileListFilter`
* Deprecate weird `FileReadingMessageSource.onSend()` method and remove its usage from tests
* Modify `WatchServiceDirectoryScannerTests` for the new logic
* Document changes
Add `MODIFY` and `DELETE` test coverage
Optimize the `filesFromEvents()` logic replacing item with the fresh event sources.
Remove the item from the result set in case of `DELETE` event, because file removal generates both `MODIFY` and `DELETE` events.
* Add `FileReadingMessageSource.setWatchEvents` to allow to listen to the specific events,
not only `CREATE` or all of them.
* Modify tests and docs to reflect the new API
Address PR comments
* Improve `FileReadingMessageSource.setWatchEvents()`
* Add `file.exists()` before adding file from event.
Add more DEBUG logs into the `WatchServiceDirectoryScanner`
With the fact of those logs provide more optimizations:
* Don't register the same directory for watching:
- use the `ConcurrentMap<Path, WatchKey> pathKeys` to track registrations
- any modification within the directory causes the `ENTRY_MODIFY` for the directory as well.
So, skip such an event exactly for the directory during `walkDirectory`
* Add debug logs in case of `ENTRY_DELETE`
https://build.spring.io/browse/INT-B43-164/
* Fix CheckStyle for `System. out .println(` expression commenting out such a code line
* Fix generics in the `JdbcMessageStoreTests`
* Increase receive timeout in the `FileMessageHistoryTests`
* Change the `AsyncAmqpGatewayTests` logic to deal with the same channel for both `ack/nack`.
The `timeout` case without reply may end up with the `Consumer cancel`, therefore `nack` not `ack` like we expected
JIRA: https://jira.spring.io/browse/INT-3387,
https://jira.spring.io/browse/INT-3806
* Introduce
```
MessageGroupStore
void addMessagesToGroup(Object groupId, Message<?>... messages);
```
And implement it in all stores.
* Use new `addMessagesToGroup` where it is reasonable, e.g. `DelayHandler`
* Optimize test-case to use a new store method (where it is possible)
* Fix timing delays in the `JdbcMessageStoreTests`
* Introduce `PersistentMessageGroup`
* Add `AbstractMessageGroupStore#proxyMessageGroupForLazyLoad` to wrap the raw `MessageGroup` to the `PersistentMessageGroup` for lazy-load
* Rework `MessageGroupMetadata` do not be `immutable` and allow to store/restore in the `AbstractKeyValueMessageStore` only the `MessageGroupMetadata`
* Refactor `ResequencingMessageHandler` and `SequenceSizeReleaseStrategy` a bit for better performance when interact with the `MessageGroup`
* Add `AbstractMessageGroupStore#setLazyLoadMessageGroups` to switch off the `lazy-load` behavior and restore the previous full `MessageGroup` logic
* Add `What's New` note and `message-store.adoc` paragraph for the lazy-load functionality
`GroupType.PERSISTENT` and not lazy by default
PR Comments
Fix `JdbcMessageStoreTests` timing issues
Address PR comments
* Add performance test to the `ConfigurableMongoDbMessageGroupStoreTests`
* Add JavaDocs for the `MessageGroupFactory` methods
* Add `log4j.properties` into the `test` MongoDB module for better traceability
* Fix `JdbcMessageStore#getOneMessageFromGroup()` over the `doPollForMessage()` delegation.
The `jdbcTemplate.queryForObject()` requires exactly one and only one raw in `resultSet`
* Add performance test results into the `message-store.adoc`
checkstyle EmptyBlock
checkstyle fixRightCurly Script
checkstyle EmptyStatement
checkstyle RightCurly
checkstyle TailingWhite
checkstyle NeedBraces
Fix the line separator in the `fixRightCurly.gradle`
fixes
fixModifiers after fixFinal
Revert CachingSessionFactory
Class is spied in tests.
checkstyle - Import Rules
checkstyle InterfaceIsType
checkstyle InnerTypeLast
checkstyle OneStatementPerLine
CovariantEquals
OneTopLevelClass
* Revert `'\n'` -> `System.lineSeparator()` in the Gradle scripts to meet Git `autocrlf = true` on Windows
* Fix timing issue with the `LastModifiedFileListFilterTests`, when the `age = 1` might not be enough for the file object when we have some delay before checking
JIRA: https://jira.spring.io/browse/INT-3944
Polishing (PR comments) and Doc Polish
Polishing - PR Comments
Remove custom `async` boolean in favor of the now public setter.
Add exception to `JmsException` hierarchy.
INT-3944: Polishing
* Rename `asyncReplySupported` just to `async`. As well as its getter and setter. Plus fix docs on the matter
* Polishing for the `JmsOutboundGateway` according PR comments
* Rework `JmsOutboundGateway` to return `AbstractIntegrationMessageBuilder` instead of `Message`
since `AbstractMessageProducingHandler` rebuilds `Message` for a new one to copy headers from request
* Add `getPayload()` and `getHeaders()` to the `AbstractIntegrationMessageBuilder` to make the `Routing Slip`
users happy, when the `AbstractIntegrationMessageBuilder` is pushed to the `Routing Slip path` function
* Fix race condition in the `ChatMessageListeningEndpointTests`: the `stanza` parsing is done in the different Thread.
Doc Polishing (Routing Slip)
Fix timing issue in the `LastModifiedFileListFilterTests`:
`Thread.sleep()` not always reflects the reality.
Switch to the "past simulation" via explicit `File.setLastModified()`
* `gradlew clean check -x test --parallel --continue` - to collect reports
* `gradlew fixThis --parallel` - to fix all possible vulnerabilities. With `-Dfile.encoding=UTF-8` on Windows
Since the `RequireThisCheck` doesn't see parents for anonymous classes (e.g. `Runnable` callback), its report doesn't contains the outer class name with `this.`,
therefore we still have to fix those cases manually.
Thanks to the wrong `replacer` just with `this.` we have uncompilable code enough easy to find problems.
Not so easy to fix for good readability though...
* Upgrade to Grade 2.12
* Upgrade to SonarQube native plugin
The fix contains at about 300 files. So, will be done on merge.
Fix `fixThis.gradle` according PR comments
Apply `fixThis` and also `fixModifiers` for test classes.
Fix some `this.` inner issues manually.
Make code polishing for long lines after `fixThis`
Fix conflicts and vulnerabilities after the rebase
Add `fixModifiers.gradle` to run after `check` task.
The `RedundantModifierChecker` doesn't catch all errors at once.
We need run `check -> fixModifiers` pair several times to reach `SUCCESSFUL`.
The `fixThis` has been ported from the SA, but without applying.
* Polishing for `fixModifiers.gradle`
* Fix all redundant modifier with the `fixModifiers.gradle`
NOTE: Since all these task modify files on Windows we have to run them with the `-Dfile.encoding=UTF-8`.
Otherwise they are read and write with the Windows default `cp1251` making them incompatible with Unix system.
https://build.spring.io/browse/INT-MJATS41-569/
The `fixHeaders` Gradle task (and similar) must be run from Windows with the `-Dfile.encoding=UTF-8`.
Otherwise read/write operations for files are done in the native Windows `cp1251`
