JIRA: https://jira.spring.io/browse/INT-4175
Fixes GH-1980 (https://github.com/spring-projects/spring-integration/issues/1980)
When the `isSharedSession` is used for the `DefaultSftpSessionFactory`,
there is some race condition window when we can call the target `this.jschSession.connect()` several times and end up with the session is already connected.
Wrap `sftpSession.connect()` to the `this.sharedSessionLock.readLock().lock()` when `isSharedSession` to protect from that race condition.
Note: there is no test for this change because it is pretty tricky to build barriers between threads for this kind of race conditions.
Especially after introduction `this.sharedSessionLock.readLock().lock()` around guilty code
**Cherry-pick to 4.3.x & 4.2.x**
Move `sharedJschSession` connect logic to the existing locking block
Fix mock test to reflect the current ConnectionFactory state
Conflicts:
spring-integration-sftp/src/test/java/org/springframework/integration/sftp/outbound/SftpOutboundTests.java
Resolved.
* Fix several typos in log messages. And some test on the matter as well
* Add comment to `AbstractPersistentAcceptOnceFileListFilter.rollback()` to clarify the reason of `rollingBack` variable
* Make `RemoteFileTemplate.StreamHolder` as `static` to avoid extra internal variable to outer class instance
* Replace `MessagingException` with `AbstractInboundFileSynchronizingMessageSource` in `init()` method of some components. It isn't Messaging yet in that phase
* Fix `SubscribableRedisChannel.MessageListenerDelegate` to handle `Object` not `String`, because with the `serializer` injection there is no guaranty that incoming is always `String`
* Move `JSch.setLogger(new JschLogger());` in the `DefaultSftpSessionFactory` to `static` block. It really should be done only once
* Remove `Assert.isTrue(this.port >= 0)` from the `DefaultSftpSessionFactory`. The subsequant `initJschSession()` convert it to default `22` port
* Change in the `JschProxyFactoryBean` `UnsupportedOperationException` to `IllegalArgumentException`. Wrong enum is wrong argument. That isn't a problem of operation
* Simplify `stop()` in the `CuratorFrameworkFactoryBean` and mark it as a `this.running = false`. Otherwise it wasn't able to be restarted
* Expose `leaderEventPublisher` in the `LeaderInitiatorFactoryBean` and fix `stop(Runnable callback)` with propagation `callback` to delegate.
Fix `SubscribableRedisChannelTests` for new `handleMessage(Object)` signature
JIRA: https://jira.spring.io/browse/INT-4046
Since not all FTP servers provide proper `STAT` command implementation,
plus the `NLIST` doesn't work properly for directories cases, introduce the `FtpRemoteFileTemplate.ExistsMode`
to let:
* to perform `STAT` by default (previous) behavior;
* to switch to `NLIST` for `FtpRemoteFileTemplate` internal use;
* perform the full `NLIST` and `FTPClient.changeWorkingDirectory()` algorithm if needed.
* Improve (S)Ftp components to use proper `RemoteFileTemplate` for internal instantiation
* Introduce `FtpMessageHandler` to wrap `FtpRemoteFileTemplate` with the proper `NLIST` `ExistsMode`
* Cover `NLIST` switching from the `FtpOutboundChannelAdapterParser` and `FtpOutboundGatewayParser`
* Document the `FtpRemoteFileTemplate.ExistsMode`
* Fix typo in the recently introduced `RemoteFileOperations.getSession()` method name
* Add JavaDoc to `Session.exists()`
* Add `NLIST` support for the `FtpSession.exists()` to meet the API requirements
**Cherry-pick to 4.2.x and 4.1.x**
Addressing PR comments
Doc Polishing
JIRA: https://jira.spring.io/browse/INT-4021
Previously the `SimpleMessageStore` unconditionally followed with the super class options
and provided the `lazy-load` functionality by default.
* Since `persistent` and `lazy-load` logic does not make sense for the `in-memory` store, disable it in the `SimpleMessageStore`
* Fir `AggregatorTests` for better coverage.
* NOTE: The same message can't be persisted in the Persistent `MessageStore`. The store key is fully based on the `messageId`.
And also we provide the header which indicates that the messages has been stored before. See `JdbcMessageStore.addMessage()` for example:
```
if (message.getHeaders().containsKey(SAVED_KEY)) {
Message<T> saved = (Message<T>) getMessage(message.getHeaders().getId());
if (saved != null) {
if (saved.equals(message)) {
return message;
} // We need to save it under its own id
}
}
```
* Fix (S)FTP Streaming tests to use `AbstractPersistentAcceptOnceFileListFilter` instead of raw `AcceptOnceFileListFilter`,
which relies on the object identity, but neither `FtpFile`, nor `ChannelSftp.LsEntry` provides good `equals()` and `hashCode()` implementations.
* Make `StompIntegrationTests` as `DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD` because the sporadic failure on the Travis around wrong value from the queue isn't clear (yet).
Looks like some interim event is generated by the Websocket Container on the Tomcat.
JIRA: https://jira.spring.io/browse/INT-4015https://jira.spring.io/browse/INT-3854
Initial commit.
Reworked to emit an input stream and use the file splitter.
Add StreamTransformer.
Add CLOSABLE_RESOURCE header so we can close the session automatically.
Implement INT-3854, FTP, SFTP
(S)FTP Namespace Changes
Docs - also fixes a PDF overflow
Polishing - PR Comments
checkstyle fixes
Polishing - Add Namespace for StreamParser
Polishing - PR Comments
* Upgrade to SA and SF snapshots
* Address TODOs regarding those upgrades
* Resolve some other TODOs
* Replace `PropertyPlaceholderConfigurer` beans with the `<context:property-placeholder>`
* Upgrade to Spring Social Twitter 1.1.2 and resolve deprecations via mocks (https://github.com/spring-projects/spring-social-twitter/issues/91)
* Upgrade to Curator `3.1.0` and resolve deprecation in the `ZookeeperMetadataStore`
checkstyle EmptyBlock
checkstyle fixRightCurly Script
checkstyle EmptyStatement
checkstyle RightCurly
checkstyle TailingWhite
checkstyle NeedBraces
Fix the line separator in the `fixRightCurly.gradle`
* `gradlew clean check -x test --parallel --continue` - to collect reports
* `gradlew fixThis --parallel` - to fix all possible vulnerabilities. With `-Dfile.encoding=UTF-8` on Windows
Since the `RequireThisCheck` doesn't see parents for anonymous classes (e.g. `Runnable` callback), its report doesn't contains the outer class name with `this.`,
therefore we still have to fix those cases manually.
Thanks to the wrong `replacer` just with `this.` we have uncompilable code enough easy to find problems.
Not so easy to fix for good readability though...
* Upgrade to Grade 2.12
* Upgrade to SonarQube native plugin
The fix contains at about 300 files. So, will be done on merge.
Fix `fixThis.gradle` according PR comments
Apply `fixThis` and also `fixModifiers` for test classes.
Fix some `this.` inner issues manually.
Make code polishing for long lines after `fixThis`
Fix conflicts and vulnerabilities after the rebase
Even it isn't harmful, the one useless class in the `.file.test` package looks confusing.
Move it out to the `test` scope and change the dependant projects respectively.
* `ClientWebSocketContainer`: add some synchronization to avoid race conditions: https://build.spring.io/browse/INT-B41-492
* `TomcatWebSocketTestServer`: convert to `0` port to rely on the OS resolution for `localPort`
* Add `LogAdjustingTestSupport` for STOMP test
* `SftpServerTests`: use `0` port to rely on the OS resolution for `localPort`
* `ImapMailReceiver`, `OutboundGatewayFunctionTests` (JMS), `CachingClientConnectionFactoryTests`,
`AsyncGatewayTests`, `AsyncMessagingTemplateTests`, `GatewayParserTests`, `PriorityChannelTests`, `AggregatorIntegrationTests`: increase timeout
* `EnableIntegrationTests`: use `LogAdjustingTestSupport`
* `FileOutboundChannelAdapterParserTests`: rework `Thread.sleep()` with `CountDownLatch`
* `ConnectionToConnectionTests`: increase timeout and count iteration. Previously with `1sec` we may lose some events. And we can't just rely on the `10sec`,
because the last iteration will be so long
* `TcpOutboundGatewayTests`: `500ms` is so big timeout to wait for the `Exception` that in the high load environment we can yield to other Thread so long.
Like in our case to `server` Thread to send the reply for us. Therefore decrease the Exception timeout to the `50ms` and increase server delay to `2sec`
* `StompInboundChannelAdapterWebSocketIntegrationTests`: remove `@Qualifier("taskScheduler")` as a potential candidate to test against latest SF changes.
We're fine with `SF-4.2.2` and it is just a test-case. So, I don't see reason to wait for their fix here.
STOMP: `session = null` in adapters for any transportError
Polishing
JIRA: https://jira.spring.io/browse/INT-3840
* In addition: fix some typos in the `SftpInboundRemoteFileSystemSynchronizerTests`
**Cherry-pick to 4.1.x**
JIRA: https://jira.spring.io/browse/INT-3827
Provide a hook to enable removing a file from an `AcceptOnceFileListFilter`,
for example after a message processing failure.
Make the `CompositeFileListFilter` a `ReversibleFileListFilter` so it can
delegate to any of its composed filters that are reversible.
INT-3827: Polishing - PR Comments
JIRA: https://jira.spring.io/browse/INT-3796
When a `UserInfo` is provided, use it for the password and passphrase.
Disallow local properies when a `UserInfo` is provided.
Simple polishing and fixing `TestSftpServer` for the `this.server.stop(true);`
to avoid `port is already in use` for other tests which use default SSH port.
JIRA: https://jira.spring.io/browse/INT-3796
Previously, the `DefaultSftpSessionFactory` unconditionally accepted keys
from unknown hosts, or when a host key changed.
Permit this behavior to be configured with `allowUnknownHosts` and/or a custom
`UserInfo` properties in the factory.
`allowUknownHosts` is false by default (since 4.2) and true by default in 4.1.x for
backwards compatibility.
INT-3796: Fix Tests
INT-3796: Polishing
JIRA: https://jira.spring.io/browse/INT-3749
Since `IntegrationEvaluationContextAware` isn't so "context-free" resource like `BeanFactory`
and `ApplicationContext`, but just a specific bean in the context, we can't follow with `BeanPostProcessor` logic - bad architecture by level of responsibility.
Therefore we should follow with standard Dependency Injection mechanism to retrieve `evaluationContext` for the particular component.
Since we can't rely on the `@Autowired` because SI can be used from the raw XML configuration,
we use utility method instead. The pattern to get the proper `integrationEvaluationContext` is:
```
@Override
protected void onInit() throws Exception {
super.onInit();
this.evaluationContext = ExpressionUtils.createStandardEvaluationContext(getBeanFactory());
}
```
JIRA: https://jira.spring.io/browse/INT-3737
Support returning an `InputStream` from a GET operation.
This can be used in conjuction with a `<file:splitter/>` to stream a text file.
The user is responsible for releasing the session when the download is complete.
The session object is stored in the message header and `RemoteFileUtils.closeSession()` should be called
to clean up and close the session.
INT-3737: Polishing and Docs
Fix `CachedSession.close()` bug
Preventing `Caused by: java.io.IOException: Previous raw read was not finalized`
when `Session` is returned to the pool, but `readingRaw` hasn't been finalized
because the real `close()` isn't invoked in case of normal return to pool.
JIRA: https://jira.spring.io/browse/INT-3589, https://jira.spring.io/browse/INT-3624
* SF - 4.2. Fix Breaking changes for the `ApplicationEventPublisher`
* AMQP 1.5. Without issues
* Reactor - 2.0. Fix for the new `Stream` foundation
* jsonPath - 1.2.0. Fix for new `Predicate` abstraction. Add 'fail-fast error' to the `IntegrationRegistrar`
* Sshd - 0.13.0. Fix for new `VirtualFileSystemFactory` usage
* Spring Data - Fowler
* And others without issues
* Get rid of `reactor.util.StringUtils` usage
INT-3589: Address PR comments
Polishing
JIRA: https://jira.spring.io/browse/INT-3581
Move schemas to 4.2.
Add `selector-expression` to `<wire-tap/>`.
INT-3781: Fix What's New
Bump Namespace Version to 4.2
JIRA: https://jira.spring.io/browse/INT-3606
Close the store when an (S)FTP synchronizing MessageSource is stopped.
Will need another commit on 4.2.WIP to change to `Lifecycle`.
Add dirty Flag to Properties MetadataStore
Avoid unnecessary persists.
Remove unnecessary implementation of `AbstractInboundFileSynchronizingMessageSource#getComponentType()`
JIRA: https://jira.spring.io/browse/INT-3412
Initial commit - review only.
TODO:
- SFTP Tests
- Namespace/Adapter support for file append
- Docs
INT-3412 Polishing
- Addressed PR comments
- Completed SFTP implementation
- Added namespace/parser support for `FileExistsMode` (append, etc)
- Added SFTP Tests
- Created Embedded SFTP server for tests (similar to FTP)
- Converted tests that needed a real server to use the embedded server
INT-3412 Docs and Polish (PR Comments)
JIRA: https://jira.spring.io/browse/INT-3464
Previously, if an IO exeption occurred while synchronizing
files, and an AcceptOnceFileListFilter is being used, files that were
not transferred would not be fetched next time.
Add strategy `ReversibleFileListFilter` that can rollback previously
accepted files.
Implement this interface on `AcceptOnceFileListFilter` and
`AbstractPersistentAcceptOnceFileListFilter`.
Add test cases.
Polishing
