Commit Graph

259 Commits

Author SHA1 Message Date
Gary Russell
a4f7412cf5 GH-3026 Support chmod with FTP
Resolves https://github.com/spring-projects/spring-integration/issues/3026

* Fix exception messages; remove test TODOs; test works on Windows
2019-08-15 17:38:24 -04:00
Gary Russell
cd0f56bc87 Add Apache MINA SftpEventListener
- republish certain events as `ApplicationEvent`s.

* * Add ApacheMinaFtplet to provide the same functionality with FTP
* Fix typo

* * Polishing javadocs and event toString() methods
2019-07-30 10:31:23 -04:00
Artem Bilan
80d679a9b0 GH-2748: More bean definitions into exceptions
Fixes https://github.com/spring-projects/spring-integration/issues/2748

* Refactor more `MessageHandlingException`s to include `this` into an
exception message
* Revert using `MessagingException` in some places which really are not
about messaging.
This helps to wrap them into `MessageHandlingException` later in the
`MessageHandler` for the `BeanDefinition` reference
* Remove `volatile` from configuration properties in the affected
classes
* Remove already deprecated `JmsOutboundGateway.setPriority()`
* Add `resource` and `source` for `BeanDefinition` in the
`AbstractChannelAdapterParser` & `AbstractInboundGatewayParser`
* Document the feature
2019-07-23 15:19:56 -04:00
Artem Bilan
c109e1df34 GH-2987: Add HTTPS entries into spring.schemas
Fixes https://github.com/spring-projects/spring-integration/issues/2987

To resolve XSD files properly from the classpath, their HTTPS reference
must be present in the `spring.schemas` to avoid the Internet interaction
for resolving an XSD file

**Cherry-pick to 5.1.x, 5.0.x & 4.3.x**
2019-07-18 14:05:18 -04:00
Gary Russell
86c7e36667 Sonar issues - complexity
* Fix checkstyle issue.
2019-05-03 20:05:16 -04:00
Gary Russell
6d7bc1fc39 Sonar: repeated literals
* Polishing - PR Comments

* GatewayParser: Restore suppress warnings; remove size from `toArray()`.

* Merge conflict resolution
2019-05-03 12:39:02 -04:00
Gary Russell
0d6faa2e34 Sonar - protected fields 2019-05-01 17:28:42 -04:00
Gary Russell
36c33bb5ab Sonar fixes
- a few complexities
- final method calls from ctor
- raw exception throwing
- useless overrides
- loss of stack trace
2019-04-29 18:20:40 -04:00
Artem Bilan
d21e8f0349 URL Cleanup
This commit updates URLs to prefer the https protocol.
Redirects are not followed to avoid accidentally expanding intentionally
shortened URLs (i.e. if using a URL shortener).

# Fixed URLs

## Fixed Success
These URLs were switched to an https URL with a 2xx status.
While the status was successful, your review is still recommended.

* [ ] http://www.apache.org/licenses/ with 1 occurrences migrated to:
  https://www.apache.org/licenses/ ([https](https://www.apache.org/licenses/) result 200).
* [ ] http://www.apache.org/licenses/LICENSE-2.0 with 2638 occurrences migrated to:
  https://www.apache.org/licenses/LICENSE-2.0 ([https](https://www.apache.org/licenses/LICENSE-2.0) result 200).
2019-03-21 19:22:26 -04:00
Artem Bilan
20e8f0af2f Fix HTTP urls to HTTPS
* Do not fix urls for the `http://apache.org/xml/features` when we
configure XML components
* Do not fix urls for localhost and fake links in tests
* Fix all the dead links in docs
2019-03-21 17:16:30 -04:00
Artem Bilan
8380346821 GH-2820: Close session in FtpRemFileTempl.exists
Fixes: https://github.com/spring-projects/spring-integration/issues/2820

The current `FtpRemoteFileTemplate.exists()` implementation obtains a
`Session` for the `NLST_AND_DIRS` mode, but doesn't close it.

* Fix the `FtpRemoteFileTemplate.exists()` method to delegate to `super`
for the proper interaction with a `Session`

**Cherry-pick to 5.1.x, 5.0.x & 4.3.x**
2019-03-19 11:16:04 -04:00
Artem Bilan
d2e974a6de Fix new Sonar smells (#2768)
* Fix new Sonar smells

* Fix some old Sonar smells as well
* Fix Micrometer leaks in the `PollableChannel` when we register
meters, but don't remove them.

* * Fix NPE around `MetricsCaptor` in channels

* * Fix new smells according test report

* * Further Sonar smell fixes

* * More smell fixes for `MessagingMethodInvokerHelper`
* Remove `throws Exception` from `AbstractMessageHandler.destroy()`

* * Fix complexity in the `MessagingMethodInvokerHelper.processInvokeExceptionAndFallbackToExpressionIfAny()`
2019-02-27 15:17:51 -05:00
Artem Bilan
7e3b56939e Fix XSDs for actual core schema version 2019-02-21 12:43:21 -05:00
Gary Russell
520ebbbd2c Move master to 5.2 2019-02-15 12:01:24 -05:00
Gary Russell
4ea3b1c23c GH-2705: public CTORs on AMFPFileListFilter Impls
Fixes https://github.com/spring-projects/spring-integration/issues/2705
2019-01-18 11:12:38 -05:00
Gary Russell
e40cfe101e Sonar fixes
Hidden fields, `o.s.i.a*` through `o.s.i.j*`.
2019-01-09 10:02:19 -05:00
Gary Russell
e8df546b04 Sonar Fixes - Javadoc-only imports 2019-01-02 12:26:22 -05:00
Gary Russell
62fc7df693 Sonar Fixes
Critical smells, packages `o.s.i.a*` to `f*`.

Plus new smells caused by these changes.
2018-12-03 10:44:26 -05:00
Gary Russell
12bc24dd02 File, FTP, IP Sonar Fixes 2018-10-26 13:53:54 -04:00
Gary Russell
bd7a3bc4ed GH-2605: (S)FTP test cached sessions
Resolves https://github.com/spring-projects/spring-integration/issues/2605

* Suppress unused field warning.

* Fix typos.

* Use lstat() - don't follow symbolic lincs
2018-10-23 13:59:00 -04:00
Gary Russell
11d39b7da1 INT-4496: (S)FTP: Sort file list earlier
JIRA: https://jira.spring.io/browse/INT-4496

Sort the file list before applying `maxFetchSize` and filters.

* Polishing - PR Comments
2018-07-02 16:49:23 -04:00
Gary Russell
3f218a97a0 INT-4495: (S)FTP: fix max-fetch with directories
JIRA: https://jira.spring.io/browse/INT-4495

Previously, `maxFetch` was applied before directories were removed from
the fetch list.

Remove the directories before filtering and applying `maxFetch`.

* Polishing - PR Comments

* More polishing

* More polishing.

* Check for empty array.

* Remove test main method.
2018-06-29 14:21:06 -04:00
Artem Bilan
027021d6e4 Move XSDs to version 5.1
* Upgrade to Spring AMQP-2.1 and fix compatibility
2018-03-17 09:58:40 -04:00
Artem Bilan
077b5ffffc INT-4344: AbstRemoteFileOutGw: Add assert for NPE
JIRA: https://jira.spring.io/browse/INT-4344

The `FtpOutboundGateway` provides ctors without expression for remote path.
In this case it is treated as a `working directory` but only for the
`LS`, `NLST`, `PUT` and `MPUT` commands.

* Add assertion in the `AbstractRemoteFileOutboundGateway` to discard
configuration for all other commands when `expression` is `null`

**Cherry-pick to 4.3.x**
2017-10-16 17:04:22 -04:00
Venil Noronha
797e5045bf INT-4334: Adds a DirectoryScanner to RemoteFileMS
JIRA: https://jira.springsource.org/browse/INT-4334

* Add a `DirectoryScanner` setter named `setScanner` in
`AbstractInboundFileSynchronizingMessageSource` to set the
`FileReadingMessageSource`'s scanner.
* Add tests to `AbstractRemoteFileSynchronizerTests`.

Polish.

Updates xsd, tests and documentation.

Fix xsd formatting.

Addresses review comments.

* Some code style polishing
2017-09-25 21:08:51 -04:00
Gary Russell
2c7bf9271c INT-4283: MarkerFilePresentFileListFilters
JIRA: https://jira.spring.io/browse/INT-4283

Filter that only passes files where a corresponding "marker" file is also present.

Implementations for file and(S)FTP.

* Polishing - PR Comments

* Missed one
2017-06-02 19:31:05 -04:00
Artem Bilan
0379565264 Fix JavaDocs for Remote File components
https://build.spring.io/browse/INT-MASTER-684/
2017-05-25 14:34:29 -04:00
Artem Bilan
3daeaab1b4 INT-4060: FTP Gateway: Add NLST and workDir
JIRA: https://jira.spring.io/browse/INT-4060

* Add `NLST` command to the `AbstractRemoteFileOutboundGateway` to perform
`listNames` on the target session.
Useful in case of server doesn't allow to perform `LS` or the names set is
sufficient for application requirements
* Add `workingDirExpression` to the `FtpOutboundGateway` to allow to perform
`FtpClient.changeWorkingDirectory()` based on the current request message
* Change `slf4j-log4j12` to the `testCompile` -
the FTP tests fail in the IDE with `ClassNotFoundException`

Address PR comments

* Add `nlst` to XSD config
* Reinstate `ls -1` test-case for the `FtpServerOutboundTests`
* wrap more commands to the `doInWorkingDirectory()`

Fix `MV` command in the `AbstractRemoteFileOutboundGateway`

* Implement `RemoteFileOperations#invoke(OperationsCallback<F, T>)`
for thread-bound `session`s
* Use a new `invoke()` for `put()` and `mPut()` commands in the `AbstractRemoteFileOutboundGateway`
* Add delegation for the `put()` and `mPut()` commands in the `FtpOutboundGateway`
* Add DSL support for the `workingDirExpression` and add `whats-new.adoc` note

Document changes

Address some PR comments:

* Add `invokeScope` variable to `execute` to track `ThreadLocal` session or not
* Check for `null` in the `getSession()` and fallback to regular
`sessionFactory.getSession()`.
Most likely the `invoke()` is called from other thread

Doc Polishing
2017-05-25 11:23:30 -04:00
Artem Bilan
10ce68d3e3 INT-4115: (S)FtpPersistentFileFilter by Default
JIRA: https://jira.spring.io/browse/INT-4115

Apply `(S)FtpPersistentAcceptOnceFileListFilter` for the `(S)FtpInboundFileSynchronizer` by default to avoid cases to sync the same remote files to the local directory again.
Especially when `localFileName` strategy is applied and we end up with new local files, but with the same remote content

Accept `(S)FtpPersistentAcceptOnceFileListFilter` for streaming adapters

Make `doSetFilter()` as `protected final`

Fix tests after rebase

Fix "What's New" after rebase

Compose `PersistentAcceptOnceFileListFilter` together with the regex or pattern filters
Document such a behavior

Address PR comments for formatting and typo
2017-03-14 15:37:43 -04:00
Artem Bilan
a4bfd2cc42 INT-4221: Properly use Spring's Assert class
JIRA: https://jira.spring.io/browse/INT-4221

* Upgrade to those versions of Spring project dependencies which potentially will provide similar fix

**Cherry-pick to 4.3.x**
2017-01-30 22:27:59 -05:00
Artem Bilan
9836828f54 INT-4204: Introduce ExpressionFileListFilter
JIRA: https://jira.spring.io/browse/INT-4204

Fix error message in the `AbstractRemoteFileOutboundGatewayParser`
2017-01-19 16:30:44 -05:00
Gary Russell
57806b92c9 INT-4171: Document Solution for FTPS Shared SSL
JIRA: https://jira.spring.io/browse/INT-4171

Since the solution requires reflection on `sun` classes, document only.

Also fix some PDF text overflows in (S)FTP.
2016-11-26 16:24:33 -05:00
Gary Russell
8070f72c44 INT-4155: Port FTP DSL
JIRA: https://jira.spring.io/browse/INT-4155

- Add streaming support
- Rework tests to use dynamic flow registrations due to the test FTP server is now a test support class.

Javadoc fixes

Checkstyle fix

Polishing - PR Comments

Polishing and Port SFTP
2016-11-14 13:22:45 -05:00
Artem Bilan
5cca8e8e01 INT-3770: Add TX Support from Mid-flow
JIRA: https://jira.spring.io/browse/INT-3770,
https://jira.spring.io/browse/INT-4107

Having `TransactionHandleMessageAdvice` we can start TX from any `MessageHandler.handleMessage()`

* Add `<transactional>` alongside with the `<request-handler-advice-chain>` for those components which produce reply
* Merge `<transactional>` and `<request-handler-advice-chain>` configuration to a single `ManagedList`
* Rework JPA `<transactional>` in favor of common solution
* Some polishing and refactoring

AbstractPollingEndpoint: avoid `new ArrayList` if we don't  have `receiveOnlyAdvice`s
2016-11-07 09:06:17 -05:00
Gary Russell
a43299213e INT-4149: Improve (S)FTP Recursive MGET
JIRA: https://jira.spring.io/browse/INT-4149

Add an option to always pass directories while recursing.

* Fix typos in docs
* Revert `SyslogReceivingChannelAdapterTests` `Thread.sleep()` fixes
2016-10-28 16:22:46 -04:00
Gary Russell
5a44216201 INT-4140: File Streaming Adapter: Add maxFetchSize
JIRA: https://jira.spring.io/browse/INT-4140
2016-10-18 10:32:23 -04:00
Gary Russell
fc3088d77d INT-4128: FTP ControlEncoding Fix
JIRA: https://jira.spring.io/browse/INT-4128
2016-09-29 10:00:25 -04:00
Gary Russell
e5ae7d886d (S)FTP Lambdas
AMQP Lambdas

Event Lambdas

Feed/File Lambdas

Gemfile/Groovy Lambdas

* Ensure that JavaDocs are checked via `	check.dependsOn javadoc`
* Add `@return` tag to the `MessageBuilder.readOnlyHeaders()`
2016-09-26 16:50:01 -04:00
Artem Bilan
12fb590b95 Mainly Checkstyle Violation Fixes
* Upgrade to Checkstyle `7.1`
* Relax `RequireThis` rule a bit. Right now it does the effort only in case of overlapping. See https://github.com/checkstyle/checkstyle/issues/2362 for more info
* Enable some annotation rules and provide fixes for violations
* Enable `tabs indents` rule. This was the biggest fix in this PR
* Resolve `TODO` in the `MessagingMethodInvokerHelper` and fix tests to meet `IllegalStateException` now

Remain Checkstyle version `6.16.1` and reinstate `RequireThis` rule

The latest Checkstyle has a bug with local scope variables if they have the same names as property.

Revert some literals splitting

Fix some line length exceeding and code style
2016-09-22 13:23:51 -04:00
Artem Bilan
370be4853d Apply Some Java 8 Code Changes
* Make most functional interfaces as `@FunctionalInterface`
* Convert some abstract classes to `@FunctionalInterface` with `default` methods
* Apply Lambda style implementation in some places
* Remove `Function` in favor of similar in Java 8

*  Remove redundant code from `DefaultAmqpHeaderMapper` since we are already on Spring AMQP-2.0
* Add several ctors to the `ExpressionEvaluatingMessageListProcessor`
* Populate explicit `Boolean.class` `expectedType` from the `ExpressionEvaluatingReleaseStrategy`
2016-08-29 09:14:31 -04:00
Gary Russell
ea4763faa9 INT-4095: Support Limiting (S)FTP Files Fetched
JIRA: https://jira.spring.io/browse/INT-4095

Limit the number of remote files fetched on each poll (when it is necessary to fetch files).

Polishing - PR Comments

Polishing - Decouple MaxFetchSize from Poller

Polishing - PR Comments

Schemas and Docs

More Polishing

* Polishing according PR comments
2016-08-26 13:36:14 -04:00
Gary Russell
04a42f54be Rename Schemas to 5.0
Fix Schema Imports

spring.schemas
2016-08-18 14:40:38 -04:00
Artem Bilan
c620438913 INT-4046: Add FtpRemoteFileTemplate.ExistsMode
JIRA: https://jira.spring.io/browse/INT-4046

Since not all FTP servers provide proper `STAT` command implementation,
plus the `NLIST` doesn't work properly for directories cases, introduce the `FtpRemoteFileTemplate.ExistsMode`
to let:
* to perform `STAT` by default (previous) behavior;
* to switch to `NLIST` for `FtpRemoteFileTemplate` internal use;
* perform the full `NLIST` and `FTPClient.changeWorkingDirectory()` algorithm if needed.

* Improve (S)Ftp components to use proper `RemoteFileTemplate` for internal instantiation
* Introduce `FtpMessageHandler` to wrap `FtpRemoteFileTemplate` with the proper `NLIST` `ExistsMode`
* Cover `NLIST` switching from the `FtpOutboundChannelAdapterParser` and `FtpOutboundGatewayParser`
* Document the `FtpRemoteFileTemplate.ExistsMode`

* Fix typo in the recently introduced `RemoteFileOperations.getSession()` method name
* Add JavaDoc to `Session.exists()`
* Add `NLIST` support for the `FtpSession.exists()` to meet the API requirements

**Cherry-pick to 4.2.x and 4.1.x**

Addressing PR comments

Doc Polishing
2016-06-08 14:34:35 -04:00
Gary Russell
287d924fc0 INT-4015: Streaming Remote File Inbound Adapter
JIRA: https://jira.spring.io/browse/INT-4015
      https://jira.spring.io/browse/INT-3854

Initial commit.

Reworked to emit an input stream and use the file splitter.

Add StreamTransformer.

Add CLOSABLE_RESOURCE header so we can close the session automatically.

Implement INT-3854, FTP, SFTP

(S)FTP Namespace Changes

Docs - also fixes a PDF overflow

Polishing - PR Comments

checkstyle fixes

Polishing - Add Namespace for StreamParser

Polishing - PR Comments
2016-04-29 13:24:23 -04:00
Gary Russell
05cc7be644 checkstyle WhiteAround
WhiteAroundCheck script
2016-04-05 13:50:11 -04:00
Gary Russell
842aded9a4 checkstyle MutableException
checkstyle EmptyBlock

checkstyle fixRightCurly Script

checkstyle EmptyStatement

checkstyle RightCurly

checkstyle TailingWhite

checkstyle NeedBraces

Fix the line separator in the `fixRightCurly.gradle`
2016-04-05 13:10:33 -04:00
Artem Bilan
2b0598291c RequireThis rule and fixThis Gradle task
* `gradlew clean check -x test --parallel --continue` - to collect reports
* `gradlew fixThis --parallel` - to fix all possible vulnerabilities. With `-Dfile.encoding=UTF-8` on Windows

Since the `RequireThisCheck` doesn't see parents for anonymous classes (e.g. `Runnable` callback), its report doesn't contains the outer class name with `this.`,
therefore we still have to fix those cases manually.
Thanks to the wrong `replacer` just with `this.` we have uncompilable code enough easy to find problems.
Not so easy to fix for good readability though...

* Upgrade to Grade 2.12
* Upgrade to SonarQube native plugin

The fix contains at about 300 files. So, will be done on merge.

Fix `fixThis.gradle` according PR comments

Apply `fixThis` and also `fixModifiers` for test classes.
 Fix some `this.` inner issues manually.
 Make code polishing for long lines after `fixThis`

Fix conflicts and vulnerabilities after the rebase
2016-03-22 20:18:27 -04:00
Artem Bilan
66cd92a6e7 INT-3964: More @EnableIntegration Documentation
JIRA: https://jira.spring.io/browse/INT-3964

Doc Polish

XSD: document classes for element implementations

Fix Redis XSD typo for CDATA
2016-03-22 10:41:07 -04:00
Artem Bilan
ef40a939cf Add check header rule and fixHeaders task
Add "new line" in the header end

* Optimization for the `fixHeaders.gradle`
* Apply the `fixHeaders` for the affected classes
2016-03-07 15:08:51 -05:00
Artem Bilan
7595e4f142 INT-3919: FTP: Allow null for Remote Directory
JIRA: https://jira.spring.io/browse/INT-3919

Since `FtpClient` supports `null` for the `LS` command, treating it as a current `working directory`,
there is no reason to forbid `null` from the FTP adapters end-user perspective.

* Allow `null` for the `FtpSession` `list()` and `listNames()` methods
* Allow `null` in the `remote-directory` for the `<int-ftp:inbound-channel-adapter>`
* Allow `null` in the `expression` for the `FtpOutboundGateway`

Polishing - send error if error on async output

Cover `onFailure()` from `onSuccess()` with the `errorChannel`

Address PR Comments

* Get rid of `null` population for the `remoteDirectoryExpression` in the `AbstractPollingInboundChannelAdapterParser`
* Populate `new LiteralExpression(null)` from the `FtpInboundFileSynchronizer` ctor
* Introduce `buildRemotePath(parent, child)` function in the `AbstractRemoteFileOutboundGateway` with the `null` logic for `parent`
* Rework `mGetWithoutRecursion()` to use `LS` command and allow `null` for the dir.
* Fix tests according the new `mGetWithoutRecursion()` logic

Polishing
2016-02-16 14:45:22 -05:00