Commit Graph

484 Commits

Author SHA1 Message Date
lsgeme
1820c05c26 GH-3105: Fix (S)FTP streaming single filter logic
Fixes https://github.com/spring-projects/spring-integration/issues/3105

The loop in the `AbstractRemoteFileStreamingMessageSource` doesn't check
the next polled file for filtering.

* Add `continue;` when we filter the current file and poll the next one.
So, we go over into the `while()` beginning
* Clean up the code style and cover more code path in test
2019-11-08 11:44:42 -05:00
Joaquin Santana
7aea76c6c4 GH-3026: Fix chmod support for DSL
Fixes https://github.com/spring-projects/spring-integration/issues/3026

**Cherry-pick to `5.1.x`**

* Populate proper `FileTransferringMessageHandler` impl from DSL spec
implementations.
This way we are able to use a provided `chmod` from Java DSL
* Added `FileTransferringMessageHandlerSpec` ctor TODO
* Update SftpTests
* Code cleanup; `@Ignore` `SftpTests.testSftpOutboundFlowWithChmod()`
since it doesn't work properly on Windows
2019-10-29 15:20:08 -04:00
Emmanuel Roux
c420c414d3 GH-3076: NioFileLocker: add file existence check
Fixes spring-projects/spring-integration#3076

* Return `null` instead of throwing an exception

* Fix license header

* Add missing `@since`

* Avoid over-engineered test logic

* Fix code style

* Test non-existent file does not get created

* Fix: remove unused import
2019-10-17 14:52:46 -04:00
Gary Russell
9994997917 GH-3067: Use default NullChannel instance if poss.
Resolves https://github.com/spring-projects/spring-integration/issues/3067

Also add missing receive counter.

* Late binding of null discard channel; checkstyle

* Fix test; fall back to new NullChannel(); always evaluate tx expressions
2019-09-27 17:14:51 -04:00
Artem Bilan
075086775d GH-3038: Delegate more RemoteFileTempalte options
Fixes https://github.com/spring-projects/spring-integration/issues/3038

* Add more delegating setters into the `AbstractRemoteFileOutboundGateway`
for its `RemoteFileTemplate` property
* Restrict externally provided `RemoteFileTemplate` from modifications
in the `AbstractRemoteFileOutboundGateway`: those options must be
configured on that external `RemoteFileTemplate`
* Expose new options in the Java DSL specs
2019-08-29 11:20:52 -04:00
Artem Bilan
a756e6334d GH-3043: Add FileHeaders.REMOTE_HOST header (#3044)
* GH-3043: Add FileHeaders.REMOTE_HOST header

Fixes https://github.com/spring-projects/spring-integration/issues/3043

* Populate a `FileHeaders.REMOTE_HOST` from the
`AbstractRemoteFileStreamingMessageSource` and "get"-based commands
in the `AbstractRemoteFileOutboundGateway`
* Extract the value from the a `Session.getHost()` contract
* The `AbstractInboundFileSynchronizingMessageSource` cannot be
addressed with this because the real message is already based on the
locally stored file
* Adjust some affected tests according our code style requirements

* * Add remote file info support into `AbstractInboundFileSynchronizingMessageSource`
* Introduce a `MetadataStore` functionality into the `AbstractInboundFileSynchronizer`
to gather a remote file info an save it in the URI style against local file
* Retrieve such an info in the `AbstractInboundFileSynchronizingMessageSource`
during local file polling
* Introduce `protocol()` contract for the `AbstractInboundFileSynchronizer`
to build a proper URI in the metadata for external readers to distinguish
remote files properly
* Document the feature

* * Fix some typos in Docs

* * Rename property and header constant to the `HOST_PORT` pair
* Fix typos in Docs
* Add  `remote-file-metadata-store` and `metadata-store-prefix` into XSD
of (S)FTP Inbound Channel Adapters
* Add `remoteFileMetadataStore` and `metadataStorePrefix` options
into `RemoteFileInboundChannelAdapterSpec` for Java DSL
2019-08-28 08:58:13 -04:00
Artem Bilan
a835a63aad Apply changes from 5.1 for StandardRotationPolicy
Related to https://github.com/spring-projects/spring-integration/issues/3027

* Remove `Assert.isTrue()` from the `onRotation()` to make the behavior
as void in case of non-standard `MessageSource` provided
* Add JavaDocs into `onRotation()` to describe the behavior and
possibility to override
2019-08-19 13:34:12 -04:00
Artem Bilan
13a8623b59 GH-3027: Decouple RotationPolicy
Fixes https://github.com/spring-projects/spring-integration/issues/3027

This is a polishing for PR https://github.com/spring-projects/spring-integration/pull/3029

* Move `KeyDirectory` in to the `RotationPolicy` since it looks like fully
coupled concept with that abstraction and the class name is so bad for
the public API
* Remove `AbstractStandardRotationPolicy` in favor of a `StandardRotationPolicy`
extendability
* Change `ftp.adoc` and `sftp.adoc` to reflect a new API reality
* Mantion these changes in the `whats-new.adoc`
2019-08-15 11:32:25 -04:00
Gary Russell
a66b5ed6e2 GH-3027: Fix @since 2019-08-14 14:13:28 -04:00
David Turanski
ba017686fb GH-3027: Enhancements to RotatingServerAdvice
Resolves https://github.com/spring-projects/spring-integration/issues/3027

Fix checkstyle errors

Polishing - wrap code at 120, javadocs at 90; don't use this. on method calls
2019-08-14 13:45:11 -04:00
Gary Russell
0edce21cfe Remove remaining redundant superinterfaces 2019-07-31 11:52:22 -04:00
Gary Russell
cd0f56bc87 Add Apache MINA SftpEventListener
- republish certain events as `ApplicationEvent`s.

* * Add ApacheMinaFtplet to provide the same functionality with FTP
* Fix typo

* * Polishing javadocs and event toString() methods
2019-07-30 10:31:23 -04:00
Artem Bilan
80d679a9b0 GH-2748: More bean definitions into exceptions
Fixes https://github.com/spring-projects/spring-integration/issues/2748

* Refactor more `MessageHandlingException`s to include `this` into an
exception message
* Revert using `MessagingException` in some places which really are not
about messaging.
This helps to wrap them into `MessageHandlingException` later in the
`MessageHandler` for the `BeanDefinition` reference
* Remove `volatile` from configuration properties in the affected
classes
* Remove already deprecated `JmsOutboundGateway.setPriority()`
* Add `resource` and `source` for `BeanDefinition` in the
`AbstractChannelAdapterParser` & `AbstractInboundGatewayParser`
* Document the feature
2019-07-23 15:19:56 -04:00
Gary Russell
ce85b64aa8 Improve RemoteFileTemplate invalid payload except.
- include the invalid payload type
2019-07-23 14:11:07 -04:00
Gary Russell
d2eee913d7 INT-3746: Polishing
- move `InputStream` test later
- add `name` for error messages
- add test
2019-07-23 13:01:08 -04:00
Gary Russell
8bb0e194c0 Clear lock cache in NioFileLocker (#2998)
Fixes https://github.com/spring-projects/spring-integration/issues/2980
2019-07-22 15:10:18 -04:00
Artem Bilan
c109e1df34 GH-2987: Add HTTPS entries into spring.schemas
Fixes https://github.com/spring-projects/spring-integration/issues/2987

To resolve XSD files properly from the classpath, their HTTPS reference
must be present in the `spring.schemas` to avoid the Internet interaction
for resolving an XSD file

**Cherry-pick to 5.1.x, 5.0.x & 4.3.x**
2019-07-18 14:05:18 -04:00
Artem Bilan
315f0e711f Fix Sonar vulnerabilities for varargs
* Fix smell for static `AmqpInboundGateway.attributesHolder`

* Fix readOnlyHeaders in the `MessageBuilder`
2019-05-30 15:41:53 -04:00
Gary Russell
86c7e36667 Sonar issues - complexity
* Fix checkstyle issue.
2019-05-03 20:05:16 -04:00
Gary Russell
0d6faa2e34 Sonar - protected fields 2019-05-01 17:28:42 -04:00
Gary Russell
19b9944dd8 Sonar fixes
* use equals
* inner assignment
* local before return
* unnecessary locals
* catch and throw
* uninstantiable with no statics
* implement `Serializable` in `Comparator` to enable `TreeMap` serialization
* exceptions as flow control
2019-04-30 18:47:43 -04:00
Gary Russell
130e1bba31 Synchronize RemoteFileUtils.makeDirectories()
https://stackoverflow.com/questions/55918648

Avoid the race with concurrent threads.

**cherry-pick to 5.x.x, 4.3.x**
2019-04-30 14:09:45 -04:00
Gary Russell
0bb901b286 Sonar fixes
- printStackTrace in test mail server
- Illegal throws
- Large anon. classes

* - indexOf char
- stored external object

* - ignored exceptional return values

* - checkstyle
2019-04-30 13:57:12 -04:00
Gary Russell
36c33bb5ab Sonar fixes
- a few complexities
- final method calls from ctor
- raw exception throwing
- useless overrides
- loss of stack trace
2019-04-29 18:20:40 -04:00
Artem Bilan
818be4cbe8 GH-2765: Add discardChannel for splitter (#2883)
* GH-2765: Add discardChannel for splitter

Fixes https://github.com/spring-projects/spring-integration/issues/2765

When encountering empty collections, splitter should be able to send
the result to a discard channel.
Currently, when encountering an empty collection,
the splitter ends the flow.
Some use-cases may rely on a custom split function which may returns
empty collections.
These use-cases should be able to define a discard channel
so they can proceed with a possible compensation flow.

* Add `discardChannel` option to the `AbstractMessageSplitter`
* Delegate `discardChannel` population from everywhere it is possible:
DSL, XML, `AbstractMessageSplitter` extension like `FileSplitter` etc.
* Fix `FileSplitterTests` for broken charset
* Document new feature; fix some typos and out-dated code sample

* * Fix `SplitterFactoryBean` for NPE on the `discardChannelName`
propagation

* * Check `this.discardChannel` first
* `Assert.state()` in `doInit()` for mutual exclusiveness
2019-04-08 16:42:57 -04:00
Artem Bilan
d21e8f0349 URL Cleanup
This commit updates URLs to prefer the https protocol.
Redirects are not followed to avoid accidentally expanding intentionally
shortened URLs (i.e. if using a URL shortener).

# Fixed URLs

## Fixed Success
These URLs were switched to an https URL with a 2xx status.
While the status was successful, your review is still recommended.

* [ ] http://www.apache.org/licenses/ with 1 occurrences migrated to:
  https://www.apache.org/licenses/ ([https](https://www.apache.org/licenses/) result 200).
* [ ] http://www.apache.org/licenses/LICENSE-2.0 with 2638 occurrences migrated to:
  https://www.apache.org/licenses/LICENSE-2.0 ([https](https://www.apache.org/licenses/LICENSE-2.0) result 200).
2019-03-21 19:22:26 -04:00
Artem Bilan
20e8f0af2f Fix HTTP urls to HTTPS
* Do not fix urls for the `http://apache.org/xml/features` when we
configure XML components
* Do not fix urls for localhost and fake links in tests
* Fix all the dead links in docs
2019-03-21 17:16:30 -04:00
Ruslan Stelmachenko
0ce6216e49 Fix typo in exception message 2019-03-21 14:04:00 -04:00
Gary Russell
78199dca9b Avoid throws Exception where possible - Phase III 2019-03-07 17:59:53 -05:00
Gary Russell
b187bca36e Avoid throws Exception where possible - Phase I
* Polishing - PR Comments
2019-03-07 12:53:52 -05:00
Gary Russell
efea8eba95 Fix new Sonar issues 2019-03-07 08:54:19 -05:00
Gary Russell
931df86274 GH-2777: Remote File Filter Improvements
Resolves https://github.com/spring-projects/spring-integration/issues/2777

If the filter supports it, defer filtering until the last possible moment.
Then, the worst case scenario after a catastrophic failure (e.g. power loss),
would be that at most one file will be incorrectly filtered on restart.

Polishing and add more tests.

Polishing Javadocs

More Polishing

Final polishing

More polishing.

Polishing and docPolishing and docs.

* Fix typos in Docs
2019-03-05 17:49:56 -05:00
Gary Russell
a167290f9b GH-2776: Fix Streaming Remote File MessageSource
Fixes https://github.com/spring-projects/spring-integration/issues/2776
Also see https://github.com/spring-projects/spring-integration/issues/2777

- reset the filter for the current file if the fetch fails
- implement `Lifecycle` and clear the `toBeReceived` queue and corresponding
  filter entries

* Polishing - PR Comments **cherry-pick to all supported**

* Polishing
2019-03-01 14:35:31 -05:00
Artem Bilan
d2e974a6de Fix new Sonar smells (#2768)
* Fix new Sonar smells

* Fix some old Sonar smells as well
* Fix Micrometer leaks in the `PollableChannel` when we register
meters, but don't remove them.

* * Fix NPE around `MetricsCaptor` in channels

* * Fix new smells according test report

* * Further Sonar smell fixes

* * More smell fixes for `MessagingMethodInvokerHelper`
* Remove `throws Exception` from `AbstractMessageHandler.destroy()`

* * Fix complexity in the `MessagingMethodInvokerHelper.processInvokeExceptionAndFallbackToExpressionIfAny()`
2019-02-27 15:17:51 -05:00
Artem Bilan
221393e02f MessageHandlingException consistency
* Use `IntegrationUtils.wrapInHandlingExceptionIfNecessary()` whenever
it is possible to avoid double wrapping into the `MessageHandlingException`
* Some code polishing for affected classes `@Nullable`, streams, diamonds etc.
2019-02-22 09:43:59 -05:00
Artem Bilan
7e3b56939e Fix XSDs for actual core schema version 2019-02-21 12:43:21 -05:00
Gary Russell
520ebbbd2c Move master to 5.2 2019-02-15 12:01:24 -05:00
Artem Bilan
020ea76d59 Make IOS.setTaskScheduler() as public (#2725)
* Make IOS.setTaskScheduler() as public

* Make `IntegrationObjectSupport.setTaskScheduler()` as `public` and
remove all the overrides for visibility
* Fix Sonar smells for all the affected classes

* * Fix `throws Exception` for affected classes to avoid compilation errors

* * Fix "merely rethrow" smell
* Revert `throws Exception` for `AbstractEndpoint.destroy()`

* * Catch a couple exceptions from `destroy()`
2019-01-30 13:59:37 -05:00
Artem Bilan
a609bd9398 Some file module Sonar fixes (#2718)
* Some file module Sonar fixes

* * Fixes after testing

* * Some further Sonar fixes for file module

* * Further Sonar fixes for file module

* * More Sonar fixes for file module

* * Fix `RemoteFileTemplate` issues

* * Fix new sonar smells after previous one

* * Review feedback
2019-01-25 14:37:57 -05:00
Gary Russell
4ea3b1c23c GH-2705: public CTORs on AMFPFileListFilter Impls
Fixes https://github.com/spring-projects/spring-integration/issues/2705
2019-01-18 11:12:38 -05:00
Artem Bilan
1d5e66b0ee RemoteFileTemplate: Fix Checkstyle violation
* Add `if (session != null)` in the `RemoteFileTemplate` around `dirty()`
2019-01-10 11:05:18 -05:00
Alen Turkovic
6ddcfb3a8d Custom Session dirty logic
Fixes spring-projects/spring-integration#2687

Allows any session to implement its dirty logic.

* License year was not updated

**Cherry-pick to 5.0.x**
2019-01-10 10:52:45 -05:00
Gary Russell
e40cfe101e Sonar fixes
Hidden fields, `o.s.i.a*` through `o.s.i.j*`.
2019-01-09 10:02:19 -05:00
Gary Russell
1bafe89d49 Sonar Fixes
- avoid parameter assignments
2019-01-07 11:35:58 -05:00
Gary Russell
e8df546b04 Sonar Fixes - Javadoc-only imports 2019-01-02 12:26:22 -05:00
Gary Russell
113a371f2c Remove Exception from IOS.onInit() 2018-12-22 17:25:56 -05:00
Artem Bilan
81b4ea1bef Sonar fixes according latest report (#2676)
* Sonar fixes according latest report

* Fix initialization order in the `GatewayMethodInboundMessageMapper`

* Fix mock in the `DelegatingSessionFactoryTests`

* Fix `AbstractRemoteFileOutboundGateway.listFilesInRemoteDir`

* * PR comments

* * Fix `AbstractRemoteFileOutboundGateway.listFilesInRemoteDir` complexity
2018-12-22 11:57:35 -05:00
Artem Bilan
93d7c58b64 Enable ModifierOrderCheck Checkstyle rule (#2673)
* Enable ModifierOrderCheck Checkstyle rule

* Fix violations for `static` and `abstract` modifier
* Remove redundant code in the `TcpNioConnection`
* Mark `connectionFactoryName` as `@Nullable` in the `TcpConnectionSupport`
ctor and its inheritors
* Fix some smells according IDEA suggestions in the affected classes
* This should fix some Sonar smells as well

* * Fix `HeaderMapperTests`

* * Polishing `TcpConnection` code style and fix Javdocs
2018-12-20 19:19:47 -05:00
Gary Russell
62fc7df693 Sonar Fixes
Critical smells, packages `o.s.i.a*` to `f*`.

Plus new smells caused by these changes.
2018-12-03 10:44:26 -05:00
Gary Russell
39fef2c70a INT-4560: Fix Race in FileSystemPersistentAOFLF
JIRA: https://jira.spring.io/browse/INT-4560

Reproduced and tested with

```java
@SpringBootApplication
public class So53521593Application {

	private static final Logger logger = LoggerFactory.getLogger(So53521593Application.class);

	public static void main(String[] args) {
		SpringApplication.run(So53521593Application.class, args);
	}

	@Bean
	public IntegrationFlow flow() {
		ExecutorService exec = Executors.newFixedThreadPool(10);
		return IntegrationFlows.from(Files.inboundAdapter(new File("/tmp/foo")).filter(
					new MyFilter(new SimpleMetadataStore(), "foo")),
						e -> e.poller(Pollers.fixedDelay(5, TimeUnit.SECONDS)
								.maxMessagesPerPoll(10)))
				.channel(MessageChannels.executor(exec))
				.<File>handle((p, h) -> {
					try {
						p.delete();
						logger.info(p.toString());
						Thread.sleep(10_000);
					}
					catch (InterruptedException e1) {
						Thread.currentThread().interrupt();
					}
					return null;
				})
				.get();
	}
}

class MyFilter extends FileSystemPersistentAcceptOnceFileListFilter {

	public MyFilter(ConcurrentMetadataStore store, String prefix) {
		super(store, prefix);
	}

	@Override
	protected long modified(File file) {
		long modified = super.modified(file);
		System.out.println(modified);
		return modified;
	}

}
```

**cherry-pick to 5.0.x, 4.3.x**
2018-11-28 14:18:09 -05:00