Commit Graph

9623 Commits

Author SHA1 Message Date
Artem Bilan
6ecd948a32 Close file resource after (S)FTP streaming tests
Windows is too scrupulous for non closed file resources.
Our `RemoteFileTestSupport` recreates a test directory with files
for each test.
When we don't close some file handler, we are not able to delete the
directory and we fail with subsequent tests.

* Close `InputStream` for each polled file in new tests in the
`SftpStreamingMessageSourceTests`

* Close `CLOSEABLE_RESOURCE` in new tests in the
`FtpStreamingMessageSourceTests`
* Rework `FtpStreamingMessageSourceTests.testAllContents()` do not
poll all the files on each polling cycle - this causes a race condition
when we don't close `CLOSEABLE_RESOURCE` yet in the `StreamTransformer`,
but try to proceed with recreation a test directory structure

**Cherry-pick to 5.0.x**
2018-06-29 21:28:28 -04:00
Gary Russell
6affb3bb2f INT-4495: Fix SFTP tests 2018-06-29 14:59:26 -04:00
Gary Russell
3f218a97a0 INT-4495: (S)FTP: fix max-fetch with directories
JIRA: https://jira.spring.io/browse/INT-4495

Previously, `maxFetch` was applied before directories were removed from
the fetch list.

Remove the directories before filtering and applying `maxFetch`.

* Polishing - PR Comments

* More polishing

* More polishing.

* Check for empty array.

* Remove test main method.
2018-06-29 14:21:06 -04:00
Artem Bilan
e83b472a37 RotatingSTests: Don't delete files between polls
On Windows `File.delete()` causes to file to be modified first of all.
with the high polling interval (`fixedDelay(1)`) there is a chance
to pick the same file from the local directory again, since the
`AbstractPersistentAcceptOnceFileListFilter` is based on the `lastModified`

* Do not delete files manually at all - just rely on the `clean()`
with the `@Before`

**Cherry-pick to 5.0.x**
2018-06-27 16:32:55 -04:00
Gary Russell
26cfab579c INT-4491: (S)FTP inbound rotate dirs/servers
JIRA: https://jira.spring.io/browse/INT-4491

Add Rotating Server/Directory Polling Advice.

**cherry-pick to 5.0.x**

* Polishing - PR Comments.

* Polishing

* Polishing; revert `KeyDirectory`; WARN about `TaskExecutor` and `MessageSoureMutator`(s).

* More polishing - PR comments

* Apply stashed changes.

* Fix WARN log - the `SyncTaskExecutor` is wrapped.
2018-06-27 13:43:41 -04:00
Gary Russell
586afc9901 INT-3916: ZK: Fix missing doc update
JIRA: https://jira.spring.io/browse/INT-3916

Changed CTOR injection to setters.
2018-06-27 12:26:55 -04:00
Artem Bilan
21fff59112 Fix compatibility with the latest Spring AMQP
Related to the https://jira.spring.io/browse/AMQP-820
2018-06-27 12:09:05 -04:00
Artem Bilan
69af967c78 INT-4488: Require @EnablePublisher
JIRA: https://jira.spring.io/browse/INT-4488

Since `PublisherAnnotationBeanPostProcessor` is quite expensive
(1.5x the CPU used by `MessagingAnnotationPostProcessor` in a simple
application)

* So not register a `PublisherAnnotationBeanPostProcessor` by default.
The `@EnablePublisher` has to be presented on the `@Configuration`
to allow the `@Publisher` AOP
* Change the `<int:annotation-config>` to require a new
`<int:enable-publisher>` sub-element for similar purpose - do not
register `PublisherAnnotationBeanPostProcessor` by default
2018-06-26 16:00:45 -04:00
Artem Bilan
13510df575 INT-4452: Expire immediately when group timeout<0
JIRA: https://jira.spring.io/browse/INT-4452

This is pretty typical in practice to get a `groupTimeout` to be
evaluated to a negative value: some business decisions, sensitive
calculations and so on.

* Treat any non-positive `groupTimeout` as a signal to expire group
immediately without scheduling.
Only `null` is considered as a signal do nothing for the current message
* Polishing for some tests for better performance - saves some execution
time
2018-06-26 12:41:48 -04:00
Artem Bilan
ee501c801f INT-4436: Register HTTP endpoints at runtime
JIRA: https://jira.spring.io/browse/INT-4436

* Fix `IntegrationRequestMappingHandlerMapping` and
`IntegrationRequestMappingHandlerMapping` to implement a
`DestructionAwareBeanPostProcessor` to allow to register and
destroy HTTP endpoints at runtime, e.g. via `IntegrationFlowContext`
with the dynamic `IntegrationFlow`s
2018-06-25 17:11:23 -04:00
Artem Bilan
07c32ae93b INT-4489: MessageHistory: Allow runtime beans
JIRA: https://jira.spring.io/browse/INT-4489

Make `MessageHistoryConfigurer` as a `BeanPostProcessor` and apply
tracking logic to processed beans if it is already started.

**Cherry-pick to 5.0.x**
2018-06-22 13:55:20 -04:00
Artem Bilan
e6a35c4a60 INT-4487: Allow reconfiguring ARPMH.adviceChain
JIRA: https://jira.spring.io/browse/INT-4487

Currently the `AbstractReplyProducingMessageHandler` applies advices
only in the `afterPropertiesSet()` phase.
When we use bean references for `MessageHandler`s in Java DSL, the
`.advice()` provided in the flow definition is not applied to the
`AbstractReplyProducingMessageHandler`.

* For consistency with the endpoint configured by the DSL definition,
override and reconfigure advices in the `AbstractReplyProducingMessageHandler`
bean reference during DSL parsing and processing.

Fix `TwitterSearchOutboundGatewayParserTests` for the proper assert
2018-06-22 13:06:18 -04:00
Gary Russell
0735ddb0ed INT-4390: Fix tangle betweewn dsl and dsl.context
JIRA: https://jira.spring.io/browse/INT-4390

Move IFBPP and lifecycle advice to `....dsl.context`.
2018-06-20 12:33:37 -04:00
Gary Russell
bf10821d32 Add hack for Atom asciidoc syntax highlighting bug
https://github.com/atom/language-java/issues/88
2018-06-19 14:52:44 -04:00
Artem Bilan
32030c3233 INT-2260: JdbcPollingChA: rename prop to maxRows
JIRA: https://jira.spring.io/browse/INT-2260

Having a feedback about confusing with the `max-rows-per-poll` property
name and its responsibility it would be better do not mention `per-poll`
at all

* Deprecate `max-rows-per-poll` in favor of new `max-rows`
* Some code style polishing, tests improvements
* Docs polishing on the matter

* Add `What's New` bullet

* Optimize `maxRows` logic
* Document vendor-specific native SELECT limiting options
* Raise warning in the parsers about deprecated `max-rows-per-poll`

Doc polishing
2018-06-19 12:13:39 -04:00
Artem Bilan
5a362b62ff Destroy RmiInboundGateway.RmiServiceExporter (#2481)
* Destroy RmiInboundGateway.RmiServiceExporter

The internal instance `RmiServiceExporter` of the `RmiInboundGateway`
has to be destroyed together with the outer instance to unbind `service`
from the RMI registry

* Perform some polishing for the `RmiInboundGateway` and optimize
a `AbstractInboundGatewayParser` for late channels binding

* * Polishing some tests
* Expose getters for channels on the `MessagingGatewaySupport`
* Fix RMI tests to use random port
2018-06-15 08:40:11 -04:00
Artem Bilan
ad4dc11758 GH-2478: RecursiveDScanner: Files.walk().close()
Fixes https://github.com/spring-projects/spring-integration/issues/2478

The `Files.walk()` stream must be closed in the end after usage

* Wrap `Files.walk()` to the `try-with-resources`

**Cherry-pick to 5.0.x**
2018-06-13 12:33:16 -04:00
Artem Bilan
2267f72e5b INT-4484: Fix SyndEntryPublishedDateComparator
JIRA: https://jira.spring.io/browse/INT-4484

Fix the `SyndEntryPublishedDateComparator` to move entries without a
date into the end of the sorted list

**Cherry-pick to 5.0.x and 4.3.x**
2018-06-13 12:25:40 -04:00
Gary Russell
a954290561 Fix docs isClientModeConnected() 2018-06-12 11:43:46 -04:00
Artem Bilan
da53ed6838 INT-4485: Don't register bean twice for DSL Specs
JIRA: https://jira.spring.io/browse/INT-4485

The `IntegrationFlowBeanPostProcessor` performs extra bean registration
for the `IntegrationComponentSpec.get()` result.
Essentially it is going to be the same object in the end, but during
bean registration phase we end up with names conflict.

* Remove an explicit `registerComponent()` for the
`IntegrationComponentSpec.get()`
* Modify `CorrelationHandlerTests` for all possible usage for the
`MessageChannelSpec`, which is essentially a `FactoryBean`

**Cherry-pick to 5.0.x**
2018-06-12 10:11:41 -04:00
Artem Bilan
85bc183849 INT-4486: Properly implement stop(Runnable)
JIRA: https://jira.spring.io/browse/INT-4486

The `SmartLifecycle.stop(Runnable callback)` must always call the
`callback` in the end independently of the internal state

* Revise all the `SmartLifecycle` implementations for the proper
`callback` handling

**Cherry-pick to 5.0.x and 4.3.x**
2018-06-11 17:51:20 -04:00
Artem Bilan
eb1efdb0ff No JSONP WebSocket transport any more
https://build.spring.io/browse/INT-MASTER-1070/

See: https://jira.spring.io/browse/SPR-16914
2018-06-11 17:11:53 -04:00
Artem Bilan
0fc8bec645 GH-2464: WebFlux: Get rid of Mono.block() (#2465)
* GH-2464: WebFlux: Get rid of Mono.block()

Fixes https://github.com/spring-projects/spring-integration/issues/2464

The `WebFluxInboundEndpoint` resolves a `Principal` via `Mono.block()`
operation.
This is prohibited situation in the non-blocking thread, like Reactor
Netty

* Defer `Mono<Principal>` resolution to the message header via
deferring the whole `Message` creating via `flatMap()` operation on the
main `doHandle()` `Mono`

**Cherry-pick to 5.0.x**

* * Fix Checkstyle for the Reactive Spring Security testing utils static imports
* Add `defaultIfEmpty()`, when `exchange.getPrincipal()` is an empty `Mono`
2018-06-11 16:34:04 -04:00
brempusz
4bb1a619a9 GH-2467: JdbcLockReg: retry on TransDataAccessExc
* GH-2467: JdbcLockRegistry should retry on DeadlockLoserDataAccessException

Fixes #2467

MySQL 5.7.15 introduced setting `innodb_deadlock_detect` (enabled by
default). As a result MySQL JDBC driver throws
`DeadlockLoserDataAccessException` when deadlock is detected.
`JdbcLockRegistry` doesn't handle it causing lock to be lost.

* Retry `doLock()` on data access deadlock instead of loosing the lock

* Use TransientDataAccessException instead of derived exceptions
2018-06-07 12:56:33 -04:00
Gary Russell
f9cea64a79 GH-2471: MQTT: Fix Thread Leak
Fixes https://github.com/spring-projects/spring-integration/issues/2471

Call `close()` on the client whenever the connection is lost or can't be
established, to release resources in the client.

**cherry-pick to 5.0.x, 4.3.x**
2018-06-06 12:32:17 -04:00
Gary Russell
88120a667e INT-4482: AMQP: Fix Double ErrorMessage
JIRA: https://jira.spring.io/browse/INT-4482

The outer try/catch sends an `ErrorMessage` for all exceptions; it should
only do so for `MessageConversionException`. Integration flow exceptions
will have been already handled by `MessageProducerSupport`.

Also, populate the raw message header consistently - previously it only
was populated for flow exceptions. Although the LEFE contains the raw
message, it should be in the `ErrorMessage` header for consistency.

**cherry-pick to 5.0.x, 4.3.x**

* Polishing - PR Comments
2018-06-05 16:34:58 -04:00
Gary Russell
bb5528c761 INT-4481: Fix notice.txt line delimiters
CRLF->LF
2018-06-05 13:07:47 -04:00
Artem Bilan
89f902885a INT-4481: Add licence and notice to each artifact
JIRA: https://jira.spring.io/browse/INT-4481

* Now each generated jar has `licence.txt` and `notice.txt` entries in
its `META-INF`
* Copy `notice.txt` from the Spring Framework with an appropriate
polishing
2018-06-05 13:04:12 -04:00
Artem Bilan
f4809aa997 INT-4424: No @Inherited on method annotations
JIRA: https://jira.spring.io/browse/INT-4424

* Polishing for `phase` JavaDocs
2018-06-05 09:44:49 -04:00
Gary Russell
35c7ef1131 INT-4387: Fix incorrect contentType mapping
JIRA: https://jira.spring.io/browse/INT-4387

Don't put a copy in the message properties headers map.
2018-06-01 12:45:01 -04:00
Artem Bilan
94cded7575 Remove dependant beans recursively
https://build.spring.io/browse/INT-MASTER-1058/

When we destroy manually registered `IntegrationFlow`, we need to
count with sub-flows and iterate all the dependant beans recursively
for full flow removal

**Cherry-pick to 5.0.x**
2018-06-01 09:45:30 -04:00
Gary Russell
997b07afed INT-4476: Fall back for ID and Timestamp Headers
JIRA: https://jira.spring.io/browse/INT-4476

When mapping outbound headers, if the `AmqpHeaders` `ID` and `Timestamp` headers
are not present, fall back to mapping the `MessageHeaders` variants (if present).

Also fix some PDF overflows.
2018-05-31 17:18:39 -04:00
Artem Bilan
4c0f767ff3 Upgrade to Reactor-Netty-0.8
https://build.spring.io/browse/INT-MASTER-1059/
https://github.com/spring-projects/spring-framework/pull/1845
2018-05-31 17:13:43 -04:00
Artem Bilan
03a89337cb INT-4474: ConsumerEndpointFactoryBean: Fix phase
JIRA: https://jira.spring.io/browse/INT-4474

* Populate proper `Integer.MIN_VALUE` phase for non-`PollingConsumer`s
in the `ConsumerEndpointFactoryBean`

**Cherry-pick to 5.0.x and 4.3.x**
2018-05-31 16:50:34 -04:00
Artem Bilan
17e794d2c8 INT-4471: PubSubChannel: Add errorHandler warn (#2459)
* INT-4471: PubSubChannel: Add errorHandler warn

JIRA: https://jira.spring.io/browse/INT-4471

* When an `Executor` is not provided, log warn that the provided
`ErrorHandler` is ignored.

**Cherry-pick to 5.0.x and 4.3.x**

* * Polish warn message
2018-05-31 15:25:38 -04:00
Artem Bilan
9a88140fb3 Fix LambdaMessageProcessor for Map payload
JIRA: https://jira.spring.io/browse/INT-4478

When we use a `GenericHandler` and an incoming payload is a `Map`, we
copy it into both arguments into the `Object` for the payload
and `Map` for the headers.
This way we lose headers in the target lambda

* Check a size of arguments on the target lambda and don't set a
payload into the `Map` argument if we have more than 1 arguments

**Cherry-pick to 5.0.x**
2018-05-31 15:18:44 -04:00
Artem Bilan
af8973345a Fix ZK LeaderInitiator Checkstyle violation
**Cherry-pick to 5.0.x**
2018-05-31 11:34:40 -04:00
Gary Russell
73dbdb80be INT-4477: Add getRole() to LeaderInitiator Context
JIRA: https://jira.spring.io/browse/INT-4477

When running multiple roles, it is useful to be able to determine the
role for a particular context, e.g. in an event.

* Docs

**cherry-pick to 5.0.x**
2018-05-31 10:54:35 -04:00
Artem Bilan
c9faf3c8ce Fix LockRegLeaderIn for interrupted Thread.sleep (#2455)
* Fix LockRegLeaderIn for interrupted Thread.sleep

https://build.spring.io/browse/INT-FATS5IC-517

If current thread is interrupted, the `Thread.sleep()` interrupts
immediately.
In the catch block of the main loop in the `LockRegistryLeaderInitiator`
we have such a dangerous `sleep()` and don't restart election in this
candidate any more

* Move `Thread.sleep()` to else after checking the current thread for
interrupted state
* Remove `LongRunningIntegrationTest` rule from the
`RedisLockRegistryLeaderInitiatorTests` since it now works much faster
after proper `busy-wait` handling

**Cherry-pick to master**

* Ignore interruption on the sleep i catch and move on with loop
2018-05-30 15:23:26 -04:00
Gary Russell
7ab024df88 Fix broken links in reference appendices 2018-05-28 14:40:34 -04:00
Gary Russell
324c54d8ef Remove use of deprecated ChannelInterceptorAdapter
Deprecated in favor of default methods on `ChannelInterceptor`.

* Fix race in RouterConcurrencyTest; no assertion on executor shutdown; increase time.
2018-05-24 15:11:11 -04:00
Artem Bilan
b087fddd4d Fix CorrelationHandlerTests compilation warnings
https://build.spring.io/browse/INT-MASTER-JOB1-1050
2018-05-22 14:41:46 -04:00
Artem Bilan
c7aa618328 Ignore BFTypeConverterTests.initialConcurrency()
https://build.spring.io/browse/INT-MASTER-1049/

Since this test is very sensitive for the time, `@Ignore` it
2018-05-22 14:31:10 -04:00
Gary Russell
97d08d9720 INT-4473: Support prefix bean names with flow id
JIRA: https://jira.spring.io/browse/INT-4473

Previously, dynamic registration of integration flows with components configured
with the same `id` would fail with duplicate bean names.

Add `useFlowIdAsPrefix()` to the registration builder to enable the option.
Then, in the BPP, check the flag before naming the beans.

**cherry-pick to 5.0.x**

* Polishing according PR comments
* Widen `flowNamePrefix` responsibility in the
`IntegrationFlowBeanPostProcessor` since we may have many other
components in the dynamic flow with the same id, not only consumer
endpoints
2018-05-22 13:51:13 -04:00
Artem Bilan
8a61e16413 Increase response timeout for WebSocketClientTests 2018-05-21 15:22:04 -04:00
Artem Bilan
b21dc0dff3 Fix IntegrationFlowContext concurrency issue
When we register `IntegrationFlow` s concurrently at runtime, we may
end up with the problem when we register the same object with the same
bean name, but in different places.
Or when we turn off bean overriding, we end up with the exception that
bean with the name already registered

* Wrap `IntegrationFlow` bean registration in the
`StandardIntegrationFlowContext` into the `Lock` when its bean name
is generating
* Make `StandardIntegrationFlowContext.registry` as `ConcurrentHashMap`
to avoid `ConcurrentModificationException` during `put()` and `remove()`
* Fix concurrency for beans registration with the generation names in
the `IntegrationFlowBeanPostProcessor` using an `IntegrationFlow` id
as a prefix for uniqueness.

**Cherry-pick to 5.0.x**

Fix generated bean name in the WebFluxDslTests

Use only single `Lock` in the `StandardIntegrationFlowContext`:
we don't need a fully blown `LockRegistry` there anymore since we have
only one synchronization block there and it is always around the same
type
* Add `What's New` note, and mention changes in the `dsl.adoc`

Minor doc polishing.
2018-05-21 13:05:45 -04:00
Artem Bilan
a750a7847c Upgrade to Reactor-Netty-0.7.8 2018-05-18 15:58:25 -04:00
Artem Bilan
096bec5627 Fix compatibility with latest Spring Data Geode
* Some configs and code style polishing in the Gemfire module tests
and in the `DefaultHeaderChannelRegistry`
2018-05-18 15:14:08 -04:00
Artem Bilan
1f05d3ec4c Polishing some Docs and JavaDocs
* Mentioned `byte[]` support for the `UnmarshallingTransformer`
* Fix `AbstractInboundFileSynchronizingMessageSource` JavaDocs to
mention the current `FileSystemPersistentAcceptOnceFileListFilter`
usage by default, not an `AcceptOnceFileListFilter`
* Polishing `AbstractInboundFileSynchronizingMessageSource` code style
a bit
2018-05-16 16:10:50 -04:00
Artem Bilan
7113236e24 Fix DelayHandlerTests timing issue
https://build.spring.io/browse/INT-MASTER-1042

The `verifyShutdownWithWait()` test uses extra thread for waiting for
the `TaskScheduler` to finish its tasks.
Also the test doesn't verify the actual behavior at all

* Move `awaitTermination()` to the assert
* Decrease `delay` for the message
* Assert the latch from the handler to be sure that message is scheduled
properly even if `destroy()` call, thanks to required
`waitForTasksToCompleteOnShutdown = true`
2018-05-16 15:17:01 -04:00