Honour the `SCS_KAFKA_TEST_EMBEDDED` flag when set as an environment variable
or Java property: when explicitly set to `false`, use an external broker and observe the
`SCS_EXTERNAL_SERVERS_REQUIRED` property (i.e. fail if server is missing). By default,
continue to run tests using an embedded broker.
Fixes#498
- Honour the retry settings from ConsumerProperties;
- Add an additional `enableDlq` option in KafkaConsumerProperties that enables forwarding failed messages to a DLQ topic.
- Change the Binder interface to support ConsumerProperties/ProducerProperties beans and subclasses
- Binders can subclass the property beans to add new supported properties that will be automatically populated
- Spring Cloud Stream will infer the target type and populate the beans from the environment based on a `spring.cloud.stream.bindings..<bindingName>` prefix
- Remove binder defaults and retain only general binder configurations
TODO: a) decide on instanceIndex/partitionIndex alignment (we do not need both)
b) support `defaultProducer`/`defaultConsumer` properties
c) add leniency control on binding (fail/ignore for unknown properties)
d) add a `requiredProperties` configuration for consumer/producer properties to finely tune the mandatory properties expected to be supported by a bound application
Changes made during review:
- Add support for consumer and producer defaults
- Remove partitionIndex, keeping only instanceIndex
- Fix default properties for Kafka binder
- Move batching properties to Rabbit only
- When deserialising the message, replace `contentType` with the `originalContentType` only if the `originalContentType` isn't null which means the original content type was set by the producer before serialising.
- Add and fix tests
This resolves#404
- When deserializing the payload at the consumer endpoint, the non-byte stream payload type requires to use `String` object when the underlying message content-type is of any `text` type contentType (text/plain, text/xml and text/html).
- This fix is only needed to support any non-SCSt applications that will have the 'text/*` contentType of the message that is being deserialized
- Add test
This resolves#403
- Since the `originalContentType` header is preserved across the wire while the serialization/de-serialization processes take place using the deduced `contentType`,
it is convenient to set the `String` value of the `originalContentType` instead of setting the actual `MimeType` object in the header.
- Once the `de-serialization` has taken place, the deduced `contentType` is always replaced with the `originalContentType` as the `contentType` in the message header and this preserved contentType can be resolved to its corresponding MimeType using `ContentTypeResolver` anytime. Hence, using the `String` object for `originalContentType` doesn't do any harm.
- Fix the issue while setting the deduced contentType based on the `payload` when the `originalContentType` is `application/json`
- Currently, the contentType is set to `text/plain` which seems incorrect
- Also, add the `application/json` contentType to return `String` object during de-serialization process.
This resolves#383
Use `text/plain` contentType when the payload is of type `String`
- This is for the intermediate contentType that is used for serialization/de-serialization
- The `spring-cloud-stream-tuple` is now split into `spring-tuple`, `spring-integration-tuple` along with `spring-batch-tuple`
- Update dependencies to use the new ones
This resolves#326
Resolves#317
Remove the `durable` binder configuration property
Make subscriber groups durable by default
Introduce `requiredGroups` property
Kafka groups (non-anonymous) now start by default at EARLIEST, which is more appropriate for new stream consumers
Addressing PR comments
Resolves#330
- removes `unbind` from the binder and moves it to the `Binding` instance itself;
- `Binding` is now an interface with a default implementation provided by SCS
- Removed all methods from Binder except for unbind()
- Removed old and unused code
- Removed circular reference between default Binder and Binding implementations
- Removed Binder type
Fixesspring-cloud/spring-cloud-stream#314
Give more time to redis on travis
Make multipliers more versatile, apply to kafka
Extract receive() method. Default is now 1s
bump multiplier for kafka on travis
* removed pub/sub methods from binder
* consumer group is now a parameter of the remaining bindConsumer method
* remove DynamicProducer from Binder
* Move logic to create the dynamic channel to the channel resolver.
* Return bindings from bind methods and use them for unbinding
* Suffix for dlq
Move All Rabbit Binder CleanUp to Test Bindera
More RabbitMQ Binder Test Cleanup
Clean up declarations for remaining tests.
removed BinderUtils
use Redis ZSET for consumer groups
copyright dates
AutoBindDLQ: Single DLQ Per Group When Partitioned
Configure a single DLQ for each group for all partitions.
Add DLX Exchange binding for each original queue routing key, including the partition.
Fix DLQ Binding (Producer Side)
Option was not allowed and the routing key was wrong.
Add test to verify producers can be bound before consumers.
`autoBindDLQ` must be set (or reset) on both sides for success.
- Per-binding properties defined in `BindingProperties` need to hold the binder specific configurations needed when binding producer/consumers.
Previously, these were added in `ChannelBindingServiceProperties` as producer/consumer properties. Since these configuration properties are for
per-binding based on the input/output (consumer/producer) channel being bound it is better to move these properties into `BindingProperties`.
- Add some of the supported producer/consumer properties into BindingProperties
- This fixes#256
Rename CommonBinderProperties -> BinderPropertyKeys
Rename class AbstractBinderPropertiesAccessor.java -> AbstractBindingPropertiesAccessor.java
- instead of publishing to the default exchange, producers now publish to a topic exchange, to which the main queue is bound
- taps on a stream simply bind a queue to the same exchange with a name including the group for the tapping stream and a
routing key pattern `#`
- this means that multiple instances of the same tap stream will compete but multiple tap streams will get a copy
- for partitioned data, a single topic exchange is used, which each partition queue bound with its name as the routing key
- this means taps on partitioned data see the consolidated messages
Remove unused binder utilities.
Use TopicExchange Always
Use a TopicExchange when binding a pubSub producer for consistency.
Remove Deprecated Binder Method
Add group to BindingProperties
- also fix artifact name in test support pom
- also fix some kafka tests (does not imply that kafka taps work - yet)
Fixes GH-166 (https://github.com/spring-cloud/spring-cloud-stream/issues/166)
Since `onInit()` in the `MessageChannelBinderSupport` was called before the `evaluationContext` population,
the `RedisMessageChannelBinder#errorAdapter` caused `NPE` for its `evaluationContext`.
* Change the initialization order
* Use `ExpressionUtils` to populate `evaluationContext` - creates `StandardEvaluationContext` if there is no one in the `applicationContext`
* Add more initializations for the `RedisMessageChannelBinder#errorAdapter`
* Remove explicit `evaluationContext` population from the `AbstractTestBinder` to rely on the fix for this ticket
* Some other simple polishing
Address PR comments
