Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ build succeed, please raise a ticket to get the settings added to
source control.

The projects that require middleware (i.e. Redis) for testing generally
require that a local instance of [Docker](https://www.docker.com/get-started) is installed and running.
require that a local instance of https://www.docker.com/get-started[Docker] is installed and running.


=== Documentation
Expand Down Expand Up @@ -296,10 +296,11 @@ If you need to suppress some rules (e.g. line length needs to be longer), then i

It's advisable to copy the `${spring-cloud-build.rootFolder}/.editorconfig` and `${spring-cloud-build.rootFolder}/.springformat` to your project. That way, some default formatting rules will be applied. You can do so by running this script:

```bash
[source,bash]
----
$ curl https://raw.githubusercontent.com/spring-cloud/spring-cloud-build/master/.editorconfig -o .editorconfig
$ touch .springformat
```
----

=== IDE setup

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,12 @@ AdminClientConfigCustomizer's configure method provides access to the admin clie
Binder's Kafka topic provisioner gives the highest precedence for the properties given through this customizer.
Here is an example of providing this customizer bean.

```
[source,java]
----
@Bean
public AdminClientConfigCustomizer adminClientConfigCustomizer() {
return props -> {
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
};
}
```
----
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,11 @@ The size of the batch is controlled by Kafka consumer properties `max.poll.recor

When receiving the batches, the following type signatures are allowed:

```
[source,java]
----
List<Person>
Message<List<Person>>
```
----

In the first option of `List<Person>`, the listener will not get any message headers.
If the second type signature (`Message<List<Person>>`) is used, then the headers can be accessed; however, all the headers are still be in the form of a `Collection`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ The custom implementation must be present in the application configuration as a
When the binder discovers the custom implementation, it will use that instead of the default implementation.
Here is an example of such a custom implementation bean in the application.

```
[source,java]
----
@Bean
public KafkaBinderHealth kafkaBinderHealthIndicator() {
return new KafkaBinderHealth() {
Expand All @@ -22,7 +23,7 @@ public KafkaBinderHealth kafkaBinderHealthIndicator() {
}
};
}
```
----

[[custom-kafka-binder-health-indicator-example]]
== Custom kafka Binder Health Indicator Example
Expand All @@ -33,7 +34,8 @@ In this example, we try to override the binder provided Kafka HealthIndicator by

First, we need create a custom implementation of the `KafkaBinderHealth` interface.

```
[source,java]
----
public class KafkaBinderHealthImplementation implements KafkaBinderHealth {
@Value("${spring.cloud.bus.destination}")
private String topic;
Expand Down Expand Up @@ -66,16 +68,17 @@ public class KafkaBinderHealthImplementation implements KafkaBinderHealth {
// Your implementation
}
}
```
----

Then we need to create a bean for the custom implementation.

```
[source,java]
----
@Configuration
public class KafkaBinderHealthIndicatorConfiguration {
@Bean
public KafkaBinderHealth kafkaBinderHealthIndicator(final KafkaAdmin admin) {
return new KafkaBinderHealthImplementation(admin);
}
}
```
----
5 changes: 3 additions & 2 deletions docs/modules/ROOT/pages/kafka/kafka-binder/metrics.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,15 @@ refer to the <<kafka-binder-properties, kafka binder properties section>> for mo

You can exclude `KafkaBinderMetrics` from creating the necessary infrastructure like consumers and then reporting the metrics by providing the following component in the application.

```
[source,java]
----
@Component
class NoOpBindingMeters {
NoOpBindingMeters(MeterRegistry registry) {
registry.config().meterFilter(
MeterFilter.denyNameStartsWith(KafkaBinderMetrics.OFFSET_LAG_METRIC_NAME));
}
}
```
----

More details on how to suppress meters selectively can be found https://micrometer.io/docs/concepts#_meter_filters[here].
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,8 @@ See this https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAA

For example, if you have two clusters in your application with separate JAAS configuration, then the following is a template that you can use:

```
[source,yaml]
----
spring.cloud.stream:
binders:
kafka1:
Expand All @@ -121,7 +122,7 @@ spring.cloud.stream:
configuration:
security.protocol: SASL_PLAINTEXT
sasl.mechanism: PLAIN
```
----

Note that both the Kafka clusters, and the `sasl.jaas.config` values for each of them are different in the above configuration.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@ This could be useful when you are consuming records from a partitioned topic.
For example, assume that the incoming topic has at least three partitions.
Then you can set the following property.

```
[source,properties]
----
spring.cloud.stream.bindings.lowercase-in-0.consumer.concurrency=3
```
----

That will create three dedicated `KafkaReceiver` objects that generate three separate `Flux` implementations and then stream them to the handler method.
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,10 @@ RecordMessageConverter fullRawReceivedRecord() {
Then, you need to instruct the framework to use this converter for the required binding.
Here is an example based on our `lowercase` function.

```
spring.cloud.stream.kafka.bindings.lowercase-in-0.consumer.converterBeanName=fullRawReceivedRecord"
```
[source,properties]
----
spring.cloud.stream.kafka.bindings.lowercase-in-0.consumer.converterBeanName=fullRawReceivedRecord
----

`lowercase-in-0` is the input binding name for our `lowercase` function.
For the outbound (`lowercase-out-0`), we still use the regular `MessagingMessageConverter`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@ This section describes how Micrometer-based observability is enabled in the reac
There is built-in support for observability in producer binding.
To enable it, set the following property:

```
[source,properties]
----
spring.cloud.stream.kafka.binder.enable-observation
```
----

When this property is set to `true`, you can observe the publishing of records.
Both publishing records using `StreamBridge` and regular `Supplier<?>` beans can be observed.
Expand All @@ -28,7 +29,8 @@ In the second case, if there was no upstream observation started, it will start

=== Example: Function with Observability

```
[source,java]
----
@Bean
Function<Flux<ReceiverRecord<byte[], byte[]>>, Flux<Message<String>>> receive(ObservationRegistry observationRegistry) {

Expand All @@ -50,7 +52,7 @@ Function<Flux<ReceiverRecord<byte[], byte[]>>, Flux<Message<String>>> receive(Ob
.contextWrite(context -> context.put(ObservationThreadLocalAccessor.KEY, receiverObservation));
});
}
```
----

In this example:

Expand All @@ -63,7 +65,8 @@ In this example:

=== Example: Consumer with Observability

```
[source,java]
----
@Bean
Consumer<Flux<ReceiverRecord<?, String>>> receive(ObservationRegistry observationRegistry, @Value("${spring.kafka.bootstrap-servers}") String bootstrap) {
return f -> f.doOnNext(record -> KafkaReceiverObservation.RECEIVER_OBSERVATION.observation(
Expand All @@ -73,7 +76,7 @@ Consumer<Flux<ReceiverRecord<?, String>>> receive(ObservationRegistry observatio
observationRegistry).observe(() -> System.out.println(record)))
.subscribe();
}
```
----

In this case:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@

Following are the maven coordinates for the reactive Kafka binder.

```
[source,xml]
----
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-reactive</artifactId>
</dependency>
```
----
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
Binder allows to have multiple Kafka Streams processors within a single Spring Cloud Stream application.
You can have an application as below.

```
[source,java]
----
@Bean
public java.util.function.Function<KStream<Object, String>, KStream<Object, String>> process() {
...
Expand All @@ -23,7 +24,7 @@ public java.util.function.BiFunction<KStream<Object, String>, KTable<Integer, St
...
}

```
----

In this case, the binder will create 3 separate Kafka Streams objects with different application ID's (more on this below).
However, if you have more than one processor in the application, you have to tell Spring Cloud Stream, which functions need to be activated.
Expand Down Expand Up @@ -52,21 +53,23 @@ In the case of the functional model, you can attach it to each function as a pro

For e.g. imagine that you have the following functions.

```
[source,java]
----
@Bean
public java.util.function.Consumer<KStream<Object, String>> process() {
...
}
```
----

and

```
[source,java]
----
@Bean
public java.util.function.Consumer<KStream<Object, String>> anotherProcess() {
...
}
```
----

Then you can set the application id for each, using the following binder level properties.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ The lifecycle phases `PAUSED` and `RESUMED` are not available in Kafka Streams b

In order to activate binding visualization and control, the application needs to include the following two dependencies.

```
[source,xml]
----
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
Expand All @@ -16,19 +17,21 @@ In order to activate binding visualization and control, the application needs to
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
```
----

If you prefer using webflux, you can then include `spring-boot-starter-webflux` instead of the standard web dependency.

In addition, you also need to set the following property:

```
[source,properties]
----
management.endpoints.web.exposure.include=bindings
```
----

To illustrate this feature further, let us use the following application as a guide:

```
[source,java]
----
@SpringBootApplication
public class KafkaStreamsApplication {

Expand All @@ -47,15 +50,16 @@ public class KafkaStreamsApplication {
}

}
```
----

As we can see, the application has two Kafka Streams functions - one, a consumer and another a function.
The consumer binding is named by default as `consumer-in-0`.
Similarly, for the function, the input binding is `function-in-0` and the output binding is `function-out-0`.

Once the application is started, we can find details about the bindings using the following bindings endpoint.

```
[source,bash]
----
curl http://localhost:8080/actuator/bindings | jq .
[
{
Expand Down Expand Up @@ -89,23 +93,25 @@ Once the application is started, we can find details about the bindings using th
"extendedInfo": {}
}
]
```
----

The details about all three bindings can be found above.

Let us now stop the consumer-in-0 binding.

```
[source,bash]
----
curl -d '{"state":"STOPPED"}' -H "Content-Type: application/json" -X POST http://localhost:8080/actuator/bindings/consumer-in-0
```
----

At this point, no records will be received through this binding.

Start the binding again.

```
[source,bash]
----
curl -d '{"state":"STARTED"}' -H "Content-Type: application/json" -X POST http://localhost:8080/actuator/bindings/consumer-in-0
```
----

When there are multiple bindings present on a single function, invoking these operations on any of those bindings will work.
This is because all the bindings on a single function are backed by the same `StreamsBuilderFactoryBean`.
Expand Down
Loading
Loading