Azure Function with Kafka Trigger & Output Binding

Spring Cloud Function example for implementing an Azure functions with KafkaTrigger and Kafka Binding support.

The Azure function is triggered by messages sent on the trigger topic and in turn calls the uppercase SCF with the trigger payload. The SCF capitalizes the input json value fields and sends the result to an output Kafka topic called: output.

Running Locally

First start a Kafka server locally. The ./src/main/resources/docker-compose-demo.yaml helps to start locally Zookeeper, Kafka and Kafka UI.

docker-compose -f ./src/main/resources/docker-compose-demo.yaml up

You can reach the Kafka UI (Redpanda) dashboard on http://localhost:8080/topics

The docker-compose pre-creates the trigger and output topics used by the function.

Next build and run the Azure function:

./mvnw clean package
./mvnw azure-functions:run

From the Kafka UI, got to the trigger topic view (http://localhost:8080/topics/trigger), select Actions/Publish Message and submit a new JSON message:

{ "foo" : "bar"}

Push the Publish button and let the function do its job and check the output topic (http://localhost:8080/topics/output) :

e.g.the bar is in uppercase:

{ "foo" : "BAR"}

Running on Azure (TODO: WIP)

Make sure you are logged in your Azure account.

az login

Build and deploy

./mvnw clean package
./mvnw azure-functions:deploy

Implementation

Configure the Kafka extension in the host.json settings:

{
    "functionTimeout": "00:05:00",
    "version": "2.0",
    "extensions": {
        "kafka": {
            "maxBatchSize": 64,
            "SubscriberIntervalInSeconds": 1,
            "ExecutorChannelCapacity": 1,
            "ChannelFullRetryIntervalInMs": 50
        }
    },
    "extensionBundle": {
        "id": "Microsoft.Azure.Functions.ExtensionBundle",
        "version": "[3.3.0, 4.0.0)"
    }
}

Also to allow your functions to scale properly on the Premium plan when using Kafka triggers and bindings, you need to enable runtime scale monitoring.

Notes

Disable the spring-boot-maven-plugin in favor of the azure-functions-maven-plugin.
Exclude the org.springframework.boot:spring-boot-starter-logging dependency from the org.springframework.cloud:spring-cloud-function-adapter-azure.
In local.settings.json set the local values for the %BrokerList, %ConfluentCloudUsername% and the %TriggerKafkaTopic% trigger and binding variables:

{
	"IsEncrypted": false,
	"Values": {
...
		"BrokerList": "localhost:9092",
		"ConfluentCloudUsername": "test",
		"TriggerKafkaTopic": "trigger"
	}
}

References

Apache Kafka for Confluent Cloud - Azure portal - show how to create an instance of Apache Kafka for Confluent Cloud.

Create a Cluster in Confluent Cloud

3.7 KiB Raw Blame History