Apache Kafka

Apache Kafka (opens new window) is an open-source distributed event streaming platform. It is used by many industries and organizations to process payments and perform other financial transactions in real-time, to collect and immediately react to customer interactions and orders, and to serve as the foundation for data platforms, microservices, and much more.
Using Workato Apache Kafka connector you can link with your local or web Kafka cluster instance and:

• Consume messages using the New message trigger.
• Send messages using the Publish message action.

Prerequisites

You must configure Workato's on-prem agent as a Kafka client to connect Apache Kafka to Workato.

Complete the following steps to authenticate Kafka using AWS MSK IAM:

1

Download the AWS MSK IAM authentication JAR from the official AWS repository (opens new window) and place it in your OPA's lib_ext directory.

2

Ensure your Kafka broker is configured to accept AWS MSK IAM authentication.

3

Add the following properties to the additionalProperties section of your Kafka client configuration:

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Refer to the AWS IAM access control (opens new window) guide for more information about IAM configuration or the Connection Profiles page for more information about configuring Kafka connections in the OPA's config.yml file.

Complete the following steps to authenticate Kafka using SHA-256:

1

Ensure your Kafka broker is configured to accept SCRAM-SHA-256 authentication.

2

Add the following properties to the additionalProperties section of your Kafka client configuration:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=YOUR_USERNAME password=YOUR_PASSWORD;

3

Replace the following placeholders:

Placeholder	Definition
YOUR_USERNAME	The username for your Kafka client.
YOUR_PASSWORD	The password for your Kafka client.

Refer to the Connection Profiles page for more information about configuring Kafka connections in the OPA's config.yml file.

Complete the following steps to authenticate Kafka using SSL:

1

Ensure your Kafka broker is configured to accept SSL authentication.

2

Add the following properties to the additionalProperties section of your Kafka client configuration:

security.protocol=SSL
ssl.keystore.location=/PATH/TO/KEYSTORE.jks
ssl.keystore.password=YOUR_KEYSTORE_PASSWORD
ssl.truststore.location=/PATH/TO/TRUSTSTORE.jks
ssl.truststore.password=YOUR_TRUSTSTORE_PASSWORD

3

Replace the following placeholders:

Placeholder	Definition
/PATH/TO/KEYSTORE.jks	The file path to your keystore .jks file.
YOUR_KEYSTORE_PASSWORD	The password to your keystore.
/PATH/TO/TRUSTSTORE.jks	The file path to your truststore .jks file.
YOUR_TRUSTSTORE_PASSWORD	The password to your truststore.

Refer to the Connection Profiles page for more information about configuring Kafka connections in the OPA's config.yml file.

How to connect to Apache Kafka in Workato

CONFIG.YML SETUP

The following steps configure an Apache Kafka connection using a Cloud Profile. Refer to the Apache Kafka Profile guide to configure a connection using a config.yml file.

Complete the following steps to establish a connection to Apache Kafka in Workato using a cloud profile OPA:

1

Provide a Connection name that identifies which Apache Kafka instance Workato is connected to.

Create your connection

2

Use the Location drop-down menu to select the project where you plan to store the connection.

3

Use the On-prem group drop-down menu to select the on-prem group you plan to use.

4

Enter a server URL with either the kafka or kafka+ssl protocol.

5

Specify a Timeout in milliseconds for general operations.

6

Enter the following values in .pem format to connect to Kafka using SSL/TLS:

1

Enter an X509 Server certificate and SSL client certificate.

2

Enter an RSA SSL client key.

3

Optional. Enter an RSA SSL client key password. Password-protected private keys cannot be inlined.

7

Optional. Click Add parameter in the Additional properties for Kafka connection field to define additional properties for your connection.

Define additional properties

You can define any Kafka producer (opens new window) or consumer (opens new window) configuration properties, for example: bootstrap.servers or batch_size.

Some properties can't be configured because the on-prem agent overrides them. Protected properties include:

Property name	Details
key.serializer	Only StringSerializer is supported by agent.
value.serializer	Only StringSerializer is supported by agent.
key.deserializer	Only StringSerializer is supported by agent.
value.deserializer	Only StringSerializer is supported by agent.
auto.offset.reset	Defined by recipes.
enable.auto.commit	Defined internally.

The Kafka connector uses Apache Avro, a binary serialization format. Avro defines what fields are present and their type using JSON schemas. The connector stores data in a schema registry. The Kafka connector only works with schema registry version 6.1.4 and above. For example:

Kafka connection configuration with url for schema registry

8

Click Connect.