3.16.3.2. Apache Kafka
From the Edit Historian Connection window you can select Apache Kafka.
![[Note]](images/note_1156523861.png) | |
This option is used for writing data to Kafka. To collect data from a Kafka event stream, please see Event Streams. |
Connection Settings
- Label
A unique text string used to identify this connection. This label is used to identify the connection when configuring store and forward, and to create the database name or retention policy in some historians. The label can contain only letters, numbers and the underscore (
_) character.- Topic name
A unique topic created when setting up the Apache Kafka cluster. This topic will contain all the points that will be written. If using Azure Event Hubs for Kafka (see below), this entry will be the Event Hub name.
- Bootstrap Server
A comma-separated list of host and port pairs that are the addresses of the Kafka brokers in a “bootstrap” Kafka cluster. If using a local Kafka cluster, use:
localhost:9092.If using Azure Event Hubs for Kafka, the topic name will be:
Namespace.servicebus.windows.net:9093Namespace- Partition ID
Enter a specific partition ID if you would like to write to a specific partition only. If you would like to write to all partitions, leave this field blank. If you would like to write to multiple specific partitions, create an Apache Kafka connection for each partition.
- Producer Count
The number of producers to send items to Apache Kafka. The default is
1. This ensures that the messages sent maintain the proper time sequence. Using more than one producer increases throughput, but time order of messages is lost.- Message Split Count
The amount of times to split a message batch. For example, if there are 10,000 points, a message split of 5 would split the message into 5 messages of 2000 points each. This is useful to match with the producer count to increase throughput.
- Accumulation time (ms)
The number of milliseconds that the DataHub instance will buffer data in memory before sending it to Kafka. Setting this value to zero means that no accumulation time will be enforced.
If both this value and Maximum # of buffered values (below) are specified, then the DataHub instance will write all buffered values to Kinesis whenever the first of these two limits is reached.
- Maximum # of buffered values
The maximum number of values that the DataHub instance will buffer in memory before transmitting them to Kafka. Setting this value to zero means that all values will be sent as soon as possible.
- Log writes at information level
Checking this option causes messages regarding successful writes to the database to be logged at the level in the DataHub Event Log. If this is not checked then these messages are logged at the level and are therefore normally hidden from the user.
Security
The authentication mechanism used to connect to Kafka.
- Security Protocol
Select one of the following that the Kafka Cluster is set to accept: Plaintext, Ssl, SslPlaintext, SaslSsl. If using Azure Event Hubs for Kafka [link], use SaslSsl.
- SASL Mechanism
If SaslSsl is selected as the Security Protocol, select one of the following Sasl mechanisms specified by the Kafka Cluster to authenticate your Kafka connection: Gssapi, Plain, ScramSha256, ScramSha512, OAuthBearer.
SASL Credentials
- SASL Username
The SASL username. For a connection to an Azure Event Hub, the username will be:
$ConnectionString.- SASL Password
The corresponding password. For a connection to an Azure Event Hub, the password will be the primary connection string of the Event Hub namespace.
Kerberos Credentials
Credentials for the Kafka Cluster if it uses Kerberos Authentication.
SSL Credentials
Credentials for the Kafka Cluster if it uses SSL Authentication.
- SSL CA Location
The location of the SSL CA certificate. Use the button with the three dots
to open the file selector.- SSL CA Password
The password for the SSL CA certificate
- SSL Certificate Location
The location of the SSL certificate. Use the button with the three dots
to open the file selector.- SSL Certificate Password
The password for the SSL certificate
- SSL Key Location
The location of the SSL key. Use the button with the three dots
to open the file selector.- Enable Hostname Verification
Host name verification checks that the broker host name matches the name in the broker certificate.
Azure Event Hubs for Kafka
- Enable Event Hubs for Kafka
Checking this box enables writes to an Event Hub, using Apache Kafka.
Document Settings
- Document definition
Click on the 3 dots on the right hand side to open the Document Definition Dialog.
You can use this editor to construct document definitions. Double-click on any of the items in Common Formulas to put the formula into the editing pane. The Samples list contains a number of sample entries that you can use as-is or as a basis for your own definitions. The syntax and more explanation about ASP documents can be found under ASP Document Definition about 1/2 way down this page.
- Document contains multiple points
Specifies whether or not the document definition is processing more than one point. If accumulation time is being used, this should be selected. If using the provided template, please select this checkbox.
Modify Point Names, Data Sampling and Forwarding
Please see Modify Point Names, Data Sampling or Forwarding in Connection Configuration for how to configure these options.
Once you have completed the Apache Kafka-specific configuration, you can return to Picking Points in General Options to continue.