Notice: Confluent Platform is the trademark and property of Confluent Inc.

Kafka 0.90 comes with Kafka Connect. Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other data systems. It makes it simple to quickly define connectors that move large data sets into and out of Kafka. Kafka 0.90 comes with console and file connectors. The connector which we think is going to be most useful is JDBC connector. JDBC source connector is provided by Confluent and is built with Confluent platform.

Assumptions:
1. Confluent Platform is installed in /opt/confluent
2. Mysql is installed locally with database “bigdata”
3. There is a “Person” table in “bigdata” with following columns id,first_name,,last_name,age
4. Configure mysql connector jar in following location
/opt/confluent/share/java/kafka-connect-jdbc/mysql-connector-java-5.1.38-bin.jar
5. Create quickstart-mysql.properties in /opt/confluent/etc/kafka-connect-jdbc/
with following content

name=test-mysql-jdbc-autoincrement
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:mysql://127.0.0.1/bigdata?user=hduser&password=******
mode=incrementing
incrementing.column.name=id
topic.prefix=test-mysql-jdbc-

Steps:
1: CD to /opt/confluent

$ cd /opt/confluent

2. Start zookeeper

$ bin/zookeeper-server-start etc/kafka/zookeeper.properties &

3. Start Kafka Broker

$ bin/kafka-server-start etc/kafka/server.properties &

4. Start Schema Registry

$ bin/schema-registry-start etc/schema-registry/schema-registry.properties &

5. Start Mysql Source

$ bin/connect-standalone etc/schema-registry/connect-avro-standalone.properties etc/kafka-connect-jdbc/quickstart-mysql.properties

6. Connect Avro console consumer

 $ bin/kafka-avro-console-consumer --new-consumer --bootstrap-server localhost:9092 --topic test-mysql-jdbc-Person --from-beginning
Top