Our Kafka Producer writes messages on a Kafka topic acting transactionally and it could rollback the transaction when certain events occur. As of now, the behaviour of the Kafka connector is to read also uncommitted messages (see case TS003289786), meaning that rolled back messages will be ingested. The Kafka Connector should be able to let the user decide if reading only committed messages (isolation.level=READ_COMMITTED) or also non committed messages (READ_UNCOMMITTED)
Why is it useful?
|Who would benefit from this IDEA?||As a customer, we need to have a transactional behaviour of the kafka consumer in order to use Big Integrate and Information Server to interact with our Kafka Cluster|
How should it work?
1) The Kafka Connector lets the user define the behaviour of the KafkaConsumer class
2) The isolation.level property will be set accordingly and the transactional consumer should be developed as per the Kafka guidelines (https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
|Priority Justification||It is a mandatory requirement for the italian ministry of finance. The penalty would be the choice of a different system to ingest data|