08. Kafka as messaging system

Date: 2018-05-23

Status

Accepted

Context

Valamis is full of messages and events and robust microservice architecture is often event-based. Platform for sending and receiving of these events and reacting to them is needed. Microservices can communicate with messages or push events to pipeline for asynchronous processing. Learning Record Store (LRS) is a good example of storage service that subscribes to an event feed.

Decision

Apache Kafka is used as a general messaging system and distributed streaming platform.

Kafka is a flexible and scalable distributed streaming platform that has emerged as one of the de facto solutions in the industry and seems to have a good momentum and future.

From Kafka's homesite:

A streaming platform has three key capabilities:

* Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.
* Store streams of records in a fault-tolerant durable way.
* Process streams of records as they occur.

Kafka is generally used for two broad classes of applications:

* Building real-time streaming data pipelines that reliably get data between systems or applications
* Building real-time streaming applications that transform or react to the streams of data

Therefore Kafka can be used as a more traditional messaging system and as a persistent storage for these messages when applicable.

Consequences

Any Valamis service can produce and consume messages from different topics in Kafka cluster. Kafka Streams can be used to transform messages to other types of messages.

Developers need to learn new asynchronous style of communication instead of familiar request/response. Resource needs of Kafka need to be learned.