Clojure and Datomic Studies with Docker and Kafka

Welcome to my monorepo project dedicated to exploring Clojure and Datomic while leveraging Docker and Kafka technologies. This repository documents my study path and progress as I delve into these exciting areas of software development.

Some articles I wrote

• Dockerized Clojure API + Datomic
• Dependency injection with Clojure
• Writing a Kafka consumer with Clojure

Project Structure

The repository is organized into the following sections:

• clojure-api: Contains Web API projects built with Clojure.
• clojure-async: Includes asynchronous Clojure applications that interact with Kafka.
• clojure-api-bff : Includes a Clojure API that acts as a BFF (Backend for Frontend) for the Clojure Async application.
• datomic-datbase: Includes a sample Datomic database and tools to interact with it.
  • datomic-console: Contains Datomic Console configuration files.
• kafka: Contains Kafka configuration files and scripts.
  • zookeeper: Contains Zookeeper configuration files.
  • kafka-ui: Includes a Kafka UI tool to visualize topics and messages.
• prometheus: Contains Prometheus configuration files.
  • kafka-exporter: Contains Kafka exporter configuration files.
  • jmx-exporter: Contains JMX exporter configuration files.
• grafana: Contains Grafana configuration files.`

How to put thing up!.

TL;DR To run the entire project, execute the following command:

docker compose up

This command will start the following services:

• Grafana: http://localhost:3000
• Prometheus: http://localhost:9090
• Datomic Console: http://localhost:9090/browse
• Kafka Topics UI: http://localhost:9099
• Clojure API: http://localhost:9000/swagger
• Clojure Async: http://localhost:9001/swagger

Full explanation:

• Grafana
• Prometheus
• Datomic - Storage Services and Transactor
  • Dev Mode
  • Datomic Console
• Kafka
• Clojure API
• Clojure Async

Some useful commands

• Pre requisites:

  • Install jq to generate JSON payloads.
  • Install Vegeta to generate load tests.

• To produce some messages in Kafka:

cd etc/load_test && vegeta attack -targets=tmp -rate=10 -duration=60s | tee results.bin | vegeta report

Grafana

Official Documentation: Grafana

Run Grafana:

docker compose up grafana

http://localhost:3000

Prometheus

Official Documentation: Prometheus

Run Prometheus:

docker compose up prometheus

http://localhost:9090

Datomic - Storage Services and Transactor

Official Documentation: Storage Services

Dev Mode

Official Documentation: Provisioning dev mode

Run the Datomic Transactor:

docker compose up datomic-transactor

To restore a backup of the MusicBrainz Sample Database:

docker compose run datomic-tools ./bin/datomic restore-db file:/usr/mbrainz-1968-1973 "datomic:dev://datomic-transactor:4334/my-datomic?password=unsafe"

Datomic Console

To run Datomic Console:

docker compose up datomic-console

http://localhost:9090/browse

Kafka

Official Documentation: Kafka Quickstart

Run Kafka:

docker compose up kafka

Kafka Topics UI

Run Kafka UI:

docker compose up kafka-ui

To access the Kafka Topics UI:

http://localhost:9099

Clojure API

To run the Web application:

docker compose up clojure-api

You can access the swagger UI through the following link:

http://localhost:9000/swagger

Clojure Async

To run the Async Kafka producer/Web application:

docker compose up clojure-async

You can access the swagger UI through the following link:

http://localhost:9001/swagger

Study Path

1. Getting Started with Clojure

   • Explore basic syntax and functional programming concepts.
   • Implement small coding exercises to solidify understanding.

2. Diving into Datomic

   • Study Datomic's data model and architecture.
   • Set up a sample Datomic database and interact with it from a Clojure application.

3. Working with Docker

   • Learn Docker fundamentals and containerization concepts.
   • Create Dockerfiles for Clojure applications and Datomic instances.

4. Building with Kafka

   • Understand the principles of event streaming and Kafka.
   • Develop a simple event producer/consumer system using Kafka and Clojure.

5. Integration and Projects

   • Combine knowledge gained from previous steps to build more complex applications.
   • Experiment with connecting Clojure applications to Datomic using Dockerized environments.
   • Explore advanced Kafka scenarios in Clojure-based microservices.

Contributions

Contributions to this repository are welcome! Feel free to submit pull requests if you find any issues, have improvements to suggest, or want to add new study resources related to Clojure, Datomic, Docker, or Kafka.

Resources

• Clojure Official Documentation
• Datomic Documentation
• Docker Documentation
• Apache Kafka Documentation

Acknowledgments

Icons made by Iconfinder and Freepik from www.flaticon.com.

References

https://github.com/demystifyfp/BlogSamples/blob/0.10/clojure/resultful-crud/src/resultful_crud/core.clj https://gsfl3101.medium.com/kafka-kraft-monitoring-with-prometheus-and-grafana-1994ef272f48

Extra

I'm using portainer to manage the containers, you can access it through the following link:

https://docs.portainer.io/start/install/server/setup

You should be able to manage your docker containers through a web interface like this:

License

This project is licensed under the MIT License - see the LICENSE file for details.