To run the entire stack, simply use:
docker-compose up --build
if the bash closed with error 'service X is unhelathy' so please run the command docker-compose up again after few seconds,its happens sometimes because the dependencies
by default the main rails server will run on
http://localhost:3000
and go server for create chat and create message
will run on
http://localhost:8080
see the endpoints section below for all available requests and the parameters or body for each request
This application is a chat system designed to efficiently manage chats and messages for multiple applications. The system uses a Rails server with Sidekiq for background job processing, Redis for atomic operations and concurrency control, and Elasticsearch for advanced search capabilities.
Additionally, a Go server running on port 8080
has been implemented to handle create chat
and create message
requests. The Rails server (on port 3000
) still supports these endpoints, but the Go server processes them faster. Both implementations share the same behavior:
- Retrieve the next
chat_number
ormessage_number
from Redis with atomic updates to prevent race conditions. - Return the generated number to the client.
- Push the creation request to a background job for further processing.
The system has comprehensive test coverage implemented using RSpec:
- Models Tests: Ensures data integrity and validation rules are enforced.
- Controllers Tests: Validates API endpoints and request handling.
- Jobs Tests: Verifies the behavior of background job processing.
Tests generate reports in the following formats:
- Text:
rspec_report.txt
- JSON:
rspec_report.json
- HTML:
rspec_report.html
These reports provide detailed insights into the system's functionality and can be used for debugging and quality assurance.
the main part of the system founded in app/models , app/controllers , config/initializers
the background workers founded in app/sidekiq
-
Rails Server: Handles API requests and coordinates operations.
-
Go Server: Optimized for handling
create chat
andcreate message
requests at faster speeds. -
Sidekiq: Manages background jobs, such as creating chats and messages, and updating counters asynchronously.
-
Redis:
- Used to store atomic counters for generating unique
chat_number
(per app) andmessage_number
(per chat). - Ensures concurrency control during simultaneous requests.
- Used to store atomic counters for generating unique
-
Elasticsearch: Provides support for partial and proximity search for messages.
- Unique Identifiers:
- Upon initialization, the system preloads the number of existing chats per app and messages per chat from the database.
- Redis atomic operations generate unique
chat_number
andmessage_number
to avoid race conditions.
- Chat Creation:
- Step 1: The main server (Rails or Go) uses Redis to create a unique
chat_number
and atomically increment it. It then returns thechat_number
to the client and pushes the creation request into the message queue. - Step 2: The background job receives the creation request, creates the chat entity in the database, and increments the
chat_number
in the database.
- Step 1: The main server (Rails or Go) uses Redis to create a unique
- Message Creation:
- Users specify the chat via app token and
chat_number
. - A unique
message_number
is generated and stored in Redis. - The system increments the message counter for the chat asynchronously.
- Users specify the chat via app token and
- Partial Search:
- Elasticsearch is used to find messages that closely match the user query within a specific chat or across the entire system.
you could see all of it in ./db/migrate
- App: Represents an application with chats and messages.
id
(Primary Key)token
(Unique, indexed for fast search)name
chat_count
(Default: 0)
- Chat: Represents a conversation linked to an app.
id
(Primary Key)app_id
(Foreign Key, indexed for search)chat_number
(Unique per app, indexed)messages_count
(Default: 0)
- Message: Represents messages in a chat.
id
(Primary Key)chat_id
(Foreign Key, indexed for search)message_number
(Unique per chat, indexed)body
- Apps Table:
token
: Optimizes lookup by app token.
- Chats Table:
- Composite index on
[app_id, chat_number]
: Ensures uniqueness and speeds up searches. app_id
: Optimizes searches by app.
- Composite index on
- Messages Table:
- Composite index on
[chat_id, message_number]
: Ensures message uniqueness within a chat. chat_id
: Optimizes retrieval of messages by chat.
- Composite index on
you could find add endpoints in file ./config/routes.rb.
in the following all endpoints available with the required body and expected response in JSON format, note that if the body does not specify in any endpoint it is not needed any body in this endpoint
- POST /apps/: Create a new application,default chat_count=0.
- Request Body:
{ "app": { "name": "app name" } }
- Response:
{ "token": "32 byte app token", "name": "app name", "chat_count": 0 }
- Request Body:
- GET /apps/:token: Retrieve application details (name, chat count).
- Response:
{ "name": "app name", "chat_count": 2 }
- Response:
- PATCH /apps/:token: Update the application name.
- Request Body:
{ "app": { "name": "this is the new name" } }
- Response:
{ "name": "this is the new name", "chat_count": 0 }
- Request Body:
- GET /apps: Retrieve all application details (name, chat count).
- Response:
[ { "name": "app 1 name", "token": "32 byte token", "chat_count": 3 }, { "name": "app 2 name", "token": "32 byte token", "chat_count": 2 } ]
- Response:
- POST /apps/:app_token/chats: Create a new chat under a specific app, default message_count=0,return the created chat_number.
- Response:
{ "chat_number": 1 }
- Response:
- GET /applications/:token/chats: Retrieve all chats for a specific app.
- Response:
[ { "chat_number": 1 }, { "chat_number": 2 } ]
- Response:
- GET /chats: Retrieve all chats across the system.
- Response:
[ { "chat_number": 1, "messages_count": 0, "app_token": "32 byte app 1 token" }, { "chat_number": 2, "messages_count": 0, "app_token": "32 byte app 1 token" }, { "chat_number": 3, "messages_count": 0, "app_token": "32 byte app 1 token" }, { "chat_number": 1, "messages_count": 0, "app_token": "32 byte app 2 token" }, { "chat_number": 2, "messages_count": 0, "app_token": "32 byte app 2 token" } ]
- Response:
-
POST /apps/:app_token/chats/:chat_number/messages: Create a new message in a chat.
- Request Body:
{ "message": { "body": "ahmed want to go to home" } }
- Response:
{ "message_number": 5 }
- Request Body:
-
GET /apps/:app_token/chats/:chat_number/messages?query="text": Perform a partial search within a chat using Elasticsearch,if the user doesn't pass the query then the response will Retrieve all messages in a specific chat with matching_score=0.
- Response:
[ { "matching_score": 3.7836456, "body": "ahmed want to go to home", "message_number": 5 }, { "matching_score": 0.57843524, "body": "yasser play football", "message_number": 1 } ]
- Response:
-
GET /messages: Retrieve all messages across the system.
-
Response:
[ { "app_token": "32 byte app token", "chat_number": 1, "message_number": 1, "body": "text" }, { "app_token": "32 byte app token", "chat_number": 2, "message_number": 3, "body": "text" } ]
-
Redis ensures atomic operations to generate unique numbers:
- Initialization:
- Load chat and message counts into Redis on startup.
- Concurrency Control:
- Increment chat and message counters atomically.
- Prevent race conditions during simultaneous requests.
- Search Capabilities:
- Partial and proximity search for messages.
- Query-based filtering at the chat.
- Database:
- Use MySQL (Docker setup provided).
- Set up necessary indexes for optimization.
- Redis:
- Build and run the Redis container.
- Sidekiq:
- Configure for background job processing.
- Elasticsearch:
- Ensure Elasticsearch is running for search endpoints.
- add cashing for all data requested to return direct when asked again
- Enhanced analytics and reporting.
- API rate-limiting for better control.
- app_token collision not handled "very rare because it is 32 bytes"
- redis initializes loading in RAM for all app.chat_count by app token and all chat.message_count by app_token+chat_number, if there are too many records, redis or the whole system may crash
- In creating a chat request, the system gets the next unused chat_number of "specific app" from atomic RAM storage and returns it to the client and the background worker will handle the actual creation asynchronously, but if the background worker fails due to internal issues or duplicate chat_number or any other reason, there will be a problem because the user will assume that this chat_number is the creator, and this chat_number will be cached in redis but it will not appear in the actual database