Industrial Cognition AI System

A Project Blueprint for Autonomous Vision and Robotics
Explore the docs »

Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgements

Industrial Cognition AI System

About The Project

Project Overview

This project implements a high-throughput, industrial-grade computer vision system utilizing a modern, decoupled * Microservices Architecture* to perform Open-World Localization Vision Transformer (OWLv2) recognition from multiple real-time industrial cameras. It's goal is to create an autonomous, real-time system that uses computer vision to detect and locate specific components (metal sheets, parts) in a physical workspace, and then reliably issues corrective/directive commands to a physical robot controller.

Architectural Overview (Layered Design)

The system adheres to a strict Layered Architecture to maximize resilience and maintainability.

Flow: Data flows from the physical world $\rightarrow$ Camera Service $\rightarrow$ Vision Service $\rightarrow$ Robot Interface $\rightarrow$ Physical Robot.

Layers:

Here are main layers of the MVM (Machine Vision Module):

1. Presentation (FastAPI API): Provides a modern, websocket-driven control panel for monitoring and control.
2. Service Layer (Orchestrator): The main API handles state management, error recovery, and coordinating calls between specialized domain services.
3. Domain Services:
   • Camera Service: Manages all physical I/O, abstracting the complexity of video streaming via FFmpeg/GStreamer.
   • Vision Service: Executes the heavy computational lifting using OpenCV and ONNX Runtime models (e.g., OWLv2).
   • Robot Interface: Handles the low-level physical communication, strictly adhering to Modbus/RS-485 protocols.
4. Data Layer: Uses SQLAlchemy/PostgreSQL for durable storage of metadata and historical results.

Key Protocols & Constraints

• Communication Protocol: Physical robot control can use binary Modbus over RS-485 or direct TCP/IP connection to MVM stream of JSON data (API key configuration required).
• Streaming: All real-time video streams are managed via FFmpeg/GStreamer pipelines to ensure performance consistency.
• Deployment: containerized and orchestrated using Kubernetes (K8s) for high availability and horizontal scaling.

Core Functionality & Goals

• Goal: To autonomously detect, locate, and classify specific parts in an industrial setting with extreme reliability.
• Performance Target: A primary operational goal is achieving 60 FPS total throughput, requiring sub-200ms end-to-end inference latency.
• Scope Lock: The current focus is narrowed to a single robot arm interacting with a defined work area, demanding precise kinematic control and coordinate mapping.

Architecture & Technology Stack (The "How"): The system is designed for maximum operational resilience using a robust, asynchronous pipeline pattern:

1. Ingestion: Raw streams are captured via RTSP cameras and pre-processed using FFmpeg into a standardized format.
2. Orchestration: A FastAPI gateway accepts streams, which are queued into a RabbitMQ message broker.
3. Processing: Dedicated Inference Workers (running on GPU-accelerated Kubernetes pods) consume messages, run the OWLv2 model via ONNX Runtime, and determine object locations.
4. Persistence: All raw frames, processed results, and metadata are stored in MinIO (S3-compatible object storage).
5. Monitoring: A comprehensive Observability Stack (Prometheus, Loki, Grafana) tracks system health, worker load, and data throughput for proactive failure detection.

Key Technical Constraints & Focus Areas:

• Hardware Dependence: Requires NVIDIA GPUs (e.g., RTX 5060 Ti) to meet latency targets.
• Physical Grounding: All generated coordinates must be relative to a physically defined coordinate system anchored by the camera's 3000mm height and 45-degree viewing angle.
• Failure Handling: The system must explicitly report and handle diagnostic states like NO_TARGET, OUT_OF_REACH, and HUMAN_ON_SITE.

In essence, the project is building a scalable, highly reliable, real-time, closed-loop vision system designed for industrial automation, with a clear mandate to favor resilient, decoupled architecture over initial simplicity.

Built With

• Kubernetes
• Docker
• FFMPEG
• RTSP
• Modbus
• FastAPI
• PostgreSQL
• RabbitMQ
• Celery
• Redis
• MinIO
• NVIDIA RTX
• Loki
• Alloy
• Prometheus
• Grafana

Getting Started

Follow the steps in Onboarding docs for new team members to get the entire stack running locally for development.

Prerequisites

Hardware Requirements

Minimum Requirements

• CPU: Intel Core i7 or equivalent (8 cores, 16 threads)
• GPU: NVIDIA GTX 1660 or equivalent (6GB VRAM) * optional but recommended
• RAM: 16GB DDR4
• Storage: 512GB SSD (for OS, application, and database)
• Camera Interface: Compatible with industrial camera (GigE Vision, USB3 Vision, or Camera Link)
• Network: Gigabit Ethernet
• 60 FPS 2K Industrial Camera with Gigabit Ethernet connectior

Recommended Requirements

• CPU: Intel Xeon or AMD Ryzen 9 (16+ cores)
• GPU: NVIDIA RTX 3070 or equivalent (8GB+ VRAM)
• RAM: 32GB DDR4
• Storage: 1TB NVMe SSD
• Camera Interface: Dedicated frame grabber card if required by camera
• Network: 10GbE capable
• 60 FPS 4K Industrial Camera with Gigabit Ethernet connectior

Software Requirements

• Kubernetes
• Docker
• Ubuntu Server or another apt-compatible distributive.

Installation

1. Clone the repo

   git clone https://github.com/rusoil-software/industrial-cognition.git

2. Go into the directory of the project

   cd industrial-cognition

3. Run deployment script

   sh scripts/install.sh

4. Wait until installation and deployment is finished

5. Go to your-server-public-ip/dashboard (with deployment via reverse proxy) or localhost:8080/dashboard (for development on local machine) to see the MVM Web App

6. Follow initial setup instructions within the MVM App

Usage

For more examples, please refer to the Documentation

Development & Process Guide

The following files detail how to build and how to maintain the system:

• doc/workflow.md: Mandatory step-by-step guide for all feature development, emphasizing sequential workflow adherence.
• doc/conventions.md: Contains the hard rules (KISS, SRP, Modbus usage) that every line of code must follow.
• doc/tasklist.md: The master tracker for tracking all remaining development tasks.

Roadmap

See the open issues for a list of proposed features (and known issues).

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the Apache 2 License. See LICENSE for more information.

Contact

Konstantin Ustiuzhanin - email

Project Link: Industrial Cognition

Acknowledgements

• Kubernetes
• Docker
• FFMPEG
• Modbus
• FastAPI
• PostgreSQL
• RabbitMQ
• Celery
• Redis
• MinIO
• NVIDIA RTX
• Loki
• Alloy
• Prometheus
• Grafana