diff --git a/CLAUDE.md b/CLAUDE.md index fb992ef..d03f591 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,97 +1,282 @@ -# AI Base Template - Development Guide +# AI Base Template - Production-First Development Guide -A Python template for ML/AI projects with FastAPI, designed for rapid prototyping and clean architecture. +A production-ready Python template for AI/ML systems, designed for reliability, observability, and cost management from day one. + +## Project Philosophy + +This template embodies **production-first AI engineering**, where we optimize for reliability over research metrics: + +- **90% Infrastructure, 10% AI Logic** - Most code is defensive engineering, not model development +- **Engineering Discipline** - Comprehensive testing, monitoring, and error handling +- **Cost Management** - Real-time budget tracking and resource controls +- **Observable Systems** - AI-specific metrics and monitoring patterns ## Project Structure ``` ai-base-template/ -├── ai_base_template/ # Main application code +├── ai_base_template/ # Production AI service code │ ├── __init__.py -│ └── main.py # FastAPI entry point -├── tests/ # Test suite -│ └── test_main.py -├── research/ # Notebooks and experiments -│ └── EDA.ipynb # Exploratory data analysis -├── testing/ # API testing utilities -├── Makefile # Development automation -└── pyproject.toml # Project config & dependencies +│ ├── main.py # AI service with defensive patterns +│ ├── config.py # Environment-driven configuration +│ └── monitoring.py # AI-specific observability +├── tests/ # Defensive testing strategy +│ ├── test_main.py # Basic functionality tests +│ └── test_ai_service.py # AI-specific defensive tests +├── research/ # Experimental AI development +│ └── EDA.ipynb # Exploratory data analysis +├── ARCHITECTURE.md # Production system design docs +├── Makefile # Development automation +└── pyproject.toml # Project config & dependencies ``` ## Quick Start -### Setup +### Environment Setup ```bash -make environment-create # Creates Python 3.12 env with uv -make environment-sync # Updates dependencies +make init # Complete development environment setup +make sync # Update dependencies ``` ### Development Commands ```bash -make format # Auto-format with Ruff -make lint # Lint and auto-fix issues -make type-check # Type check with MyPy -make validate-branch # Run all checks before PR +# Code quality (production-ready standards) +make format # Auto-format with Ruff +make lint # Lint and auto-fix issues +make type-check # Static type validation +make validate-branch # Full pre-commit validation + +# Testing (AI-focused test strategy) +make test # Standard test suite (excludes integration) +make test-unit # Fast, isolated component tests +make test-functional # AI workflow tests +make test-integration # Service-level integration tests +make test-all # Complete suite including cost/load tests + +# Environment management +make clean-project # Clean Python caches +make clean-env # Remove virtual environment +``` + +## Production-First Development Workflow + +### 1. Configuration-Driven Development +All production concerns are configured, not hardcoded: + +```python +# ai_base_template/config.py +class AIServiceConfig(BaseSettings): + # Cost management + monthly_budget_limit: float = 10000.0 + cost_alert_threshold: float = 100.0 + + # Reliability + model_timeout: float = 5.0 + enable_fallback: bool = True + + # Observability + log_level: str = "INFO" + enable_tracing: bool = True +``` + +### 2. Defensive Testing Strategy +Test for AI-specific failure modes: + +```bash +# Run defensive AI tests +make test-unit # Input validation, cost controls +make test-integration # Service-level AI workflows +make test-all # Include load and cost validation +``` + +Test categories: +- **Unit tests**: `@pytest.mark.unit` - Fast, isolated AI component tests +- **Functional tests**: `@pytest.mark.functional` - Feature workflow tests +- **Integration tests**: `@pytest.mark.integration` - Service-level tests with dependencies +- **Performance tests**: `@pytest.mark.performance` - Cost and latency validation + +### 3. Cost-Aware Development +Every AI operation is cost-tracked: + +```python +# Cost tracking in all AI operations +with cost_tracker.track_cost(estimated_cost=0.01): + result = await model.predict(data) + +# Monitor budget status +budget_status = cost_tracker.get_budget_status() ``` -### Testing +### 4. Comprehensive Validation +Before any commit: + ```bash -make test-unit # Run unit tests -make test-functional # Run functional tests -make test # Run standard tests with coverage -make test-all # Run all tests with coverage +make validate-branch # Runs: lint → type-check → test ``` -## Development Workflow +This ensures: +- ✅ Code formatting and linting compliance +- ✅ Static type checking passes +- ✅ All defensive tests pass +- ✅ Cost controls are validated +- ✅ Performance requirements met -1. **Write code** following Python conventions: - - Classes: `PascalCase` - - Functions/variables: `snake_case` - - Constants: `UPPER_SNAKE_CASE` - - Max line length: 120 characters +## Key Technologies & Production Patterns -2. **Validate before committing**: +### Core Infrastructure +- **FastAPI**: Production-grade async web framework +- **Pydantic**: Runtime data validation and type safety +- **loguru**: Structured logging for observability +- **uv**: Fast, reliable Python package management + +### AI-Specific Engineering +- **Cost Tracking**: Real-time budget monitoring and alerts +- **Circuit Breakers**: Prevent cascading AI model failures +- **Graceful Degradation**: Fallback strategies for AI failures +- **Input Sanitization**: Prevent adversarial input attacks +- **Timeout Management**: Prevent hanging AI operations + +### Monitoring & Observability +- **AI Metrics**: Confidence distributions, fallback rates +- **Cost Metrics**: Per-request costs, budget utilization +- **Performance Metrics**: Latency percentiles, throughput +- **Error Categorization**: Input validation vs. model failures + +## Production Deployment Patterns + +### Environment Configuration +```bash +# .env.production +MODEL_VERSION=v2.1.0 +MODEL_TIMEOUT=3.0 +CONFIDENCE_THRESHOLD=0.90 +MAX_REQUESTS_PER_USER=500 +COST_ALERT_THRESHOLD=1000.0 +ENABLE_FALLBACK=true +``` + +### Health Checks +```python +# Built-in health check endpoint +def get_service_health() -> dict: + return { + "status": "healthy", + "model_version": config.model_version, + "cost_summary": cost_tracker.get_budget_status(), + "performance_summary": metrics.get_performance_summary() + } +``` + +## Best Practices for Production AI + +### Code Quality Standards +- **Type hints on all functions** - Prevent runtime AI failures +- **Comprehensive error handling** - AI systems fail uniquely +- **Input validation** - Sanitize adversarial inputs +- **Cost awareness** - Track and limit expensive operations +- **Fallback strategies** - Graceful degradation for AI failures + +### Testing Standards +- **Test coverage > 80%** - Include AI-specific edge cases +- **Defensive testing** - Validate against malicious inputs +- **Cost validation** - Ensure budget controls work +- **Performance testing** - Validate latency requirements +- **Failure scenario testing** - Test circuit breakers and fallbacks + +### Monitoring Standards +- **Real-time cost tracking** - Prevent budget overruns +- **Confidence monitoring** - Detect model drift early +- **Latency monitoring** - Maintain SLA compliance +- **Error categorization** - Distinguish AI vs. infrastructure failures +- **Capacity planning** - Monitor resource utilization trends + +## Common Production AI Challenges + +### 1. Cost Control +```python +# Rate limiting to prevent cost spirals +@rate_limit(max_requests_per_user=1000) +async def predict(request): + with cost_tracker.track_cost(): + return await ai_model.predict(request) +``` + +### 2. Input Validation +```python +# Sanitize adversarial inputs +def validate_ai_input(data: str) -> str: + if len(data) > MAX_INPUT_LENGTH: + raise ValueError("Input too large") + return sanitize_adversarial_patterns(data) +``` + +### 3. Timeout Management +```python +# Prevent hanging AI operations +result = await asyncio.wait_for( + ai_model.predict(data), + timeout=config.model_timeout +) +``` + +### 4. Graceful Degradation +```python +# Fallback strategies for AI failures +try: + return await primary_ai_model.predict(data) +except Exception: + return await fallback_model.predict(data) +``` + +## Getting Started with Production AI + +1. **Clone and initialize**: ```bash - make validate-branch # Runs linting and tests + git clone my-ai-service + cd my-ai-service + make init ``` -3. **Test thoroughly**: - - Unit tests: `@pytest.mark.unit` - - Functional tests: `@pytest.mark.functional` - - Integration tests: `@pytest.mark.integration` +2. **Understand the architecture**: + ```bash + # Read the production patterns + cat ARCHITECTURE.md + + # Examine the defensive code + cat ai_base_template/main.py + ``` -## Key Technologies +3. **Run defensive tests**: + ```bash + # Validate the foundation + make validate-branch + + # Run AI-specific tests + make test-all + ``` -- **FastAPI**: Modern Python web framework -- **Pydantic**: Data validation using Python type annotations -- **MyPy**: Static type checking -- **Ruff**: Fast Python linter and formatter -- **pytest**: Testing framework -- **uv**: Fast Python package manager +4. **Implement your AI logic**: + - Replace the mock `_make_prediction()` method with your model + - Keep all the defensive infrastructure intact + - Add AI-specific configuration in `config.py` + - Extend monitoring in `monitoring.py` -## ML/Data Science Stack +5. **Deploy with confidence**: + ```bash + # Final validation + make validate-branch + + # Deploy knowing you have production safeguards + ``` -- **scikit-learn**: Machine learning library -- **XGBoost/LightGBM**: Gradient boosting frameworks -- **PyTorch**: Deep learning framework -- **pandas/numpy**: Data manipulation -- **SHAP**: Model interpretability +## Remember: Production AI is Infrastructure Engineering -## Best Practices +> "The best AI is the AI that works." -- Type hints on all functions -- Pydantic models for data validation -- Structured logging with loguru -- Environment-based configuration -- No hardcoded secrets -- Test coverage > 80% +This template prioritizes **reliability over research metrics**. Most of your code will be infrastructure—cost controls, error handling, monitoring, and fallbacks—not AI/ML logic. -## Getting Started +That's exactly how production AI systems should be built. -1. Clone the template -2. Run `make environment-create` -3. Start coding in `ai_base_template/` -4. Add tests in `tests/` -5. Use `make validate-branch` before commits +--- -This template provides a solid foundation for ML/AI projects with all the modern Python tooling pre-configured. \ No newline at end of file +For detailed architecture patterns and production deployment strategies, see [ARCHITECTURE.md](ARCHITECTURE.md). \ No newline at end of file diff --git a/Makefile b/Makefile index 6a55ff4..b8b884c 100644 --- a/Makefile +++ b/Makefile @@ -39,6 +39,11 @@ init: ## Set up Python version, venv, and install dependencies fi @echo "🎉 Environment setup complete!" +sync: ## Sync project dependencies + @echo "Syncing project dependencies..." + uv sync --extra dev + $(GREEN_LINE) + clean-project: ## Clean Python caches and tooling artifacts @echo "Cleaning project caches..." find . -type d \( -name '.pytest_cache' -o -name '.ruff_cache' -o -name '.mypy_cache' -o -name '__pycache__' \) -exec rm -rf {} + @@ -49,10 +54,7 @@ clean-env: ## Remove the virtual environment folder rm -rf .venv $(GREEN_LINE) -sync: ## Sync project dependencies - @echo "Syncing project dependencies..." - uv sync --extra dev - $(GREEN_LINE) + # ---------------------------- # Code Quality diff --git a/README.md b/README.md index 2ea5382..4b3ef70 100644 --- a/README.md +++ b/README.md @@ -1,189 +1,166 @@ -# AI Base Template +# AI Base Template: Production-First AI Engineering -A minimal Python template for AI/ML projects with modern tooling, designed to help you start projects faster with best practices built-in. +> Based on [A Production-First Approach to AI Engineering](https://aienhancedengineer.substack.com/p/a-production-first-approach-to-ai) - a methodology for building reliable AI systems. -## What is this? +## 🎯 Why This Template Exists -This is a simple, clean Python project template that comes pre-configured with: -- Modern Python development tools -- ML/Data science libraries -- Testing infrastructure -- Code quality automation -- Clean project structure +**The Problem:** Most AI projects fail when moving from prototype to production. Research notebooks that work brilliantly in development fail catastrophically under real-world conditions—latency spikes, cost spirals, non-deterministic failures, and maintenance nightmares. -Perfect for starting new AI/ML experiments, research projects, or proof-of-concepts without setting up all the tooling from scratch. +**The Root Cause:** The AI industry focuses 90% on model development and 10% on the infrastructure needed for production. This ratio should be reversed. Production AI systems require engineering discipline, not just algorithmic innovation. -## Features +**The Solution:** This template provides a production-ready foundation for AI projects, embodying the principle that *"Research optimizes for possibility. Engineering optimizes for reliability."* -- 🐍 **Python 3.12** with modern packaging via uv -- 🧪 **Testing setup** with pytest (unit, functional, integration markers) -- 🔧 **Code quality** with Ruff (formatting + linting) and MyPy (type checking) -- 📊 **ML-ready** with pre-configured data science libraries -- 📝 **Type hints** and Pydantic for data validation -- 🔍 **Logging** with loguru for better debugging -- ⚡ **Make commands** for common development tasks -- 📓 **Jupyter** support for experimentation +## 🏗️ What This Template Provides -## Quick Start +A **modern Python foundation** designed for AI systems that need to work reliably in production: -### Prerequisites -- Python 3.12+ -- Make +- **Modern Python Tooling** - Python 3.12+, FastAPI, Pydantic, type hints throughout +- **Development Automation** - Pre-configured linting, formatting, testing, and validation +- **Production-Ready Structure** - Organized for maintainability and scaling +- **Comprehensive Testing** - Unit, functional, and integration test patterns +- **CI/CD Ready** - GitHub Actions, pre-commit hooks, semantic versioning +- **Documentation Standards** - Clear guides for development and deployment -### Setup +This isn't another ML experiment template—it's an engineering foundation for AI systems that need to work reliably at scale. -1. Clone or use this template: -```bash -git clone my-ai-project -cd my-ai-project -``` +## ⚡ Quick Start -2. Create environment and install dependencies: ```bash -make environment-create -``` +# Clone the production-ready foundation +git clone my-ai-service +cd my-ai-service -3. Start coding! Your code goes in `ai_base_template/` +# Set up the complete development environment +make init -4. Run tests to make sure everything works: -```bash -make test +# Verify everything works +make validate-branch ``` -## Project Structure +You now have a production-ready Python service foundation. Add your AI logic on top of this reliable base. -``` -ai-base-template/ -├── ai_base_template/ # Your Python package -│ ├── __init__.py # Package initialization -│ └── main.py # Example module -├── tests/ # Test files -│ └── test_main.py # Example tests -├── research/ # Notebooks and experiments -│ └── EDA.ipynb # Exploratory data analysis -├── testing/ # Test utilities and scripts -├── Makefile # Development commands -├── pyproject.toml # Project configuration -├── CLAUDE.md # Development guide -├── ADR.md # Architecture Decision Record -├── .gitignore # Git ignore rules -└── README.md # This file -``` +## 🔧 The Production-First Philosophy + +### Research vs. Production Mindset + +**Research Approach:** +- Optimize for accuracy and novel algorithms +- Success = high F1 scores, paper publications +- Acceptable to fail fast and iterate +- Focus on the happy path + +**Production-First Approach:** +- Optimize for reliability and maintainability +- Success = uptime, cost efficiency, user satisfaction +- Must handle edge cases gracefully +- Plan for failure from the start + +### The 90/10 Rule + +In production AI systems: +- **10%** of your code is the actual AI/ML logic +- **90%** is infrastructure: validation, monitoring, error handling, cost controls, testing + +This template provides that crucial 90% foundation. -## Development Workflow +## 🛠️ Development Workflow ### Essential Commands ```bash -# Environment -make environment-create # First-time setup -make environment-sync # Update after changing dependencies +# Environment management +make init # Complete development setup +make sync # Update dependencies +make clean-env # Reset environment -# Code Quality -make format # Auto-format code -make lint # Fix linting issues -make type-check # Check types -make validate-branch # Run all checks (before committing) +# Code quality +make format # Auto-format code +make lint # Fix linting issues +make type-check # Validate type hints +make validate-branch # Run all checks before committing # Testing -make test # Run unit tests -make test-functional # Run functional tests -make test # Run all tests with coverage +make test # Standard test suite +make test-unit # Fast unit tests +make test-functional # Feature tests +make test-integration # Integration tests +make test-all # Complete test suite ``` -### Adding Code - -1. Add your modules to `ai_base_template/` -2. Write corresponding tests in `tests/` -3. Use type hints for better code quality -4. Run `make validate-branch` before committing - -## Pre-installed Libraries - -### ML/Data Science -- **numpy** - Numerical computing -- **pandas** - Data manipulation -- **scikit-learn** - Classical ML algorithms -- **XGBoost** - Gradient boosting -- **LightGBM** - Fast gradient boosting -- **PyTorch** - Deep learning -- **SHAP** - Model explainability - -### Development Tools -- **pytest** - Testing framework -- **ruff** - Fast Python linter/formatter -- **mypy** - Static type checker -- **pre-commit** - Git hooks -- **loguru** - Better logging -- **python-dotenv** - Environment variables -- **jupyter** - Interactive notebooks - -## Configuration - -Use environment variables for configuration. Create a `.env` file in the project root: - -```env -# Example .env -LOG_LEVEL=DEBUG -DATA_PATH=./data -MODEL_PATH=./models -RANDOM_SEED=42 -``` +### Project Structure -Load them in your code: -```python -from dotenv import load_dotenv -load_dotenv() +``` +ai-base-template/ +├── ai_base_template/ # Your service code goes here +│ ├── __init__.py +│ └── main.py # Simple starting point +├── tests/ # Comprehensive test suite +│ └── test_main.py # Example test patterns +├── research/ # Notebooks and experiments +│ └── EDA.ipynb # Exploratory work stays here +├── Makefile # All automation commands +├── pyproject.toml # Modern Python configuration +└── CLAUDE.md # Detailed development guide ``` -## Testing Strategy +### Development Best Practices -The template includes three test levels: +1. **Type Everything** - Use type hints to catch errors before runtime +2. **Test Defensively** - Assume inputs will be malicious or malformed +3. **Validate Early** - Check your assumptions at system boundaries +4. **Fail Gracefully** - Always have a fallback plan +5. **Measure Everything** - You can't improve what you don't measure -```python -@pytest.mark.unit # Fast, isolated tests -@pytest.mark.functional # Feature/workflow tests -@pytest.mark.integration # Tests with external dependencies -``` +## 📊 Why Infrastructure Matters -Run specific test types: -```bash -make test -make test-functional -make test-integration -``` +### Common Production Failures This Template Helps Prevent -## Starting Your Project +**Model Drift** → Solution: Structured monitoring and versioning patterns +**Cost Spirals** → Solution: Resource limits and budget tracking hooks +**Latency Spikes** → Solution: Async patterns and timeout management +**Data Quality Issues** → Solution: Input validation and sanitization patterns +**Deployment Failures** → Solution: Comprehensive testing and CI/CD automation -1. **Rename the package**: Change `ai_base_template` to your project name -2. **Update pyproject.toml**: Set your project name, version, and description -3. **Clean up examples**: Remove the example code in `main.py` -4. **Start building**: Add your own modules and logic -5. **Document as you go**: Update this README with your project specifics +## 🎓 Who Should Use This Template -## Best Practices Included +### Senior Engineers New to AI +Start with a solid engineering foundation while learning AI concepts. The template provides the safety rails you're accustomed to in production systems. -- ✅ Modern Python packaging with uv -- ✅ Comprehensive .gitignore -- ✅ Pre-configured linting and formatting -- ✅ Type checking setup -- ✅ Test structure with markers -- ✅ Makefile automation -- ✅ Clean project layout -- ✅ Development guide (CLAUDE.md) +### AI Engineers Moving to Production +Stop reinventing infrastructure. Focus on your models while using battle-tested patterns for the production wrapper. -## Tips +### Technical Leaders +Give your team a consistent, production-ready starting point that embodies engineering best practices from day one. -- Use `make validate-branch` before every commit -- Keep dependencies in `pyproject.toml` -- Write tests as you code -- Use type hints everywhere -- Check CLAUDE.md for detailed guidelines +## 📚 Learn More -## License +### Core Methodology +- [A Production-First Approach to AI Engineering](https://aienhancedengineer.substack.com/p/a-production-first-approach-to-ai) - The article that inspired this template + +### Production AI Engineering +- [Google's Rules for ML](https://developers.google.com/machine-learning/guides/rules-of-ml) - Engineering discipline for ML systems +- [Hidden Technical Debt in ML Systems](https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf) - Foundational NIPS paper + +### Technologies Used +- [FastAPI](https://fastapi.tiangolo.com/) - Modern Python web framework +- [Pydantic](https://docs.pydantic.dev/) - Data validation using type annotations +- [uv](https://docs.astral.sh/uv/) - Modern Python package management + +## 🤝 Contributing + +This template embodies battle-tested patterns from production AI systems. When contributing, prioritize: + +1. **Reliability over features** +2. **Simplicity over cleverness** +3. **Documentation over assumptions** +4. **Tests over trust** + +## 📄 License Apache License 2.0 - See [LICENSE](LICENSE) file for details. --- -Built to help you start AI/ML projects faster 🚀 \ No newline at end of file +**Remember:** The hardest part of AI isn't the algorithms—it's making them work reliably in production. This template gives you a head start on that challenge. + +*"The best AI is the AI that works."* \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index 5dac2d3..58b6c48 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "ai-base-template" version = "0.2.1" -description = "AI Base Template - Python service with AI capabilities" +description = "Production-first AI engineering template - Reliable infrastructure for deploying AI systems at scale" authors = [{name="Leopoldo Garcia Vargas", email="lk13.dev@gmail.com"}] requires-python = ">=3.12" license = {text = "Apache-2.0"}