🚀 AI Website Modernizer

Transform any website into a modern, responsive, and SEO-optimized experience using Google Gemini AI.

🎯 Overview

AI Website Modernizer is an intelligent web transformation tool that analyzes existing websites and automatically rebuilds them with modern design principles, improved user experience, and optimized performance. Using advanced AI capabilities powered by Google Gemini, it creates cohesive, responsive websites that maintain brand identity while dramatically improving functionality.

🏗️ Core Components

Web Crawler (crawl_site.py) - Selenium-based content extraction with SSRF protection
AI Processor (remake_site_with_ai.py) - Gemini AI integration for holistic site redesign
Dashboard Interface (dashboard.py) - Streamlit UI for seamless user experience
Prompt Engineering (prompts/rebuild_prompt.txt) - Sophisticated AI instructions for site generation

✨ Key Features

🤖 Intelligent Website Analysis

Smart Content Extraction: Automatically crawls and analyzes website structure using crawl_site.py:get_page_content()
Comprehensive Data Collection: Captures HTML, CSS, images, and copy with configurable depth and page limits
SSRF Protection: Enhanced security with is_safe_url() validation
Content Understanding: AI processes all page data simultaneously for holistic redesign approach

🎨 AI-Powered Modernization

Holistic Redesign: Google Gemini AI analyzes entire site structure via gemini_generate_entire_site()
Mobile-First Approach: Automatically optimizes for responsive design and mobile user experience
SEO Optimization: Implements modern SEO best practices and performance improvements
Brand Consistency: Maintains brand identity while upgrading design language and user experience

🔧 Technical Excellence

Structured Output: Generates clean, maintainable code with global stylesheets and organized HTML
Scalable Architecture: Three-stage pipeline (Crawl → AI Processing → Generation) for reliable results
Security-First: Secure API key handling via load_gemini_api_key() and path traversal protection
Error Resilience: Comprehensive timeout handling and graceful error recovery

📊 User-Friendly Interface

Interactive Dashboard: Streamlit-powered UI with run_full_process()
Real-Time Monitoring: Live progress tracking via run_subprocess_and_log()
Direct Browser Access: Open generated websites directly in browser tabs
Process Transparency: Clear status indicators and detailed feedback

🔄 How It Works

graph LR
    A[Input Website URL] --> B[Web Crawling]
    B --> C[Content Extraction]
    C --> D[AI Analysis]
    D --> E[Design Generation]
    E --> F[Modern Website Output]

🕷️ Web Crawling: crawl_site.py:main() discovers and crawls website pages using Selenium WebDriver
📝 Content Extraction: save_page_data() structures HTML, CSS, images, and text content
🧠 AI Analysis: remake_site_with_ai.py processes all content with Google Gemini for comprehensive understanding
🎨 Design Generation: AI creates modern, responsive design with improved UX and SEO
📦 Output Generation: Produces clean, organized website files ready for deployment

📋 Prerequisites

Python 3.9+ (see requirements.txt)
Chrome browser and ChromeDriver (for Selenium WebDriver in crawl_site.py:setup_driver())
Google Gemini API key (Get one here)

🚀 Quick Start

1. Clone & Setup

git clone <repository-url>
cd ai-website-modernizer

2. Environment Setup

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install pinned dependencies for reproducible builds
pip install -r requirements.txt

# For development: install pip-tools to manage dependencies
pip install pip-tools

# To update dependencies: edit requirements.in, then run:
# pip-compile requirements.in

3. API Configuration

# Set your Google Gemini API key
export GOOGLE_GEMINI_API_KEY=your-api-key-here

# For permanent setup (recommended)
echo 'export GOOGLE_GEMINI_API_KEY=your-api-key-here' >> ~/.bashrc
source ~/.bashrc

🔑 Get your API key: Google AI Studio

4. Launch Dashboard

🚀 One-Click Startup (Recommended)

# macOS/Linux
./start.sh

# Windows
start.bat

Manual Launch

streamlit run dashboard.py

Open your browser to http://localhost:8501 and start transforming websites! 🎉

The startup scripts automatically handle virtual environment creation, dependency installation, and dashboard launch.

💻 Usage Options

🎯 Recommended: Interactive Dashboard

streamlit run dashboard.py

User-friendly interface with real-time progress tracking via dashboard.py
One-click transformation from URL input to modernized website
Live preview of transformed pages with modify_html_for_preview()
Detailed logging and status monitoring

🔧 Advanced: Command Line Interface

Crawl Website

python crawl_site.py https://example.com --max_pages 10 --depth 2

AI Transformation

python remake_site_with_ai.py example.com --model gemini-2.5-flash-preview-05-20

Available Models:

gemini-2.5-flash-preview-05-20 (default, fast)
gemini-1.5-pro-latest (comprehensive, large context)

📁 Output Structure

📦 example.com/                    # Original crawled data
 ┣ 📂 home/
 ┃ ┣ 📄 url.txt                   # Page URL
 ┃ ┣ 📄 page.html                 # Original HTML
 ┃ ┣ 📄 copy.txt                  # Extracted text content
 ┃ ┣ 📄 images.txt                # Image URLs
 ┃ ┗ 📄 css.txt                   # CSS styles
 ┣ 📂 about/
 ┃ ┗ 📄 ...
 ┗ 📄 crawl_manifest.json         # Crawl metadata

📦 example.com_ai/                 # 🎨 AI-Enhanced Website
 ┣ 📄 global_styles.css           # Modern global stylesheet
 ┣ 📂 home/
 ┃ ┣ 📄 index.html                # ✨ Modernized HTML
 ┃ ┗ 📄 original_*.txt            # Reference files
 ┗ 📂 about/
   ┗ 📄 index.html

🏗️ Architecture

Three-Stage Pipeline

🕷️ Crawl → 🧠 AI Processing → 📦 Generation

Crawler Module (crawl_site.py)
- Selenium WebDriver with setup_driver() for JavaScript-rendered sites
- Structured content extraction via get_page_content()
- Configurable depth and page limits with DEFAULT_MAX_PAGES and DEFAULT_CRAWL_DEPTH
AI Processing (remake_site_with_ai.py)
- Google Gemini integration with gemini_generate_entire_site()
- Holistic content analysis and design generation using prompts/rebuild_prompt.txt
- Enhanced path traversal protection with filename validation
Dashboard Interface (dashboard.py)
- Streamlit-powered user interface with st.set_page_config()
- Real-time progress monitoring via update_step_status()
- Manifest-based reliable inter-process communication with read_crawl_manifest()

🧠 AI & Technical Details

Prompt Engineering

Holistic Analysis: All crawled content processed simultaneously for cohesive design via prompts/rebuild_prompt.txt
Design Principles: Mobile-first responsiveness, SEO optimization, accessibility
Brand Preservation: Maintains original brand identity while modernizing experience
Structured Output: JSON-formatted response with global CSS and individual page HTML

Security & Reliability

🔒 Secure API Handling: Environment-based API key management via load_gemini_api_key()
🛡️ Path Traversal Protection: Enhanced filename validation in remake_site_with_ai.py:369-387
⏱️ Timeout Management: Prevents hanging on unresponsive pages with driver.set_page_load_timeout(30)
📊 Manifest Communication: Reliable inter-script communication via crawl_manifest.json

Performance Considerations

Context Window: Leverages large context windows (1M+ tokens) for comprehensive analysis
Scalable Architecture: Modular design supports easy enhancement and maintenance
Error Resilience: Graceful handling of network issues, API limits, and malformed content

🛠️ Troubleshooting

Common Issues & Solutions

Issue	Solution	Code Reference
Missing Dependencies	Run `pip install -r requirements.txt` in activated virtual environment	`requirements.txt`
ChromeDriver Issues	Ensure ChromeDriver matches Chrome version and is in PATH	`setup_driver()`
API Key Not Found	Verify `GOOGLE_GEMINI_API_KEY` environment variable is set	`load_gemini_api_key()`
Timeout Errors	Check internet connection; large sites may need multiple attempts	`TimeoutException`
JSON Parse Errors	AI model occasionally returns malformed JSON; retry the process	`json.JSONDecodeError`
Memory Issues	Use `gemini-2.5-flash-preview-05-20` for large sites instead of pro model	`gemini_generate_entire_site()`

Debug Mode

# Enable verbose logging
export DEBUG_MODE=1
streamlit run dashboard.py

Getting Help

📖 Documentation: Check docs/ folder for comprehensive guides
🐛 Issues: Report bugs via GitHub Issues
💬 Discussions: Join community discussions for usage questions
📧 Contact: Reach out for enterprise or custom solutions

📚 Documentation

Setup Guide - Complete installation and configuration
Architecture Guide - Technical system design
API Documentation - Function references and examples

🤝 Contributing

We welcome contributions! Here's how you can help:

Areas for Enhancement

🌐 Additional AI Models: Support for Claude, GPT-4, or other language models
🚀 Performance Optimization: Parallel processing for large sites
🎨 Design Templates: Pre-built design themes and templates
📱 Mobile Optimization: Enhanced mobile-specific optimizations
🔍 SEO Features: Advanced SEO analysis and recommendations

Development Setup

# Fork the repository and clone your fork
git clone https://github.com/your-username/ai-website-modernizer.git

# Create feature branch
git checkout -b feature/your-feature-name

# Make changes and test thoroughly
python -m pytest tests/  # When tests are added

# Submit pull request

📄 License

MIT License - see LICENSE file for details.

🚀 Quick Reference

Key Files & Functions

Main Crawler: crawl_site.py:main() - Entry point for web crawling
AI Generation: remake_site_with_ai.py:main() - AI-powered site rebuilding
Dashboard: dashboard.py:run_full_process() - Complete transformation pipeline
Content Extraction: get_page_content() - Page data extraction logic
Security Validation: is_safe_url() - SSRF protection implementation

Configuration Constants

Crawl Limits: DEFAULT_MAX_PAGES = 20, DEFAULT_CRAWL_DEPTH = 2
File Names: COPY_FILENAME = "copy.txt", HTML_FILENAME = "page.html"
AI Models: Available in remake_site_with_ai.py

Name		Name	Last commit message	Last commit date
docs		docs
prompts		prompts
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
crawl_site.py		crawl_site.py
dashboard.py		dashboard.py
remake_site_with_ai.py		remake_site_with_ai.py
requirements.in		requirements.in
requirements.txt		requirements.txt
start.bat		start.bat
start.sh		start.sh

Latest commit

History

Folders and files

Repository files navigation