Add FAQ automation system with GitHub Actions integration

[frederick-douglas-pearce]

FAQ Automation System with AI-Powered Triage

This PR introduces a comprehensive automated FAQ management system that uses Retrieval-Augmented Generation (RAG) and LLM-based triage to intelligently process new FAQ proposals submitted via GitHub issues.

🎯 Overview

The system automatically analyzes new FAQ proposals and determines whether to:

• NEW: Create a new FAQ entry (question not covered)
• UPDATE: Merge proposal with existing FAQ (adds valuable context)
• DUPLICATE: Mark as duplicate (already fully answered)

🚀 Features

AI-Powered Decision Making

• Uses minsearch for semantic retrieval of similar FAQs
• Leverages OpenAI GPT-4 with structured outputs for intelligent triage
• Analyzes proposals against top 5 most similar existing FAQs
• Makes context-aware decisions with detailed rationale

Automated Workflow

• GitHub issue template for structured FAQ proposals
• Automatic PR creation for NEW/UPDATE actions
• Automatic issue closure with explanation for DUPLICATE actions
• Complete git workflow (branch creation, commits, PR generation)

Developer Experience

• Comprehensive test suite (unit + integration tests)
• CLI tool for local testing and debugging
• Detailed documentation (README + CONTRIBUTING)
• Type-safe with Pydantic validation

📦 What's Included

1. FAQ Automation Module (faq_automation/)

Core Functions (core.py)

• parse_frontmatter(): Extract YAML frontmatter from markdown
• write_frontmatter(): Write structured FAQ files
• read_questions(): Load and parse all FAQs from a course
• generate_document_id(): Create collision-resistant 10-char IDs
• find_question_files(): Map document IDs to file paths
• find_largest_sort_order(): Determine next sort order number

RAG Agent (rag_agent.py)

• FAQAgent: Main agent class for processing proposals
• FAQDecision: Pydantic model for structured LLM outputs
• process_faq_proposal(): Convenience function for single proposals
• Implements complete RAG pipeline (retrieval → analysis → decision)

GitHub Actions Integration (actions.py)

• create_new_faq_file(): Generate new FAQ markdown files
• update_existing_faq_file(): Update existing FAQ content
• generate_pr_body(): Create detailed PR descriptions
• generate_duplicate_comment(): Create helpful duplicate comments

CLI Tool (cli.py)

• Command-line interface for GitHub Actions
• Issue body parsing with validation
• JSON output for workflow consumption
• Error handling and logging

2. GitHub Integration

Issue Template (.github/ISSUE_TEMPLATE/faq-proposal.yml)

Fields:

• Course selection (machine-learning-zoomcamp)
• Question (required)
• Answer (required)
• Validation checklist

Workflow (.github/workflows/faq-automation.yml)

Trigger: Issue opened with 'faq-proposal' label

Steps:

1. Extract question/answer from issue
2. Run FAQ automation CLI
3. Handle NEW/UPDATE: Create PR
4. Handle DUPLICATE: Comment and close
5. Error handling with notifications

3. Testing

Unit Tests

• test_faq_automation.py: Core function tests (frontmatter parsing, document ID generation, search result filtering)
• test_faq_actions.py: Action function tests (PR body generation, duplicate comment generation)
• test_cli_parsing.py: CLI parsing tests (issue body parsing, multi-line content handling, error cases)

4. Documentation

README.md Updates

• Complete feature overview
• Quick start guide
• Architecture documentation
• FAQ automation explanation
• Development setup instructions

CONTRIBUTING.md (New)

• Step-by-step contributor guide
• FAQ proposal best practices
• Writing guidelines for Q&A
• File structure explanation
• Manual contribution workflow

5. Dependencies

New Dependencies (added to pyproject.toml)

• minsearch>=0.4.1 - Lightweight text search
• openai>=1.0.0 - LLM integration
• pydantic>=2.0.0 - Data validation

🔄 How It Works

User Flow

1. User creates GitHub issue using "FAQ Proposal" template
2. Fills out course, question, and answer fields
3. Submits issue (automatically gets faq-proposal label)

Automation Flow

Issue Created
    ↓
Extract Q&A → Load Existing FAQs → Build Search Index
    ↓
Search Similar FAQs (top 5) → Send to GPT-4 with Prompt
    ↓
LLM Decision (NEW/UPDATE/DUPLICATE)
    ↓
├─ NEW: Create branch → Generate file → Commit → Push → Create PR
├─ UPDATE: Create branch → Modify file → Commit → Push → Create PR
└─ DUPLICATE: Post comment with link → Close issue

LLM Decision Process

The system sends GPT-4:

• The proposed Q&A
• Top 5 similar existing FAQs
• Available course sections
• Detailed instructions and rules

GPT-4 returns structured output with action, rationale, document_id, section_id, section_rationale, order, question, proposed_content, filename_slug, and warnings.

🛠️ Technical Details

RAG Pipeline

1. Indexing: minsearch indexes all FAQs with text fields (section, question, answer)
2. Retrieval: Semantic search finds top-K similar FAQs
3. Augmentation: Search results + metadata sent to LLM
4. Generation: LLM produces structured decision with reasoning

File Naming Convention

{NNN}_{document_id}_{slug}.md
001_abc1234567_when-does-course-start.md

Frontmatter Structure

---
id: abc1234567
question: 'When does the course start?'
sort_order: 1
---

📋 Setup Requirements

Before Merging

1. Add OpenAI API Key

   • Go to Settings → Secrets and variables → Actions
   • Add OPENAI_API_KEY secret

2. Verify Permissions

   • Workflow needs: contents: write, issues: write, pull-requests: write
   • Already configured in workflow file

After Merging

1. Test the Workflow

   • Create a test issue using the FAQ Proposal template
   • Verify automation runs correctly
   • Check PR creation or duplicate detection

2. Monitor Initial Runs

   • Review first few automated PRs
   • Adjust prompts if needed
   • Fine-tune decision thresholds

🧪 Testing

Run the test suite:

# All tests
make test

# Unit tests only
make test-unit

# Specific test file
pytest tests/unit/test_faq_automation.py -v

Test locally with CLI:

# Create test issue file
cat > test_issue.txt <<'TESTEOF'
### Question
How do I test the FAQ automation?

### Answer
Create a test issue using the FAQ Proposal template.
TESTEOF

# Run automation
python -m faq_automation.cli \
  --issue-body "$(cat test_issue.txt)" \
  --issue-number 999 \
  --course machine-learning-zoomcamp

🎓 Course Support

Initial Support: machine-learning-zoomcamp

Future Expansion: The system is designed to support all courses. To add a new course:

1. Update issue template dropdown
2. Ensure course has _metadata.yaml
3. Workflow automatically handles any course directory

📊 Expected Impact

For Contributors

• Lower barrier: Simple issue form vs. understanding repo structure
• Faster feedback: Immediate automated analysis
• Better quality: AI helps refine questions and placement

For Maintainers

• Reduced manual work: Auto-triage and PR creation
• Consistency: Structured decisions with rationale
• Scalability: Handles multiple simultaneous proposals

For Users

• Better FAQs: Duplicate detection prevents redundancy
• Richer content: UPDATE action merges knowledge
• Faster updates: Automated process speeds up FAQ additions

🔍 Example Outputs

NEW Decision PR

Title: [FAQ Bot] NEW: How do I troubleshoot Docker networking issues?

Body:
✨ FAQ NEW

Course: machine-learning-zoomcamp
Section: module-1 (Docker-related troubleshooting fits in Module 1)
Related Issue: #42

Question: How do I troubleshoot Docker networking issues?

Decision Rationale: This specific networking troubleshooting scenario is not covered in existing FAQs.

Placement Details:
- Section ID: module-1
- Sort Order: End of section
- Filename Slug: troubleshoot-docker-networking-issues

DUPLICATE Comment

🔄 Duplicate FAQ Entry

Thank you for your proposal! After analyzing existing FAQs, this question appears to already be covered.

Matching FAQ
Question: When does the Machine Learning Zoomcamp course start?
Document ID: 9e508f2212
Section: general

Rationale: The existing FAQ fully answers when the course starts, including registration details and Telegram channel information.

Where to Find This FAQ
- Live Site: https://datatalks.club/faq/machine-learning-zoomcamp.html#9e508f2212
- Source File: _questions/machine-learning-zoomcamp/general/

---
🤖 This issue has been automatically closed by FAQ Bot.
If you believe this is an error, please reopen and mention a maintainer.

🚦 Breaking Changes

None - this is a new feature addition.

📝 Notes

• Workflow only triggers on issues with faq-proposal label (applied automatically by template)
• PRs created by FAQ Bot require manual review and approval (intentionally not auto-merged)
• The system currently uses GPT-4; can be changed to other models via --model parameter
• All file operations preserve existing frontmatter (IDs, custom fields, etc.)

✅ Checklist

• Code implements all proposed features
• Comprehensive test coverage added
• Documentation updated (README + CONTRIBUTING)
• Dependencies added to pyproject.toml
• GitHub Actions workflow configured
• Issue template created
• No breaking changes to existing functionality

🔗 Related

• Implements functionality from notebooks/rag.ipynb
• Complements existing static site generator (generate_website.py)
• Uses same frontmatter format and directory structure

---

Ready to Review! 🎉

Once merged and configured with OpenAI API key, the FAQ automation system will be live and ready to process proposals.

[frederick-douglas-pearce]

Created this WIP PR using claude-code. Needs testing before it can be merged.

[alexeygrigorev]

That's a lot of code and text =)

Have you tested it?

[alexeygrigorev]

Also can you add a link to the contributing guide to the FAQ pages at the top?

And we probably need an issue template where people can select the course they are continuing to. Maybe we can have a drop down list? Not sure if it's possible

But at least we need a format so it's easy to extract it from the text

[frederick-douglas-pearce]

Thank you for your comments, @alexeygrigorev! I will work on addressing them. FYI, this is just an early draft that I added, made with claude-code, and needs a lot of testing still. I've already caught a few bugs but many more to come I'm sure.
Note, another student is interested in collaborating so I wanted to put this very verbose PR up so they can get familiar with what is going on.
The README.md file is very long and I'm happy to shorten it, modify it, etc however you'd like. I thought claude did a good job of explaining the site in it, and as I said, someone else wants to onboard to help out, so I'm going to leave it for now, until they have a chance to read it.

[frederick-douglas-pearce]

Also can you add a link to the contributing guide to the FAQ pages at the top?

And we probably need an issue template where people can select the course they are continuing to. Maybe we can have a drop down list? Not sure if it's possible

But at least we need a format so it's easy to extract it from the text
@alexeygrigorev, I've completed these two tasks that you requested:

• I added a link to the contributing guide at the top of the FAQ pages. It is just one line of text with a link, nothing fancy. Let me know if there are changes you'd like
• An issue template is included in the .github/ISSUE_TEMPLATE folder. It includes a dropdown to select one of the four courses currently available. I've tested it by manually triggering the faq automation workflow and it worked ok. If new courses are added, they will need to be added to the template file. I could make an issue for this, and I'm sure it could be automated, but not sure that is worth the effort. Thoughts?

[alexeygrigorev]

Is it ready for merge?

[frederick-douglas-pearce]

I've made significant updates to the README.md file in response to @alexeygrigorev 's comments, including testing all code snippets included to make sure they work. Moving on to other comments next...

[frederick-douglas-pearce]

This PR is finally ready for final review and merge:

• the GH action workflow has had all file manipulation, rag chat and any other non-github functionality moved to python with extensive unit and integration tests added (all 153 tests pass)
• the readme, contributing and tests/readme files have all been updated
• all faq-automation labeled issues close automatically if action=duplicate (with explanation comment), or upon the auto-generated PR from action=new or update being merged.

[alexeygrigorev]

Thank you!