Subreddit Personality RAG

What if every Reddit community were distilled into a single, deeply opinionated human being — and you could talk to them?

This project does exactly that. It scrapes target subreddits, processes their most significant posts and comments into a semantic vector database, uses an LLM to distill a community personality profile, and then lets you engage with that persona in real time — or pit two communities against each other in a structured, multi-round debate.

---

What This Actually Does

Most "chat with your data" RAG demos are retrieval wrappers. This project goes further:

• Personality Distillation — Community content isn't just retrieved; it's analyzed by an LLM to extract a structured behavioral profile: tone, humor style, vocabulary patterns, controversy level. The persona is the subreddit, not just a bot that quotes it.
• Debate Orchestration — Two to four subreddit personas can be pitted against each other across configurable rounds. Each participant pulls its own RAG context, tracks its prior positions across rounds, and is constrained from contradicting itself or mirroring the other side's rhetoric.
• Prompt Architecture — Keeping small models (3B–8B) in character across a multi-round adversarial debate required solving real problems: hollow personas, agreement loops, circular logic, guardrail toxicity. The solutions are documented in Prompt_optimization.md.

---

Features

Feature	Description
Automated Data Acquisition	Pulls high-signal posts and comments from any subreddit via the Reddit API, filtered by keyword, score threshold, and time window.
Semantic Embedding (RAG)	Embeds all content into a ChromaDB vector store partitioned by subreddit. Retrieval uses cosine similarity over all-MiniLM-L6-v2 embeddings.
Personality Distillation	LLM-extracted behavioral profile per subreddit: tone, humor style, vocabulary patterns, core values, controversy level. Stored in SQL, injected at inference time.
Standalone Chat Mode	One-on-one conversation with any indexed subreddit persona via a Streamlit interface. Includes conversation memory persistence and explicitly pragmatic, anti-therapist tone guardrails.
Debate Orchestrator	2–4 personas, configurable rounds, per-round RAG retrieval, cross-round stance memory, rhetorical diversity guardrails.
LLM Agnostic	Unified provider interface for Ollama (local), Anthropic, and OpenAI. Swap models via .env.
Flexible Deployment	SQLite for local development, Docker Compose + PostgreSQL for production.
Idempotent Pipeline	Safe to re-run. Deduplication is enforced at both the SQL and ChromaDB layers.

---

Architecture Overview

Reddit API (PRAW)
      │
      ▼
[Stage 1: Extraction]
  Filter by score, keywords, NSFW
  Fetch top comments per post
      │
      ▼
[Stage 2: RAG Build]
  Embed with all-MiniLM-L6-v2
  Upsert into ChromaDB (per-subreddit collection)
      │
      ▼
[Stage 3: Personality Extraction]
  LLM analyzes community content
  Outputs structured JSON profile
  Persists to subreddit_profiles table
      │
      ▼
[Streamlit App]
  ├── Chat Mode: RAG retrieval → persona prompt → stream response
  └── Debate Mode: RAG + stance memory + ideological lock → per-round generation

For the full technical breakdown, see ARCHITECTURE.md.
For the prompt engineering retrospective, see Prompt_optimization.md.

---

Quickstart

Prerequisites

• Python 3.10+
• Reddit API credentials (create an app here)
• Ollama installed locally, or an Anthropic/OpenAI API key

---

Option A: Local (SQLite + Ollama)

# 1. Clone and install
git clone <repo-url>
cd reddit-personality-RAG
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

# 2. Configure .env
REDDIT_CLIENT_ID=your_client_id
REDDIT_CLIENT_SECRET=your_client_secret
REDDIT_USER_AGENT=your_app_name

LLM_PROVIDER=ollama
LLM_MODEL=llama3.1:8b

DATABASE_URL=sqlite:///data/pipeline.db

// 3. Edit data/config.json — define your target subreddits and keywords
{
  "search_settings": {
    "target_subreddits": ["productivity", "antiwork"],
    "keywords": ["burnout", "time management", "wage theft"],
    "posts_per_keyword": 15,
    "sort_method": "top",
    "time_filter": "all"
  },
  "export_settings": {
    "include_comments": true,
    "max_comments_per_post": 3
  },
  "filter_settings": {
    "min_score": 100,
    "exclude_nsfw": true,
    "exclude_deleted_posts": true
  }
}

# 4. Run the pipeline (extracts, embeds, distills personalities)
python pipeline.py --yes

# 5. Launch the interface
python -m streamlit run src/app.py

---

Option B: Docker Compose (PostgreSQL)

# Update .env
DATABASE_URL=postgresql://pipeline_user:pipeline_pass@localhost:5432/pipeline_db

docker compose up -d db
docker compose build scraper
docker compose run scraper
python -m streamlit run src/app.py

---

Switching LLM Providers

# Anthropic
LLM_PROVIDER=anthropic
LLM_MODEL=claude-3-5-sonnet-20240620
LLM_API_KEY=sk-ant-...

# OpenAI
LLM_PROVIDER=openai
LLM_MODEL=gpt-4o
LLM_API_KEY=sk-...

---

Project Structure

reddit-personality-RAG/
├── pipeline.py                  # ETL entry point
├── src/
│   ├── app.py                   # Streamlit web interface
│   ├── extractors/
│   │   └── reddit_extractor.py  # PRAW-based data acquisition
│   ├── transformers/
│   │   ├── rag_builder.py       # Embedding + ChromaDB upsert
│   │   └── personality_extractor.py  # LLM personality distillation
│   ├── db/
│   │   ├── models.py            # SQLAlchemy schema definitions
│   │   └── vector_store.py      # ChromaDB interface
│   └── llm/
│       ├── base.py              # LLMProvider abstract class
│       ├── ollama_provider.py
│       └── anthropic_provider.py
└── data/
    ├── config.json              # Subreddit and scraping config
    ├── personality_extraction_prompt.txt
    └── chroma_db/               # Persisted vector store

---

Design Decisions & Lessons Learned

This project was built to learn RAG and agentic orchestration in practice. Some non-obvious things discovered along the way:

• Small models ignore flat trait lists. A character sheet structure with explicit sections (WHO YOU ARE, HOW YOU SPEAK, WHAT YOU NEVER SAY) is far more effective than a bulleted attribute list.
• Negative constraints backfire. Telling an 8B model "do NOT say Newsflash" causes it to fixate on the phrase more. Positive reinforcement with mandatory community-specific signature phrases crowds out generic defaults instead.
• Runtime guardrails cannot override persona-level data. If a bad phrase is baked into the stored personality profile, no amount of debate-prompt patching will remove it. Contamination must be fixed at extraction time and profiles regenerated.
• Post-generation correction is worse than pre-generation constraints at small model sizes. A reviewer LLM pass strips personality and outputs academic AI-speak. Getting the context right before generation is the correct approach.
• Debate personas collapse into agreement without an explicit ideological lock. "Organic conversation" rules don't work in an adversarial setting.
• The first speaker determines the entire debate. In a RAG-driven debate, Round 1's argument seeds the retrieval chain for all subsequent rounds. A concrete, specific opening propagates richness through the whole debate; an abstract opening keeps everything abstract. The opening statement deserves its own prompt, separate from the standard turn prompt.

The full retrospective — eight failure modes with root causes and solutions — is in Prompt_optimization.md.

---

Roadmap

• Forced novelty constraint: require each persona to introduce an argument not yet made
• Hybrid retrieval (BM25 sparse + dense) for stronger early-round context before stance memory builds up
• Embedding model upgrade (e.g., bge-large, gte-Qwen2) for richer semantic retrieval
• Score-weighted RAG context: prioritize high-upvote posts in personality extraction
• Per-debate export to shareable transcript format
• Evaluation harness: persona consistency and voice distinctiveness scoring across rounds

---

License

MIT