feat: download arxiv pdfs with shared rate limiting

.gitignore

@@ -25,6 +25,7 @@ mcpb-build/
 *.DS_Store
 
 # Virtual Environment
+.venv/
 venv/
 env/
 ENV/

README.md

@@ -115,13 +115,6 @@ uv tool install arxiv-mcp-server
 
 After this, the `arxiv-mcp-server` command will be available on your `PATH`.
 
-> **PDF fallback (older papers):** Most arXiv papers have an HTML version which
-> the base install handles automatically. For older papers that only have a PDF,
-> the server needs the `[pdf]` extra (pymupdf4llm). Install it with:
->
-> ```bash
-> uv tool install 'arxiv-mcp-server[pdf]'
-> ```
 You can verify it with:
 
 ```bash
@@ -246,18 +239,20 @@ without review. See [SECURITY.md](SECURITY.md) for the full security policy.
 
 ### Core Workflow
 
-The typical workflow for deep paper research is:
+The typical workflow for finding and saving papers is:
 
 ```
-search_papers → download_paper → read_paper
+search_papers → get_abstract → download_paper
 ```
 
-`list_papers` shows what you have locally. `semantic_search` searches across your local collection.
+`download_paper` saves the original arXiv PDF locally. `read_paper` and `list_papers`
+operate only on existing markdown caches; they do not parse PDFs downloaded by
+`download_paper`.
 
 ---
 
 ### 1. Paper Search
-Search arXiv with optional category, date, and boolean filters. Enforces arXiv's 3-second rate limit automatically. If rate limited, wait 60 seconds before retrying.
+Search arXiv with optional category, date, and boolean filters. Enforces arXiv's 3-second rate limit automatically. If rate limited, wait 10 minutes before retrying.
 
 ```python
 result = await call_tool("search_papers", {
@@ -272,32 +267,37 @@ result = await call_tool("search_papers", {
 Supported categories include `cs.AI`, `cs.LG`, `cs.CL`, `cs.CV`, `cs.NE`, `stat.ML`, `math.OC`, `quant-ph`, `eess.SP`, and more. See tool description for the full list.
 
 ### 2. Paper Download
-Download a paper by its arXiv ID. Tries HTML first, falls back to PDF. Stores the paper locally for `read_paper` and `semantic_search`. The response includes `content_length`, `returned_chars`, `next_start`, and `is_truncated` so clients can safely page through very large papers without mistaking client-side output caps for failed downloads.
+Download the original arXiv PDF by paper ID. The file is streamed directly from
+`https://arxiv.org/pdf/{paper_id}`, validated as a PDF, and saved locally. This
+tool does not extract text, convert to markdown, or update the semantic-search
+index. The response includes the saved path, filename, source PDF URL, and file
+size.
 
 ```python
 result = await call_tool("download_paper", {
     "paper_id": "2401.12345"
 })
 
-# For very large papers, request bounded chunks:
+# Optional: choose the output directory and filename.
 result = await call_tool("download_paper", {
     "paper_id": "2401.12345",
-    "start": 0,
-    "max_chars": 50000
+    "output_dir": "/path/to/pdfs",
+    "filename": "attention-paper.pdf"
 })
 ```
 
-> For older papers that only have a PDF, install the `[pdf]` extra: `uv tool install 'arxiv-mcp-server[pdf]'`
-
 ### 3. List Papers
-List all papers downloaded locally. Returns arXiv IDs only — use `read_paper` to access content.
+List paper IDs that already have markdown content in local storage. This does
+not list PDFs saved by `download_paper`.
 
 ```python
 result = await call_tool("list_papers", {})
 ```
 
 ### 4. Read Paper
-Read the full text of a locally downloaded paper in markdown. **Requires `download_paper` to be called first.** Use `start` and `max_chars` with the returned `next_start` value to page through large papers.
+Read the full text of a paper from an existing markdown cache. This does not
+parse PDFs saved by `download_paper`. Use `start` and `max_chars` with the
+returned `next_start` value to page through large markdown files.
 
 ```python
 result = await call_tool("read_paper", {
@@ -327,7 +327,7 @@ result = await call_prompt("deep-paper-analysis", {
 ```
 
 This prompt includes:
-- Detailed instructions for using available tools (list_papers, download_paper, read_paper, search_papers)
+- Detailed instructions for using available tools (search_papers, get_abstract, download_paper, list_papers, read_paper)
 - A systematic workflow for paper analysis
 - Comprehensive analysis structure covering:
   - Executive summary

src/arxiv_mcp_server/prompts/deep_research_analysis_prompt.py

@@ -5,18 +5,19 @@
 You are an AI research assistant tasked with analyzing academic papers from arXiv. You have access to several tools to help with this analysis:
 
 AVAILABLE TOOLS:
-1. read_paper: Use this tool to retrieve the full content of the paper with the provided arXiv ID
-2. download_paper: If the paper is not already available locally, use this tool to download it first
-3. search_papers: Find related papers on the same topic to provide context
-4. list_papers: Check which papers are already downloaded and available for reading
+1. search_papers: Find the target paper and related papers on the same topic
+2. get_abstract: Retrieve title, authors, abstract, categories, and PDF URL without downloading the full paper
+3. download_paper: Save the original arXiv PDF locally for manual inspection or downstream processing
+4. list_papers: Check which papers already have markdown content available for reading
+5. read_paper: Read an existing markdown cache when one is available
 
 <workflow-for-paper-analysis>
 <preparation>
-  - First, use the list_papers tool to check if the paper is already downloaded
-  - If not found, use the download_paper tool to retrieve it
-  - Then use the read_paper tool with the paper_id to get the full content
-  - If the paper is not found, use the search_papers tool to find related papers while you wait
-  - If you find related papers, use the download_paper tool to get the full content of the related papers and read those too
+  - First, use get_abstract for the target paper to collect reliable metadata and the abstract
+  - Use list_papers to check whether a markdown cache exists; if it does, use read_paper for full text
+  - If markdown is not available, use download_paper only to save the original PDF; do not assume it creates readable markdown
+  - Use search_papers to find related papers for context
+  - For related papers, use get_abstract first and download_paper only when the original PDF is needed
 </preparation>
 <comprehensive-analysis>
   - Executive Summary:

src/arxiv_mcp_server/prompts/handlers.py

@@ -101,14 +101,16 @@ async def get_prompt(
     elif name == "summarize_paper":
         content = (
             f"Summarize paper {paper_id}.\n\n"
-            "Use list_papers/download_paper/read_paper as needed before summarizing.\n\n"
+            "Use get_abstract for metadata and abstract. Use read_paper only if a markdown cache exists; "
+            "download_paper saves the original PDF but does not create markdown.\n\n"
             f"{SUMMARIZE_PAPER_PROMPT}"
         )
     elif name == "compare_papers":
         paper_ids = arguments.get("paper_ids", "")
         content = (
             f"Compare papers: {paper_ids}.\n\n"
-            "Use list_papers/download_paper/read_paper to gather full text for each paper.\n\n"
+            "Use get_abstract for each paper. Use read_paper only for papers with existing markdown caches; "
+            "download_paper saves original PDFs but does not create markdown.\n\n"
             f"{COMPARE_PAPERS_PROMPT}"
         )
     else:

src/arxiv_mcp_server/tools/arxiv_rate_limit.py

@@ -0,0 +1,137 @@
+"""Shared arXiv request throttling helpers."""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import time
+from contextlib import asynccontextmanager
+from typing import Callable, TypeVar
+
+import httpx
+
+from ..config import Settings
+
+logger = logging.getLogger("arxiv-mcp-server")
+settings = Settings()
+
+MIN_REQUEST_INTERVAL = 3.0
+RATE_LIMIT_WAIT_SECONDS = 600
+
+ARXIV_HEADERS = {
+    "User-Agent": (
+        f"{settings.APP_NAME}/{settings.APP_VERSION} "
+        "(https://github.com/blazickjp/arxiv-mcp-server; research tool)"
+    )
+}
+
+_last_request_time: float = 0.0
+_cooldown_until: float = 0.0
+_request_lock = asyncio.Lock()
+T = TypeVar("T")
+
+
+class ArxivCooldownError(RuntimeError):
+    """Raised when local arXiv cooldown is active."""
+
+
+def _cooldown_remaining(now: float | None = None) -> int:
+    current = time.monotonic() if now is None else now
+    return max(0, int(_cooldown_until - current))
+
+
+def _start_cooldown(reason: str) -> None:
+    """Prevent further arXiv requests for the configured cooldown period."""
+    global _cooldown_until
+
+    _cooldown_until = max(_cooldown_until, time.monotonic() + RATE_LIMIT_WAIT_SECONDS)
+    logger.warning(
+        "Entering arXiv cooldown for %s seconds: %s",
+        RATE_LIMIT_WAIT_SECONDS,
+        reason,
+    )
+
+
+def _raise_if_cooling_down() -> None:
+    remaining = _cooldown_remaining()
+    if remaining > 0:
+        raise ArxivCooldownError(
+            f"arXiv is cooling down after a rate-limit response. "
+            f"Please wait about {remaining} seconds before retrying."
+        )
+
+
+def _looks_like_rate_limit(exc: BaseException) -> bool:
+    text = str(exc).lower()
+    return any(
+        marker in text
+        for marker in (
+            "429",
+            "503",
+            "rate limit",
+            "rate-limit",
+            "too many requests",
+            "timed out",
+            "timeout",
+        )
+    )
+
+
+@asynccontextmanager
+async def arxiv_request_slot():
+    """Hold the shared arXiv request lock for one complete upstream request."""
+    global _last_request_time
+
+    async with _request_lock:
+        _raise_if_cooling_down()
+        elapsed = time.monotonic() - _last_request_time
+        if elapsed < MIN_REQUEST_INTERVAL:
+            await asyncio.sleep(MIN_REQUEST_INTERVAL - elapsed)
+        _raise_if_cooling_down()
+        _last_request_time = time.monotonic()
+        yield
+
+
+async def wait_for_arxiv_slot() -> None:
+    """Serialize outbound arXiv request starts and enforce a minimum spacing."""
+    async with arxiv_request_slot():
+        return
+
+
+async def run_sync_with_arxiv_slot(func: Callable[[], T]) -> T:
+    """Run a blocking arXiv operation while holding the shared request lock."""
+    async with arxiv_request_slot():
+        try:
+            return await asyncio.to_thread(func)
+        except Exception as exc:
+            if _looks_like_rate_limit(exc):
+                _start_cooldown(str(exc))
+            raise
+
+
+async def rate_limited_get(client: httpx.AsyncClient, url: str) -> httpx.Response:
+    """Make a GET request while respecting arXiv's rate limit policy."""
+    async with arxiv_request_slot():
+        for attempt in range(2):
+            try:
+                response = await client.get(url, headers=ARXIV_HEADERS)
+                if response.status_code in (429, 503):
+                    logger.warning(
+                        "arXiv rate limited this IP: HTTP %s", response.status_code
+                    )
+                    _start_cooldown(f"HTTP {response.status_code}")
+                    raise RuntimeError(
+                        f"arXiv is rate limiting this IP (HTTP {response.status_code}). "
+                        f"Please wait {RATE_LIMIT_WAIT_SECONDS} seconds before retrying."
+                    )
+                response.raise_for_status()
+                return response
+            except httpx.TimeoutException:
+                if attempt == 0:
+                    logger.warning("arXiv request timed out, retrying once")
+                    await asyncio.sleep(5.0)
+                    continue
+                _start_cooldown("request timeout")
+                raise
+
+    raise RuntimeError("arXiv request timed out after retry")