Repository navigation
Paper: Voice Capture, Structured Extraction, and Multi-Source Routing: Building and Evaluating a Field-Inspection Assistant in Python - #1222
Conversation
…ting for Field Inspections”
for more information, see https://pre-commit.ci
|
Curvenote Preview
|
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a new SciPy 2026 paper submission for Carolyn Olsen (HiveGuide), including the MyST build config, manuscript text, and bibliography.
Changes:
- Added
myst.ymlproject configuration for building/exporting the paper PDF. - Added the full manuscript (
main.md) with citations, figures, and tables. - Added a dedicated bibliography file (
mybib.bib) for references cited in the manuscript.
Reviewed changes
Copilot reviewed 3 out of 6 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| papers/carolyn_olsen/myst.yml | Defines MyST project metadata, build/export settings, and citation error rules for the paper. |
| papers/carolyn_olsen/mybib.bib | Supplies BibTeX entries referenced by the manuscript’s [@...] citations. |
| papers/carolyn_olsen/main.md | Adds the paper content, including citations that rely on bibliography configuration and BibTeX parsing. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| version: 1 | ||
| extends: ../papers.yml | ||
| project: |
There was a problem hiding this comment.
I'm wondering if this is a false positive in the copilot auto PR review comments? I don't see any leading spaces here, unless I'm looking in the wrong place.
|
Hello, @scipy-conference/2026-proceedings! I just wanted to make sure I didn't miss any steps for submission, since I haven't seen a reviewer assigned? Thank you in advance! |
ileri-oluwa-kiiye
left a comment
ileri-oluwa-kiiye
left a comment
There was a problem hiding this comment.
Overall Assessment
This is a strong, well-scoped submission. HiveGuide combines a genuine open-source Python contribution with an empirical evaluation of seven source-routing strategies on a 501-query benchmark. The paper is particularly strong in being transparent about limitations, including low absolute Hit@5 scores, agent failures, and single-pass evaluation constraints.
I recommend acceptance with minor revisions.
Fit for SciPy
Clear yes. The contribution is concretely Python-based (FastAPI, pgvector, LangChain, Pydantic, LightGBM, scikit-learn, NumPy), reusable through its pluggable architecture, and framed as a domain-agnostic template rather than only a beekeeping application.
Reproducibility
Strong. The three-command reproduction path, frozen results, fixed seed, checkpointing, and public repository provide a solid foundation for reproducibility.
Writing Quality
The paper has a clear motivation, explicit contributions, minimal marketing language, and claims that are generally well-supported by experiments. The requested inline comments are primarily to improve clarity and reproducibility for readers.
| the pgvector extension [@pgvector]; orchestration uses LangChain [@langchain]; the | ||
| client is a React Native (iOS) app; and all model calls are routed through | ||
| OpenRouter: GPT-4o for intent classification, `gpt-oss-120b` for generation, and | ||
| `text-embedding-3-large` for vectors. These three lanes are the units a forker |
There was a problem hiding this comment.
The forkability argument is a strong part of the contribution. Consider adding a small table here distinguishing domain-agnostic infrastructure (capture/ingestion/assistant lanes, validation harness, routing interfaces) from beekeeping-specific components (inspection schema, corpus, routing examples). This would make the reuse story clearer for readers evaluating what they can directly adapt.
There was a problem hiding this comment.
Great idea -- I've added some text and a table here: b95807d
| unnecessary retrieval. The embedding router achieved 87.0% with no model training | ||
| (only labeled example queries), making it attractive when training data is scarce. | ||
| Heuristics captured surprising signal (73.1%) from keyword patterns alone. | ||
|
|
There was a problem hiding this comment.
The relationship between routing accuracy and Hit@5 could use additional clarification here. The supervised classifier’s high routing accuracy does not translate into dramatically higher Hit@5 because retrieval quality remains a bottleneck after the correct source is selected. A short distinction between “selecting the correct source” and “retrieving the correct chunk from that source” would help readers interpret these metrics.
| End-to-end **latency** is measured with `perf_counter` and includes retries, and | ||
| **cost** combines per-model token usage at list prices for GPT-4o, `gpt-oss-120b`, and | ||
| the Claude Haiku 4.5 judge. | ||
|
|
There was a problem hiding this comment.
The use of Claude Haiku 4.5 as an LLM judge is reasonable, but a brief justification would strengthen this section. For example, mention whether this choice was motivated by cost, availability, or prior evidence of agreement between LLM-based evaluation and human judgments. This is particularly important because groundedness scores drive one of the more surprising findings in the results.
| boosting rounds) on TF-IDF features (unigrams and bigrams, up to 5000) of 499 labeled | ||
| queries with a stratified 75/25 split. The **embedding** router indexes 20 example | ||
| queries per category (60 vectors) and uses `k=3` neighbors, tuned on a held-out | ||
| split. The **LLM** classifier uses GPT-4o with few-shot prompting; label |
There was a problem hiding this comment.
For reproducibility, could you specify the held-out split used for tuning (e.g., number of samples and how it was separated from the evaluation benchmark)?
There was a problem hiding this comment.
This is now addressed here: 4bcb815
If the clarification isn't clear enough, please let me know.
|
@ileri-oluwa-kiiye Thank you so much for your comments and suggestions! I have made edits to address your feedback and will respond to your individual comments with the relevant commits. |
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
fwkoch
added
approved
Draft submission for 2026. Thank you for your time and consideration!