Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Classify mnemonics #3

Merged
merged 15 commits into from
Oct 27, 2024
Merged

Classify mnemonics #3

merged 15 commits into from
Oct 27, 2024

Conversation

chiffonng
Copy link
Owner

@chiffonng chiffonng commented Oct 11, 2024 •
edited
Loading

Closes #1. Send mnemonic sentences to OpenAI API 4o-mini to automatically classify mnemonics into shallow-, deep-encoding, mixed.

Features:

  • Build and send batches of mnemonics
  • Iterate on classification prompt
  • Retry + backoff to avoid RateLimitError from OpenAI APi
  • Standardize and parse API responses
  • Save classification results to a file

Fixes bugs:

  • Import error when importing across Python modules
  • Responses from OpenAI are not structured
  • Responses (classifications) don't match the number of mnemonics sent to API

Misc:

  • Improve logging and error handling for all relevant modules
  • Switch to uv for project and package management

@chiffonng chiffonng self-assigned this Oct 11, 2024
@chiffonng chiffonng linked an issue Oct 11, 2024 that may be closed by this pull request
[CAP-2] Classify mnemonics #1
Closed
4 tasks
@chiffonng
Refactor .gitignore and data_processing.py
e54f7b7 
- Refactor .gitignore to exclude /temp directory and ignore all .parquet and .csv files.
- Format mnemonics more consistently
- Drop mnemonics with only 2 words or less.
- Add more error handling for path
@chiffonng
Categorize small number of mnemonics using OpenAI's API.
bc43357
@chiffonng chiffonng changed the title Categorize mnemonics. Fix #1 Categorize mnemonics. Fixes #1 Oct 16, 2024
@chiffonng chiffonng changed the title Categorize mnemonics. Fixes #1 Categorize mnemonics. Closes #1 Oct 16, 2024
@chiffonng
Separate prompts storage to YAML files
7629b21
@chiffonng
Refactor prompt config and mnemonic processing module
4d07824
@chiffonng
Switch to uv for Python package and project management
0db6af1
@chiffonng
Fix imports between Python modules inside src
8615e9f
@chiffonng
Relax .python-version and add more instructions for installation and …
a0114a7 
…API keys
@chiffonng
Add pre-commit and its configurations
d507e4a
@chiffonng
Refactor file paths and add error handling module
a807ac2
@chiffonng
Improve mnemonics classification prompts and instructions
8f65ab8
@chiffonng
Sync dependencies in requirements.txt with virtual env
6d27a5d
@chiffonng
Extend error handling and extend ruff safe-fixes in pyproject.toml
de84c21
@chiffonng
Improve standardization of classification results.
b9c7070 
- Add JSON schema as response format for OpenAI API
- Improve classification prompts
- Handle errors when OpenAI response is too long/short
- Refactor function classify_mnemonics_api
- Improve logging
@chiffonng
Fix linting workflow
4d93612
@chiffonng chiffonng marked this pull request as ready for review October 27, 2024 20:45
@chiffonng chiffonng merged commit fd18d23 into main Oct 27, 2024
1 check passed
@chiffonng chiffonng added documentation Improvements or additions to documentation feature New feature Medium priority Created by Linear-GitHub Sync labels Oct 27, 2024
@chiffonng chiffonng deleted the chiffonng/cap-2-categorize-mnemonics branch October 30, 2024 03:50
@chiffonng chiffonng changed the title Categorize mnemonics. Closes #1 Classify mnemonics Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@chiffonng chiffonng

Labels
documentation Improvements or additions to documentation feature New feature Medium priority Created by Linear-GitHub Sync
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[CAP-2] Classify mnemonics
1 participant
@chiffonng