ORES-Inspect

Try ORES-Inspect on Toolforge: https://ores-inspect.toolforge.org/

Read our summary paper, presented at Wiki Workshop 2024: https://arxiv.org/abs/2406.08453

Repository containing code for auditing the ORES edit quality models, including the ORES-Inspect tool.

Read more about this project and leave feedback at Research:ORES Inspect: A technology probe for machine learning audits on enwiki.

For development setup, deployment instructions, and a list of modernization TODOs, see DEVELOPMENT.md.

Repository contents:

• audit_web_client - All code for the front-end and back-end of ORES-Inspect. See DEVELOPMENT.md for development details, including how to run ORES-Inspect locally and how to contribute. The ToolsDB OIDB SQL lives at audit_web_client/flask/sql.
• src - Python data-processing pipeline for downloaded dumps, organized by stage (data_extraction, ores_scoring, text_retrieval, text_computation, data_shift, revscoring_utils).
• experiments - Research code that is not part of the production pipeline:
  • notebooks - Jupyter Notebooks analyzing static Wikipedia data dumps.
  • dump_download - Scripts for downloading dumps from dumps.wikimedia.org.
  • revert_prediction - Code for training and running revert-prediction models.
  • reu2021 - Analysis conducted during the 2021 REU.
  • figures - Matplotlib or other figures generated by analysis code.

See DEVELOPMENT.md for the full repository layout.

Timeline

An initial version of this project started in January 2020. A revised version of this project, and development of ORES-Inspect, started in January 2021. ORES-Inspect was presented at the Wiki Workshop in June 2024.

Citation

To cite this work, cite our Wiki Workshop 2024 paper:

Zachary Levonian, Lauren Hagen, Lu Li, Jada Lilleboe, Solvejg Wastvedt, Aaron Halfaker, Loren Terveen. 2023. ORES-Inspect: A technology probe for machine learning audits on enwiki. In Wiki Workshop 2024, Online. DOI:https://doi.org/10.48550/arXiv.2406.08453