JSCore: A Comprehensive Feature-based Benchmark for JavaScript Unit Test Generation

Introduction

JSCore is a dataset designed to support research on automated unit test generation for JavaScript and TypeScript. This repository provides a curated benchmark of real-world open-source projects together with a static analysis pipeline for extracting language features, code complexity metrics, and project metadata.

Motivated by the lack of a broadly adopted benchmark for JavaScript test generation, JSCore captures the diversity of real-world projects while providing an extensible pipeline that can be expanded with additional analyses and projects. The dataset is intended to support the evaluation of modern test generation approaches including search-based tools, LLM-based systems, and emerging agentic systems and to contribute toward a standardized benchmark for testing research on dynamically typed languages.

How to Use

1. Using the Benchmark

The dataset consists of a curated list of JS and TS repositories. All projects are already included under the Benchmark-Projects directory. Alternatively, a complete list is provided in benchmark.txt for those who wish to clone them manually. You can manage these using the provided scripts:

• Cloning: Use scripts/clone_benchmark.sh to clone all repositories listed in benchmark.txt into the Benchmark-Projects directory.
• Installation & Validation: Run scripts/install_benchmark.sh to perform npm install on the benchmarked projects making them ready for test generation evaluation.

2. Running the Evaluation Pipeline

If you wish to execute the pipeline on your own set of repositories or modify the analysis criteria:

1. Prerequisites: Ensure you have Python (3.x), Node.js and npm installed.
2. Environment Setup: Install the necessary Python and Node.js dependencies:

   pip install -r requirements.txt
   npm install

3. Repository Discovery: Run scripts/get_gh_repos.py to populate repos.txt with potential GitHub repositories.
4. Cloning: Execute scripts/clone_repos.sh to clone the discovered repositories.
5. Feature Analysis: Run evaluation_pipeline/evaluate.py to analyze the complexity and language features of the collected projects.
6. Filtering: Use evaluation_pipeline/analyse_and_filter.py to apply filtering criteria. The final project list with analysis metrics is stored in filtered_projects_with_analysis_result.csv.

3. Explorer

The project includes a web-based explorer to visualize the dataset features. You can access it via index.html or the GitHub Pages deployment.

Benchmark Projects

The following projects constitute the core benchmark dataset:

• AdminLTE
• Autoprefixer
• Bower
• Dropzone
• Express-JWT
• Extract-Text-Webpack-Plugin
• Hoodie
• JSON-Server
• Knex
• Lodash
• Markdown-Here
• MDX
• NeDB
• Release-It
• ScrollMagic
• ScrollReveal
• ShellJS
• Sitespeed.io
• Spectacle-Code-Slide
• Supertest
• Tabulator
• PostCSS
• Puppeteer
• Tabby
• Vue-Devtools
• Weex-UI
• ZY-Player
• Commander.js
• Express
• JavaScript-Algorithms
• Crawler-URL-Parser
• Delta
• Node-Dir
• Node-Glob
• Node-Graceful-FS
• Spacl-Core
• Zip-a-folder
• Beatbump
• Felte
• Pandora
• Xyflow

The complete list with specific commit hashes is available in benchmark.txt.

Licenses

This project and its accompanying code are licensed under the MIT License. Individual projects of the benchmark within the Benchmark-Projects directory are subject to their respective original licenses.