diff --git a/README.md b/README.md
index fe1d7fd..42f2e41 100644
--- a/README.md
+++ b/README.md
@@ -161,6 +161,26 @@ bash evaluate.sh
Results will be generated in `run_logs/` with detailed metrics and analysis.
+## 🖥️ Configuration UI Tool
+
+AU-Harness includes a web-based configuration UI tool to help users easily create and customize evaluation configurations without manually editing YAML files.
+
+### Features:
+- Interactive task selection from all supported categories (Speech Recognition, Paralinguistics, Audio Understanding, etc.)
+- Model configuration with preset templates for common models like GPT-4o-mini and Gemini
+- Advanced options for filtering, judge settings, and generation parameters
+- Copy to clipboard or download functionality
+
+### Usage:
+1. Navigate to the `ui/` directory
+2. Open `index.html` in your web browser
+3. Select the tasks you want to evaluate from the categorized task list
+4. Configure your models by adding model endpoints, API keys, and parameters
+5. Adjust advanced options like sample limits, language filters, and judge settings
+6. Generate the YAML configuration, then copy or download it for use with `evaluate.sh`
+
+This tool simplifies the process of setting up complex evaluation runs by providing a user-friendly interface to build `config.yaml` files, making it easier to get started with AU-Harness evaluations.
+
## 💻 Usage
AU-Harness requires setting up a running configuration file (`config.yaml`) to define your evaluation parameters. This file controls which models, datasets, and metrics are used in your evaluation.
diff --git a/ui/README.md b/ui/README.md
new file mode 100644
index 0000000..24c8d32
--- /dev/null
+++ b/ui/README.md
@@ -0,0 +1,135 @@
+# AU-Harness UI Tool
+
+A user-friendly web interface for configuring and running audio model evaluations with the AU-Harness framework.
+
+## 🚀 Quick Start
+
+1. **Open the UI**: Simply open `index.html` in your web browser
+2. **Select Tasks**: Browse task categories and select specific tasks with their metrics
+3. **Configure Models**: Choose from preset models or configure custom endpoints
+4. **Generate Config**: Preview and download the generated YAML configuration
+
+## 📋 Features
+
+### Task Selection
+- **Visual Category Navigation**: 6 task categories with clear descriptions
+- **Smart Metric Filtering**: Automatically shows supported metrics for each task
+- **Multi-Selection Support**: Select multiple tasks across different categories
+- **Real-time Feedback**: Visual indicators show selected tasks and metrics
+
+### Model Configuration
+- **Preset Models**: Quick setup for common models (GPT-4o, Gemini, Qwen)
+- **Custom Model Support**: Add any OpenAI-compatible endpoint
+- **Sharding Configuration**: Automatic model instance management
+- **Connection Validation**: Built-in endpoint testing
+
+### Advanced Options
+- **Dataset Filtering**: Control sample limits, duration ranges, and language
+- **Judge Settings**: Configure LLM judges for evaluation
+- **Generation Parameters**: Override model parameters per task
+- **Prompt Customization**: Modify system and user prompts
+
+### Configuration Management
+- **YAML Preview**: See generated configuration
+- **Export Options**: Download as YAML file or copy to clipboard
+
+## 🛠️ Technical Details
+
+### Architecture
+- **Frontend**: Vanilla HTML5, CSS3, JavaScript (ES6+)
+- **No Dependencies**: Completely self-contained, no npm packages required
+- **Responsive Design**: Works on desktop, tablet, and mobile
+- **Modern CSS**: CSS Grid, Flexbox, Custom Properties
+- **Accessibility**: WCAG 2.1 compliant with semantic HTML
+
+### File Structure
+```
+ui/
+├── index.html # Main application page with HTML comments for sections
+├── styles.css # Complete styling with CSS custom properties and section comments
+├── app.js # Application logic with detailed function comments
+├── generate_tasks.py # Script to generate tasks.js and tasks.json with docstrings and comments
+├── tasks.js # Task categories and metrics data
+├── tasks.json # Task categories and metrics data
+└── README.md # This documentation
+```
+
+### Browser Support
+- Chrome 90+
+- Firefox 88+
+- Safari 14+
+- Edge 90+
+
+## 📖 Usage Guide
+
+### 1. Selecting Tasks
+1. Click on any category card to expand it
+2. Check the boxes next to desired tasks
+3. View selected metrics in the "Selected Tasks" section
+4. Remove tasks by clicking the "Remove" button
+
+### 2. Configuring Models
+1. Choose "Preset Models" for quick setup
+2. Check boxes next to desired models
+3. Or switch to "Custom Model" tab for custom endpoints
+4. Fill in model name, endpoint, and API key
+
+### 3. Advanced Configuration
+1. Set sample limits to control evaluation size
+2. Adjust duration filters for audio length constraints
+3. Select target language for evaluation
+4. Configure additional options as needed
+
+### 4. Generating Configuration
+1. Click "Generate Config" to create YAML
+2. Review the generated configuration in the preview
+3. Click "Download YAML" to save the file
+4. Use the config with AU-Harness evaluation engine
+
+### 5. Running Evaluations
+1. Click "Run Evaluation" to start the process
+2. Monitor progress in the Results Dashboard
+3. View scores and metrics as they complete
+4. Export results for further analysis
+
+
+## 🚀 Integration with AU-Harness
+
+The generated YAML configuration is fully compatible with the AU-Harness evaluation engine. Use it as follows:
+
+```bash
+# Using the generated config
+python evaluate.py --config your-config.yaml
+
+# Or with the UI-generated file
+python evaluate.py --config au-harness-config.yaml
+```
+
+
+## 🆘 Troubleshooting
+
+### Common Issues
+
+**Q: Configuration preview is empty**
+A: Make sure you've selected at least one task and one model before generating the config.
+
+**Q: Download doesn't work**
+A: Check your browser's download settings and ensure pop-ups are allowed for this site.
+
+**Q: Styling looks broken**
+A: Ensure you're opening `index.html` directly in a browser, not through a file:// path with restrictions.
+
+### Performance Tips
+
+- For large evaluations, consider reducing sample limits initially
+- Use preset models for faster setup
+- Clear browser cache if experiencing issues with updates
+
+## 📞 Support
+
+For issues with the UI tool, please check:
+1. Browser console for JavaScript errors
+2. Network tab for any failed resource loads
+3. This documentation for usage guidance
+
+For issues with the AU-Harness framework itself, please refer to the main project documentation.
diff --git a/ui/app.js b/ui/app.js
new file mode 100644
index 0000000..2abcf93
--- /dev/null
+++ b/ui/app.js
@@ -0,0 +1,965 @@
+// AU-Harness Configuration UI - Main Application Logic
+// This file handles the dynamic loading of tasks, model configuration,
+// and YAML generation for the AU-Harness evaluation framework.
+
+// Task categories and tasks data
+let taskCategories = {};
+let taskConfigs = {};
+
+// Theme management
+function toggleTheme() {
+ const currentTheme = document.documentElement.getAttribute('data-theme');
+ const newTheme = currentTheme === 'dark' ? 'light' : 'dark';
+
+ document.documentElement.setAttribute('data-theme', newTheme);
+ localStorage.setItem('theme', newTheme);
+
+ // Update toggle button
+ const icon = document.getElementById('theme-icon');
+ const text = document.getElementById('theme-text');
+
+ if (newTheme === 'dark') {
+ icon.textContent = '☀️';
+ text.textContent = 'Light Mode';
+ } else {
+ icon.textContent = '🌙';
+ text.textContent = 'Dark Mode';
+ }
+}
+
+// Load saved theme on page load
+function loadTheme() {
+ const savedTheme = localStorage.getItem('theme') || 'light';
+ document.documentElement.setAttribute('data-theme', savedTheme);
+
+ // Update toggle button
+ const icon = document.getElementById('theme-icon');
+ const text = document.getElementById('theme-text');
+
+ if (savedTheme === 'dark') {
+ icon.textContent = '☀️';
+ text.textContent = 'Light Mode';
+ } else {
+ icon.textContent = '🌙';
+ text.textContent = 'Dark Mode';
+ }
+}
+
+function formatDisplayLabel(key) {
+ return key
+ .replace(/_/g, ' ')
+ .replace(/\b\w/g, char => char.toUpperCase());
+}
+
+function formatConfigLabel(key) {
+ // Keep config names as-is (don't format them)
+ return key;
+}
+
+function sanitizeId(value) {
+ return value.replace(/[^a-zA-Z0-9_-]/g, '-');
+}
+
+// Load task categories from tasks.js (loaded via script tag)
+function loadTaskCategories() {
+ try {
+ // Check if TASKS_DATA is available (loaded from tasks.js)
+ if (typeof window.TASKS_DATA === 'undefined') {
+ throw new Error('TASKS_DATA not found. Please regenerate tasks.js by running generate_tasks.py');
+ }
+
+ const data = window.TASKS_DATA;
+ taskCategories = {};
+ taskConfigs = {};
+
+ Object.entries(data).forEach(([key, value]) => {
+ if (value && typeof value === 'object' && value.tasks) {
+ taskCategories[key] = value;
+ } else if (value && typeof value === 'object' && value.category) {
+ taskConfigs[key] = value;
+ }
+ });
+ } catch (error) {
+ console.error('Failed to load tasks data:', error);
+ alert('Failed to load tasks metadata. Please regenerate tasks.js via generate_tasks.py.');
+ throw error;
+ }
+}
+
+// Preset models configuration
+// These are common model configurations that users can quickly load
+const presetModels = {
+ "gpt-4o-mini": {
+ name: "gpt-4o-mini-audio-preview",
+ inference_type: "openai",
+ url: "${AZURE_ENDPOINT_URL}",
+ auth_token: "${AZURE_AUTH_TOKEN}",
+ api_version: "2025-01-01-preview",
+ delay: 100,
+ retry_attempts: 10,
+ timeout: 60,
+ batch_size: 300,
+ chunk_size: 30
+ },
+ "gemini-2.5-flash": {
+ name: "gemini-2.5-flash",
+ inference_type: "gemini",
+ location: "${GOOGLE_CLOUD_LOCATION}",
+ project_id: "${GOOGLE_CLOUD_PROJECT}",
+ model: "google/gemini-2.5-flash",
+ reasoning_effort: "medium",
+ delay: 150,
+ retry_attempts: 5,
+ timeout: 300,
+ batch_size: 100,
+ chunk_size: 30240
+ },
+ "qwen-2.5-omni": {
+ name: "qwen-2.5-omni",
+ inference_type: "vllm",
+ url: "${VLLM_ENDPOINT_URL}",
+ auth_token: "${VLLM_AUTH_TOKEN}",
+ delay: 180,
+ retry_attempts: 8,
+ timeout: 120,
+ batch_size: 50,
+ chunk_size: 40
+ }
+};
+
+// Application state
+// Holds the current configuration of selected tasks and models
+const state = {
+ selectedTasks: [],
+ models: [],
+ advancedOptions: {
+ sample_limit: 500,
+ min_duration: 1.0,
+ max_duration: 60.0,
+ language: "en",
+ accented: false,
+ metric_aggregation: "average",
+ judge_api_version: "",
+ judge_prompt_model_override: "",
+ judge_model: "gpt-4o-mini",
+ judge_type: "openai",
+ judge_api_endpoint: "${ENDPOINT_URL}",
+ judge_api_key: "${AUTH_TOKEN}",
+ judge_concurrency: 16,
+ judge_temperature: 0.0,
+ generation_params_override: "",
+ prompt_overrides: ""
+ }
+};
+
+let modelCount = 1;
+
+function addSelectedTask(entry) {
+ const exists = state.selectedTasks.some(
+ task =>
+ task.identifier === entry.identifier &&
+ task.metric === entry.metric &&
+ task.category === entry.category
+ );
+ if (!exists) {
+ state.selectedTasks.push(entry);
+ }
+}
+
+function removeSelectedTask(predicate) {
+ state.selectedTasks = state.selectedTasks.filter(task => !predicate(task));
+}
+
+function updateCategoryCardState(categoryCard) {
+ if (!categoryCard) return;
+ const hasCheckedTask = categoryCard.querySelector('.task-checkbox:checked');
+ const hasCheckedConfig = categoryCard.querySelector('.task-config-checkbox:checked');
+ if (hasCheckedTask || hasCheckedConfig) {
+ categoryCard.classList.add('selected');
+ } else {
+ categoryCard.classList.remove('selected');
+ }
+}
+
+// Initialize the application
+// Called when the DOM is fully loaded
+document.addEventListener('DOMContentLoaded', function() {
+ loadTheme();
+ loadTaskCategories();
+ initializeTaskCategories();
+ initializeTaskSelectionControls();
+ initializeModelConfiguration();
+ initializeAdvancedOptions();
+ initializePreviewActions();
+});
+
+// Initialize model configuration
+// Sets up event listeners for adding models and loading examples
+function initializeModelConfiguration() {
+ // Add model button
+ document.getElementById('add-model-btn').addEventListener('click', addNewModel);
+
+ // Load example button
+ document.getElementById('load-example-btn').addEventListener('click', loadExampleConfig);
+}
+
+// Add new model configuration
+// Creates a new model form section dynamically
+function addNewModel() {
+ const container = document.getElementById('models-container');
+ const modelIndex = modelCount++;
+
+ const modelDiv = document.createElement('div');
+ modelDiv.className = 'model-config-item';
+ modelDiv.dataset.modelIndex = modelIndex;
+
+ modelDiv.innerHTML = `
+ Model ${modelIndex + 1}
+
+
+
+ Advanced Config
+
+
+
+
+ `;
+
+ container.appendChild(modelDiv);
+ updateRemoveButtons();
+}
+
+// Remove model configuration
+function removeModel(modelIndex) {
+ const modelDiv = document.querySelector(`[data-model-index="${modelIndex}"]`);
+ if (modelDiv) {
+ modelDiv.remove();
+ updateRemoveButtons();
+ }
+}
+
+// Update remove buttons visibility
+// Hides remove button when only one model remains
+function updateRemoveButtons() {
+ const models = document.querySelectorAll('.model-config-item');
+ document.querySelectorAll('.remove-model-btn').forEach(btn => {
+ btn.style.display = models.length > 1 ? 'block' : 'none';
+ });
+}
+
+// Load example configuration
+// Fills the first model with a sample configuration
+function loadExampleConfig() {
+ const exampleConfig = {
+ name: "gpt-4o-mini-audio-preview",
+ displayName: "gpt-4o-mini-audio-preview-1",
+ inferenceType: "openai",
+ endpoint: "https://your-endpoint.openai.azure.com",
+ apiKey: "your-api-key-here",
+ authToken: "${AZURE_AUTH_TOKEN}",
+ apiVersion: "2025-01-01-preview",
+ location: "",
+ projectId: "",
+ delay: 100,
+ retry: 10,
+ timeout: 60,
+ batchSize: 300,
+ chunkSize: 30,
+ reasoningEffort: ""
+ };
+
+ // Fill the first model with example data
+ const firstModel = document.querySelector('.model-config-item');
+ if (firstModel) {
+ firstModel.querySelector('.model-name').value = exampleConfig.name;
+ firstModel.querySelector('.model-display-name').value = exampleConfig.displayName;
+ firstModel.querySelector('.model-inference-type').value = exampleConfig.inferenceType;
+ firstModel.querySelector('.model-endpoint').value = exampleConfig.endpoint;
+ firstModel.querySelector('.model-api-key').value = exampleConfig.apiKey;
+ firstModel.querySelector('.model-auth-token').value = exampleConfig.authToken;
+ firstModel.querySelector('.model-api-version').value = exampleConfig.apiVersion;
+ firstModel.querySelector('.model-location').value = exampleConfig.location;
+ firstModel.querySelector('.model-project-id').value = exampleConfig.projectId;
+ firstModel.querySelector('.model-delay').value = exampleConfig.delay;
+ firstModel.querySelector('.model-retry').value = exampleConfig.retry;
+ firstModel.querySelector('.model-timeout').value = exampleConfig.timeout;
+ firstModel.querySelector('.model-batch-size').value = exampleConfig.batchSize;
+ firstModel.querySelector('.model-chunk-size').value = exampleConfig.chunkSize;
+ firstModel.querySelector('.model-reasoning-effort').value = exampleConfig.reasoningEffort;
+ }
+}
+
+// Collect model configurations
+// Gathers all model data from the form into an array
+function collectModelConfigurations() {
+ const models = [];
+ document.querySelectorAll('.model-config-item').forEach(modelDiv => {
+ const model = {
+ name: modelDiv.querySelector('.model-name').value.trim(),
+ displayName: modelDiv.querySelector('.model-display-name').value.trim(),
+ inferenceType: modelDiv.querySelector('.model-inference-type').value,
+ endpoint: modelDiv.querySelector('.model-endpoint').value.trim(),
+ apiKey: modelDiv.querySelector('.model-api-key').value.trim(),
+ authToken: modelDiv.querySelector('.model-auth-token').value.trim() || undefined,
+ apiVersion: modelDiv.querySelector('.model-api-version').value.trim() || undefined,
+ location: modelDiv.querySelector('.model-location').value.trim() || undefined,
+ projectId: modelDiv.querySelector('.model-project-id').value.trim() || undefined,
+ delay: parseInt(modelDiv.querySelector('.model-delay').value) || 100,
+ retry: parseInt(modelDiv.querySelector('.model-retry').value) || 8,
+ timeout: parseInt(modelDiv.querySelector('.model-timeout').value) || 30,
+ batchSize: parseInt(modelDiv.querySelector('.model-batch-size').value) || 1,
+ chunkSize: parseInt(modelDiv.querySelector('.model-chunk-size').value) || 30,
+ reasoningEffort: modelDiv.querySelector('.model-reasoning-effort').value || undefined
+ };
+
+ // Only add if required fields are filled
+ if (model.name && model.displayName && model.inferenceType && model.endpoint && model.apiKey) {
+ models.push(model);
+ }
+ });
+
+ return models;
+}
+
+// Initialize task selection UI
+// Populates the task categories and their tasks
+function initializeTaskCategories() {
+ const container = document.getElementById('task-categories');
+ container.innerHTML = '';
+
+ Object.entries(taskCategories).forEach(([categoryKey, category]) => {
+ const categoryCard = document.createElement('div');
+ categoryCard.className = 'category-card expanded';
+ categoryCard.dataset.category = categoryKey;
+
+ const tasksMarkup = Object.entries(category.tasks).map(([taskKey, task]) => {
+ const configsForTask = task.configs || [];
+ const hasConfigs = configsForTask.length > 0;
+
+ if (hasConfigs) {
+ // Smart expand: auto-expand only if exactly 1 config
+ const shouldAutoExpand = configsForTask.length === 1;
+ const expandedClass = shouldAutoExpand ? 'expanded' : '';
+
+ // Create config options with metrics displayed next to each config
+ const configOptions = configsForTask.map(configKey => {
+ const configId = `config-${sanitizeId(`${categoryKey}-${taskKey}-${configKey}`)}`;
+ const metricsMarkup = task.metrics && task.metrics.length > 0
+ ? `${task.metrics.join(', ')}`
+ : '';
+ return `
+
+
+
+
+ `;
+ }).join('');
+
+ // Show bulk action buttons only if more than 1 config
+ const bulkActionsMarkup = configsForTask.length > 1 ? `
+
+
+
+
+ ` : '';
+
+ return `
+
+
+
+ ${bulkActionsMarkup}
+
+ ${configOptions}
+
+
+
+
+
+
+ `;
+ }).join('');
+
+ categoryCard.innerHTML = `
+ ${category.name}
+ ${category.description}
+
+ ${tasksMarkup}
+
+ `;
+
+ container.appendChild(categoryCard);
+ });
+
+ document.querySelectorAll('.task-checkbox').forEach(checkbox => {
+ checkbox.addEventListener('change', handleTaskSelection);
+ });
+ document.querySelectorAll('.task-config-checkbox').forEach(checkbox => {
+ checkbox.addEventListener('change', handleConfigSelection);
+ });
+}
+
+// Handle task selection
+// Updates the state when a task checkbox is toggled
+function handleTaskSelection(event) {
+ const checkbox = event.target;
+ const categoryKey = checkbox.dataset.category;
+ const taskKey = checkbox.dataset.task;
+ const category = taskCategories[categoryKey];
+ const task = category?.tasks?.[taskKey];
+
+ if (!task) return;
+
+ if (checkbox.checked) {
+ task.metrics.forEach(metric => {
+ addSelectedTask({
+ category: categoryKey,
+ task: taskKey,
+ taskName: task.name,
+ config: null,
+ configName: null,
+ metric,
+ identifier: taskKey
+ });
+ });
+ } else {
+ removeSelectedTask(
+ t => t.category === categoryKey && t.task === taskKey && t.identifier === taskKey
+ );
+ }
+
+ updateCategoryCardState(checkbox.closest('.category-card'));
+}
+
+function handleConfigSelection(event) {
+ const checkbox = event.target;
+ const { category: categoryKey, task: taskKey, config: configKey } = checkbox.dataset;
+ const category = taskCategories[categoryKey];
+ const task = category?.tasks?.[taskKey];
+
+ if (!task) return;
+
+ if (checkbox.checked) {
+ task.metrics.forEach(metric => {
+ addSelectedTask({
+ category: categoryKey,
+ task: taskKey,
+ taskName: task.name,
+ config: configKey,
+ configName: formatConfigLabel(configKey),
+ metric,
+ identifier: configKey
+ });
+ });
+ } else {
+ removeSelectedTask(
+ t => t.category === categoryKey && t.task === taskKey && t.identifier === configKey
+ );
+ }
+
+ updateCategoryCardState(checkbox.closest('.category-card'));
+}
+
+// Initialize advanced options
+// Binds form inputs to state variables
+function initializeAdvancedOptions() {
+ const inputs = {
+ 'sample-limit': 'sample_limit',
+ 'min-duration': 'min_duration',
+ 'max-duration': 'max_duration',
+ 'language': 'language',
+ 'accented': 'accented',
+ 'metric-aggregation': 'metric_aggregation',
+ 'judge-api-version': 'judge_api_version',
+ 'judge-prompt-model-override': 'judge_prompt_model_override',
+ 'judge-model': 'judge_model',
+ 'judge-type': 'judge_type',
+ 'judge-api-endpoint': 'judge_api_endpoint',
+ 'judge-api-key': 'judge_api_key',
+ 'judge-concurrency': 'judge_concurrency',
+ 'judge-temperature': 'judge_temperature',
+ 'generation-params-override': 'generation_params_override',
+ 'prompt-overrides': 'prompt_overrides'
+ };
+
+ Object.entries(inputs).forEach(([id, stateKey]) => {
+ const element = document.getElementById(id);
+ if (element) {
+ element.addEventListener('change', function() {
+ if (element.type === 'checkbox') {
+ state.advancedOptions[stateKey] = element.checked;
+ } else {
+ state.advancedOptions[stateKey] = element.value;
+ }
+ });
+ // Set initial value
+ if (element.type === 'checkbox') {
+ element.checked = state.advancedOptions[stateKey];
+ } else {
+ element.value = state.advancedOptions[stateKey];
+ }
+ }
+ });
+}
+
+// Initialize preview actions
+// Sets up event listeners for config generation, copy, and download
+function initializePreviewActions() {
+ document.getElementById('generate-config').addEventListener('click', generateConfig);
+ document.getElementById('copy-config').addEventListener('click', copyConfig);
+ document.getElementById('download-config').addEventListener('click', downloadConfig);
+}
+
+// Generate configuration
+// Creates the YAML config from current state
+function generateConfig() {
+ const models = collectModelConfigurations();
+
+ if (state.selectedTasks.length === 0) {
+ alert('Please select at least one task');
+ return;
+ }
+
+ if (models.length === 0) {
+ alert('Please configure at least one model');
+ return;
+ }
+
+ // Generate timestamp in YYYYMMDD_HHMMSS format
+ const now = new Date();
+ const timestamp = now.getFullYear() +
+ String(now.getMonth() + 1).padStart(2, '0') +
+ String(now.getDate()).padStart(2, '0') + '_' +
+ String(now.getHours()).padStart(2, '0') +
+ String(now.getMinutes()).padStart(2, '0') +
+ String(now.getSeconds()).padStart(2, '0');
+
+ // Generate aggregate from selected tasks/configs
+ const metricGroups = {};
+ state.selectedTasks.forEach(task => {
+ const targetKey = task.identifier || task.task;
+ if (!metricGroups[task.metric]) metricGroups[task.metric] = [];
+ if (!metricGroups[task.metric].includes(targetKey)) {
+ metricGroups[task.metric].push(targetKey);
+ }
+ });
+ const aggregate = Object.entries(metricGroups).filter(([metric, tasks]) => tasks.length > 1).map(([metric, tasks]) => [metric, tasks]);
+
+ const config = {
+ task_metric: state.selectedTasks.map(task => [
+ task.identifier || task.task,
+ task.metric
+ ])
+ };
+
+ // Add aggregate if there are grouped metrics
+ if (aggregate.length > 0) {
+ config.aggregate = aggregate;
+ }
+
+ config.filter = {
+ num_samples: state.advancedOptions.sample_limit,
+ length_filter: [state.advancedOptions.min_duration, state.advancedOptions.max_duration],
+ language: state.advancedOptions.language,
+ accented: state.advancedOptions.accented
+ };
+
+ config.judge_settings = {
+ judge_model: state.advancedOptions.judge_model,
+ judge_type: state.advancedOptions.judge_type,
+ judge_api_endpoint: state.advancedOptions.judge_api_endpoint,
+ judge_api_key: state.advancedOptions.judge_api_key,
+ judge_concurrency: state.advancedOptions.judge_concurrency,
+ judge_temperature: state.advancedOptions.judge_temperature,
+ ...(state.advancedOptions.judge_api_version && { judge_api_version: state.advancedOptions.judge_api_version }),
+ ...(state.advancedOptions.judge_prompt_model_override && { judge_prompt_model_override: state.advancedOptions.judge_prompt_model_override })
+ };
+
+ config.logging = {
+ log_file: `run_${timestamp}.log`
+ };
+
+ // Add generation params override if provided
+ if (state.advancedOptions.generation_params_override.trim()) {
+ config.generation_params_override = state.advancedOptions.generation_params_override.trim();
+ }
+
+ // Add prompt overrides if provided
+ if (state.advancedOptions.prompt_overrides.trim()) {
+ config.prompt_overrides = state.advancedOptions.prompt_overrides.trim();
+ }
+
+ config.models = models.map((model, index) => {
+ const modelConfig = {
+ name: model.displayName,
+ inference_type: model.inferenceType,
+ url: model.endpoint,
+ model: model.name,
+ auth_token: model.authToken || model.apiKey,
+ delay: model.delay,
+ retry_attempts: model.retry,
+ timeout: model.timeout,
+ batch_size: model.batchSize,
+ chunk_size: model.chunkSize
+ };
+
+ // Add optional fields only if they have values
+ if (model.apiVersion) modelConfig.api_version = model.apiVersion;
+ if (model.location) modelConfig.location = model.location;
+ if (model.projectId) modelConfig.project_id = model.projectId;
+ if (model.reasoningEffort) modelConfig.reasoning_effort = model.reasoningEffort;
+
+ return modelConfig;
+ });
+
+ // Add metric_aggregation
+ if (state.advancedOptions.metric_aggregation !== "average") {
+ config.metric_aggregation = state.advancedOptions.metric_aggregation;
+ }
+
+ const yaml = generateYAML(config);
+ document.getElementById('config-preview').textContent = yaml;
+}
+
+// Generate YAML from config object
+// Recursively generates YAML string from a given object
+function generateYAML(obj, indent = 0) {
+ const spaces = ' '.repeat(indent);
+ let yaml = '';
+
+ if (Array.isArray(obj)) {
+ obj.forEach(item => {
+ if (typeof item === 'object') {
+ yaml += `${spaces}-\n`;
+ yaml += generateYAML(item, indent + 1);
+ } else {
+ yaml += `${spaces}- ${item}\n`;
+ }
+ });
+ } else if (typeof obj === 'object' && obj !== null) {
+ Object.entries(obj).forEach(([key, value]) => {
+ if (typeof value === 'object' && value !== null && !Array.isArray(value)) {
+ yaml += `${spaces}${key}:\n`;
+ yaml += generateYAML(value, indent + 1);
+ } else if (Array.isArray(value)) {
+ yaml += `${spaces}${key}:\n`;
+ yaml += generateYAML(value, indent + 1);
+ } else {
+ yaml += `${spaces}${key}: ${value}\n`;
+ }
+ });
+ }
+
+ return yaml;
+}
+
+// Download configuration
+// Saves the generated YAML config to a file
+function downloadConfig() {
+ const config = document.getElementById('config-preview').textContent;
+
+ // Generate timestamp in YYYYMMDD_HHMMSS format
+ const now = new Date();
+ const timestamp = now.getFullYear() +
+ String(now.getMonth() + 1).padStart(2, '0') +
+ String(now.getDate()).padStart(2, '0') + '_' +
+ String(now.getHours()).padStart(2, '0') +
+ String(now.getMinutes()).padStart(2, '0') +
+ String(now.getSeconds()).padStart(2, '0');
+
+ const filename = `au-harness-config-${timestamp}.yaml`;
+
+ const blob = new Blob([config], { type: 'text/yaml' });
+ const url = URL.createObjectURL(blob);
+
+ const a = document.createElement('a');
+ a.href = url;
+ a.download = filename;
+ document.body.appendChild(a);
+ a.click();
+ document.body.removeChild(a);
+ URL.revokeObjectURL(url);
+}
+
+// Copy configuration to clipboard
+// Copies the generated YAML config to the clipboard
+async function copyConfig() {
+ const config = document.getElementById('config-preview').textContent;
+ const button = document.getElementById('copy-config');
+ const originalText = button.textContent;
+
+ try {
+ if (navigator.clipboard && window.isSecureContext) {
+ // Use the Clipboard API when available
+ await navigator.clipboard.writeText(config);
+ } else {
+ // Fallback for older browsers or non-HTTPS contexts
+ const textArea = document.createElement('textarea');
+ textArea.value = config;
+ textArea.style.position = 'fixed';
+ textArea.style.left = '-999999px';
+ textArea.style.top = '-999999px';
+ document.body.appendChild(textArea);
+ textArea.focus();
+ textArea.select();
+
+ try {
+ document.execCommand('copy');
+ } finally {
+ document.body.removeChild(textArea);
+ }
+ }
+
+ // Visual feedback
+ button.textContent = '✅ Copied!';
+ button.style.background = 'var(--success-color)';
+
+ setTimeout(() => {
+ button.textContent = originalText;
+ button.style.background = '';
+ }, 2000);
+
+ } catch (err) {
+ console.error('Failed to copy: ', err);
+ button.textContent = '❌ Failed';
+ button.style.background = 'var(--error-color)';
+
+ setTimeout(() => {
+ button.textContent = originalText;
+ button.style.background = '';
+ }, 2000);
+ }
+}
+
+// Initialize task selection controls
+// Sets up event listeners for select all and reset buttons
+function initializeTaskSelectionControls() {
+ document.getElementById('select-all-tasks').addEventListener('click', selectAllTasks);
+ document.getElementById('reset-selection').addEventListener('click', resetSelection);
+}
+
+// Select all tasks
+// Toggles all task checkboxes to checked
+function selectAllTasks() {
+ const taskCheckboxes = document.querySelectorAll('.task-checkbox');
+ taskCheckboxes.forEach(checkbox => {
+ if (!checkbox.checked) {
+ checkbox.checked = true;
+ checkbox.dispatchEvent(new Event('change'));
+ }
+ });
+
+ const configCheckboxes = document.querySelectorAll('.task-config-checkbox');
+ configCheckboxes.forEach(checkbox => {
+ if (!checkbox.checked) {
+ checkbox.checked = true;
+ checkbox.dispatchEvent(new Event('change'));
+ }
+ });
+}
+
+// Reset selection
+// Toggles all task checkboxes to unchecked
+function resetSelection() {
+ const taskCheckboxes = document.querySelectorAll('.task-checkbox');
+ taskCheckboxes.forEach(checkbox => {
+ if (checkbox.checked) {
+ checkbox.checked = false;
+ checkbox.dispatchEvent(new Event('change'));
+ }
+ });
+
+ const configCheckboxes = document.querySelectorAll('.task-config-checkbox');
+ configCheckboxes.forEach(checkbox => {
+ if (checkbox.checked) {
+ checkbox.checked = false;
+ checkbox.dispatchEvent(new Event('change'));
+ }
+ });
+}
+
+// Toggle task configs visibility
+function toggleTaskConfigs(headerElement) {
+ const taskItem = headerElement.closest('.task-item');
+ const icon = headerElement.querySelector('.expand-icon');
+
+ taskItem.classList.toggle('expanded');
+ icon.textContent = taskItem.classList.contains('expanded') ? '▼' : '▶';
+}
+
+// Select all configs for a specific task
+function selectAllConfigsForTask(categoryKey, taskKey, event) {
+ event.stopPropagation();
+ const checkboxes = document.querySelectorAll(
+ `.task-config-checkbox[data-category="${categoryKey}"][data-task="${taskKey}"]`
+ );
+ checkboxes.forEach(checkbox => {
+ if (!checkbox.checked) {
+ checkbox.checked = true;
+ checkbox.dispatchEvent(new Event('change'));
+ }
+ });
+}
+
+// Deselect all configs for a specific task
+function deselectAllConfigsForTask(categoryKey, taskKey, event) {
+ event.stopPropagation();
+ const checkboxes = document.querySelectorAll(
+ `.task-config-checkbox[data-category="${categoryKey}"][data-task="${taskKey}"]`
+ );
+ checkboxes.forEach(checkbox => {
+ if (checkbox.checked) {
+ checkbox.checked = false;
+ checkbox.dispatchEvent(new Event('change'));
+ }
+ });
+}
+
+// Filter configs by pattern
+function filterByPattern(pattern) {
+ const allConfigItems = document.querySelectorAll('.config-item');
+ let matchCount = 0;
+
+ allConfigItems.forEach(item => {
+ const label = item.querySelector('label');
+ if (label && label.textContent.toLowerCase().includes(pattern.toLowerCase())) {
+ item.style.display = 'flex';
+ matchCount++;
+ } else {
+ item.style.display = 'none';
+ }
+ });
+
+ // Auto-expand tasks that have matching configs
+ document.querySelectorAll('.task-item.has-configs').forEach(taskItem => {
+ const visibleConfigs = taskItem.querySelectorAll('.config-item[style*="flex"]');
+ if (visibleConfigs.length > 0) {
+ taskItem.classList.add('expanded');
+ const icon = taskItem.querySelector('.expand-icon');
+ if (icon) icon.textContent = '▼';
+ }
+ });
+
+ // Show feedback
+ if (matchCount === 0) {
+ alert(`No configs found matching "${pattern}"`);
+ }
+}
+
+// Clear filter
+function clearFilter() {
+ const allConfigItems = document.querySelectorAll('.config-item');
+ allConfigItems.forEach(item => {
+ item.style.display = 'flex';
+ });
+
+ // Restore default expand/collapse state (only 1-config tasks expanded)
+ document.querySelectorAll('.task-item.has-configs').forEach(taskItem => {
+ const configCount = taskItem.querySelectorAll('.config-item').length;
+ const shouldExpand = configCount === 1;
+
+ if (shouldExpand) {
+ taskItem.classList.add('expanded');
+ } else {
+ taskItem.classList.remove('expanded');
+ }
+
+ const icon = taskItem.querySelector('.expand-icon');
+ if (icon) icon.textContent = shouldExpand ? '▼' : '▶';
+ });
+}
diff --git a/ui/generate_tasks.py b/ui/generate_tasks.py
new file mode 100644
index 0000000..702b0ae
--- /dev/null
+++ b/ui/generate_tasks.py
@@ -0,0 +1,225 @@
+#!/usr/bin/env python3
+"""
+Script to dynamically generate tasks.json from the tasks folder structure.
+Loads metrics directly from the YAML files under the tasks directory instead of parsing README tables.
+"""
+
+import json
+from pathlib import Path
+from typing import Dict, List
+
+import yaml
+
+
+def get_category_description(category):
+ """Return a human-friendly description for a task category."""
+ descriptions = {
+ "speech_recognition": "Tasks involving automatic speech recognition (ASR), including standard ASR, long-form ASR, and code-switching ASR.",
+ "paralinguistics": "Tasks that analyze non-verbal aspects of speech such as emotion, gender, accent, and speaker characteristics.",
+ "audio_understanding": "Tasks that require understanding of the general audio signals including but not limited to music, noise, sound.",
+ "spoken_language_understanding": "Tasks that require understanding of spoken language and/or audio information including QA, translation, summarization, and intent classification.",
+ "spoken_language_reasoning": "Tasks that require reasoning over spoken input, such as instruction following or logical/mathematical reasoning.",
+ "safety_and_security": "Tasks related to assessing model behavior around safety, robustness, and vulnerability to spoofing or adversarial content.",
+ "speech_enhancement": "Tasks related to speech quality improvement, noise detection, and audio enhancement.",
+ "speech_disorder": "Tasks related to detecting and analyzing speech disorders and voice pathologies.",
+ "phonetics": "Tasks related to phonetic analysis, phoneme recognition, and speech sound processing."
+ }
+
+ return descriptions.get(category, f"Tasks related to {category.replace('_', ' ')}.")
+
+
+def get_category_display_name(category):
+ """Get display name with emoji for category."""
+ display_names = {
+ "speech_recognition": "🗣️ Speech Recognition",
+ "paralinguistics": "🎭 Paralinguistics",
+ "audio_understanding": "🔊 Audio Understanding",
+ "spoken_language_understanding": "🧠 Spoken Language Understanding",
+ "spoken_language_reasoning": "🧩 Spoken Language Reasoning",
+ "safety_and_security": "🔐 Safety and Security",
+ "speech_enhancement": "✨ Speech Enhancement",
+ "speech_disorder": "🩺 Speech Disorder",
+ "phonetics": "📢 Phonetics"
+ }
+
+ return display_names.get(category, category.replace('_', ' ').title())
+
+
+def format_task_name(task_name):
+ """Format task name for display."""
+ return task_name.replace('_', ' ').title()
+
+
+def safe_load_yaml(yaml_path: Path) -> Dict:
+ """Safely load YAML content, returning an empty dict on failure."""
+ try:
+ with open(yaml_path, "r", encoding="utf-8") as f:
+ return yaml.safe_load(f) or {}
+ except yaml.YAMLError as exc:
+ print(f"Warning: Failed to parse YAML file {yaml_path}: {exc}")
+ except OSError as exc:
+ print(f"Warning: Failed to read YAML file {yaml_path}: {exc}")
+ return {}
+
+
+def extract_metrics_from_yaml(data: Dict) -> List[str]:
+ """Extract a list of metric names from a YAML object."""
+ metrics = []
+ yaml_metrics = data.get("metrics") if isinstance(data, dict) else None
+
+ if isinstance(yaml_metrics, list):
+ for item in yaml_metrics:
+ if isinstance(item, dict) and "metric" in item:
+ metrics.append(item["metric"])
+ elif isinstance(item, str):
+ metrics.append(item)
+
+ # Deduplicate while preserving order
+ seen = set()
+ unique_metrics = []
+ for metric in metrics:
+ if metric and metric not in seen:
+ seen.add(metric)
+ unique_metrics.append(metric)
+
+ return unique_metrics
+
+
+def collect_metrics_from_task_dir(task_dir: Path) -> List[str]:
+ """
+ Collect metrics for a task directory by inspecting its YAML files.
+ Preference is given to base.yaml if present; otherwise all YAML files under the directory
+ are scanned until metrics are found.
+ """
+ yaml_files = []
+ base_yaml = task_dir / "base.yaml"
+
+ if base_yaml.exists():
+ yaml_files.append(base_yaml)
+
+ for yaml_path in sorted(task_dir.rglob("*.yaml")):
+ if yaml_path == base_yaml:
+ continue
+ yaml_files.append(yaml_path)
+
+ for yaml_file in yaml_files:
+ data = safe_load_yaml(yaml_file)
+ metrics = extract_metrics_from_yaml(data)
+ if metrics:
+ return metrics
+
+ return []
+
+
+def collect_configs_from_task_dir(task_dir: Path) -> List[str]:
+ """Collect config identifiers from YAML files excluding base definitions."""
+ configs = []
+
+ for yaml_path in sorted(task_dir.rglob("*.yaml")):
+ # Skip base.yaml files at any level
+ if yaml_path.name.lower() == "base.yaml":
+ continue
+
+ data = safe_load_yaml(yaml_path)
+ config_name = data.get("task_name") or yaml_path.stem
+ if config_name and config_name not in configs:
+ configs.append(config_name)
+
+ return configs
+
+
+def load_task_categories_from_yaml(tasks_dir: Path):
+ """
+ Build the task categories dictionary by traversing the tasks directory.
+ Each top-level directory is treated as a category and each immediate
+ sub-directory is treated as a task whose metrics/configs are discovered from YAML.
+ Returns both the categories dictionary and a flat task metadata dictionary.
+ """
+ task_categories = {}
+ task_details = {}
+
+ for category_path in sorted(tasks_dir.iterdir(), key=lambda p: p.name):
+ if not category_path.is_dir():
+ continue
+
+ category_key = category_path.name
+ tasks = {}
+ for task_path in sorted(category_path.iterdir(), key=lambda p: p.name):
+ if not task_path.is_dir():
+ continue
+
+ metrics = collect_metrics_from_task_dir(task_path)
+ if not metrics:
+ continue
+
+ task_key = task_path.name
+
+ # Collect configs for this task
+ configs = collect_configs_from_task_dir(task_path)
+
+ task_info = {
+ "name": format_task_name(task_key),
+ "metrics": metrics
+ }
+
+ # Add configs to task_info if available
+ if configs:
+ task_info["configs"] = configs
+
+ tasks[task_key] = task_info
+
+ task_metadata = dict(task_info)
+ task_metadata["category"] = category_key
+
+ task_details[task_key] = task_metadata
+
+ if not tasks:
+ continue
+
+ task_categories[category_key] = {
+ "name": get_category_display_name(category_key),
+ "description": get_category_description(category_key),
+ "tasks": tasks
+ }
+
+ return task_categories, task_details
+
+
+def main():
+ """Main function to generate tasks.json and tasks.js from tasks folder."""
+
+ script_dir = Path(__file__).parent
+ project_root = script_dir.parent
+ tasks_dir = project_root / "tasks"
+
+ if not tasks_dir.exists():
+ print(f"Error: Tasks directory not found at {tasks_dir}")
+ return
+
+ task_categories, task_details = load_task_categories_from_yaml(tasks_dir)
+
+ if not task_categories:
+ print("No task categories were discovered from YAML files.")
+ return
+
+ combined_output = {**task_categories, **task_details}
+
+ # Generate tasks.json
+ json_output_file = script_dir / "tasks.json"
+ with open(json_output_file, 'w', encoding='utf-8') as f:
+ json.dump(combined_output, f, indent=2, ensure_ascii=False)
+
+ # Generate tasks.js for local file access (no server needed)
+ js_output_file = script_dir / "tasks.js"
+ with open(js_output_file, 'w', encoding='utf-8') as f:
+ f.write("// Auto-generated by generate_tasks.py\n")
+ f.write("// This file allows the UI to work when opening index.html directly in a browser\n")
+ f.write("window.TASKS_DATA = ")
+ json.dump(combined_output, f, indent=2, ensure_ascii=False)
+ f.write(";\n")
+
+ print(f"Successfully generated {json_output_file} and {js_output_file} with {len(task_categories)} task categories")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/ui/index.html b/ui/index.html
new file mode 100644
index 0000000..cd38b13
--- /dev/null
+++ b/ui/index.html
@@ -0,0 +1,266 @@
+
+
+
+
+
+ AU-Harness Configuration UI
+
+
+
+
+
+
+
+
+
+ 📊 Task Selection
+ Select tasks for evaluation
+
+
+
+
+
+
+
+
+
Quick Filters:
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 🤖 Model Configuration
+ Configure one or more models for evaluation. Add multiple models to compare performance.
+
+
+
+
Model 1
+
+
+
+ Advanced Config
+
+
+
+
+
+
+
+
+
+
+
+
+
+ ⚖️ Judge Configuration
+ Configure the judge model for LLM-based evaluation metrics. Required when selecting tasks with judge metrics.
+
+
+
+
+
+
+ ⚙️ Advanced Configuration
+
+ Evaluation Settings
+
+
+
+ Prompt Updates
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 📋 Configuration Preview
+
+
+
+
+
+ # Configuration will appear here...
+
+
+
+