Pull Request: JuniperData Service Extraction & Integration Infrastructure#38
Conversation
…splitting utilities - Implement tests for deterministic dataset ID generation, ensuring consistent IDs for identical parameters and different parameters produce unique IDs. - Validate dataset ID format and hash characteristics. - Create comprehensive tests for the SpiralGenerator, covering output shapes, one-hot encoding correctness, and parameter validation. - Introduce tests for shuffle and split utilities, ensuring data integrity during shuffling and correct sizes during splitting.
- Created unit tests for dataset ID generation covering deterministic ID generation, parameter variations, and ID format validation. - Implemented unit tests for the SpiralGenerator, including output shapes, one-hot encoding correctness, and parameter validation. - Added tests for shuffle and split utilities to ensure data integrity during shuffling and splitting operations. - Updated `pyproject.toml` to include new testing dependencies and adjusted configurations. - Removed unnecessary `__pycache__` files from the source directories.
- Created junit test result XML files for both integration and unit tests. - Added a new directory for test scripts and created a symlink to it. - Implemented a script to run tests with specified parameters and configurations. - Developed a comprehensive script to execute all tests with coverage reporting options. - Enhanced the test execution process by setting up environment variables and paths.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Memory files for project context: - project_overview.md: Purpose, tech stack, dependencies - suggested_commands.md: Development commands reference - code_style_conventions.md: Naming, formatting, patterns - task_completion_checklist.md: Quality checks before commit - codebase_structure.md: Directory layout and components Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Detailed analysis of JuniperData testing infrastructure covering: - Test validity and correctness (207 tests analyzed) - CI/CD pipeline configuration issues - Pre-commit hook coverage gaps - Security scanning configuration - Prioritized recommendations for improvements Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add comprehensive development plan consolidating audit findings - Update audit reports with refined analysis and formatting - Update test files and coverage reports - Update Serena configuration and memory files Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Updated the metrics table for better readability and clarity. - Improved the formatting of issue descriptions and assessments in the audit report. - Added detailed recommendations for addressing identified issues, including silent test passes and security scan configurations. - Enhanced the priority matrix and best practices compliance review sections for clearer guidance. - Included specific commands for verification and outlined implementation risks and consequences of inaction. - Created new files for Dependabot and CodeQL workflows to improve dependency management and security analysis.
Security Remediation (Critical): - SEC-001: Bandit now blocks on medium+ severity findings - SEC-002: pip-audit uses --strict to fail on vulnerabilities - SEC-003: Add Dependabot for automated dependency updates - TST-001: Fix silent ImportError pass in test_main.py Code Quality Enforcement (High): - TST-002: Enable Flake8 linting on test code - TST-003: Enable MyPy type checking on test code - TST-004: Remove global pytest warning suppression - CFG-001: Remove E722/F401 from Flake8 ignores - Fix 7 unused imports in test files Infrastructure Improvements (Medium): - SEC-004: Pin all GitHub Actions to commit SHAs - CFG-003: Fix MyPy type errors in production code - CFG-005: Remove --continue-on-collection-errors - INF-001: Add CodeQL security analysis workflow - INF-002: Add Codecov coverage upload integration - INF-003: Add slow test job with weekly schedule Best Practice Refinements (Low): - L-01: Add pyupgrade hook for Python 3.11+ syntax - L-02: Add shellcheck hook for shell script linting All 207 tests passing. See TEST_SUITE_CICD_ENHANCEMENT_DEVELOPMENT_PLAN.md for full implementation details. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…dd new planning document
- Added assertions to check for non-None values in dataset metadata and artifact bytes across multiple test files to enhance robustness. - Updated test_api_app.py to safely access dynamic route attributes using getattr(). - Included type ignore comments in test_spiral_generator.py for invalid algorithm parameter validation. - Ensured artifact bytes are checked for None in test_storage.py for both in-memory and local filesystem stores. - Updated integration development plan to reflect the completion of mypy type error fixes and flake8 issues, along with the addition of a Dockerfile and health check probes.
- Created a new script `juniper_data.bash` for handling Juniper data processing with configurable parameters. - Removed the outdated `run.bash` script that contained hardcoded paths and parameters. - Updated symlink for `try.bash` to point to `juniper_data.bash`.
- Updated version numbers in ci.yml, .pre-commit-config.yaml, AGENTS.md, CHANGELOG.md, JUNIPER_DATA_API.md, __init__.py, INTEGRATION_DEVELOPMENT_PLAN.md, and pyproject.toml to reflect the new version 0.4.0. - Added detailed pull request description for JuniperData Service Extraction & Integration Infrastructure. - Enhanced CHANGELOG with new features, fixes, and modifications for version 0.4.0.
- Implemented unit tests for CachedDatasetStore, ensuring proper save, retrieve, delete, and cache functionalities. - Added unit tests for CirclesGenerator and GaussianGenerator, covering parameter validation, data generation, and class distribution. - Introduced unit tests for API security, including API key authentication and rate limiting, ensuring correct behavior under various scenarios. - Updated integration development plan to reflect new test counts and completed features.
… unit tests for new generators - Implement KaggleDatasetStore for downloading and caching datasets from Kaggle. - Implement PostgresDatasetStore for metadata storage with filesystem artifacts. - Add unit tests for CheckerboardGenerator and CsvImportGenerator, covering various scenarios and parameter validations. - Update integration development plan to reflect new features and completed tasks.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Overtoad <paul.calnon@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 149 out of 465 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Overtoad <paul.calnon@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 149 out of 465 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Overtoad <paul.calnon@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Overtoad <paul.calnon@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 149 out of 465 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 149 out of 465 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Overtoad <paul.calnon@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Overtoad <paul.calnon@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Overtoad <paul.calnon@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 149 out of 465 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Overtoad <paul.calnon@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 149 out of 465 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…fixing mock with_format
The generator's _load_and_preprocess calls ds.with_format("numpy") for bulk column
access, but test mocks didn't configure with_format(), causing np.array() to produce
empty arrays. Added formatted_ds mocks returning proper numpy data. Also added missing
n_samples handling via ds.select() in the generator.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
merging large refactor for juniper cascor |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 149 out of 465 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Overtoad <paul.calnon@gmail.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 149 out of 465 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The # nosec B615 directive was on the comment line above the flagged hf_load_dataset() call. Bandit only honors nosec on the same line as the code it suppresses. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hard `import arc_agi` in __init__.py caused ModuleNotFoundError for consumers that don't need ARC-AGI functionality (e.g. JuniperCascor), preventing all JuniperData integration tests from running. Move arc-agi from core dependencies to optional-dependencies and guard the import with try/except, exposing an ARC_AGI_AVAILABLE flag. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 149 out of 465 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 149 out of 465 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Overtoad <paul.calnon@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 149 out of 465 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| buffer = io.BytesIO() | ||
| # Ensure a stable serialization order by sorting keys before saving. | ||
| ordered_arrays = {key: arrays[key] for key in sorted(arrays.keys())} | ||
| np.savez(buffer, **ordered_arrays) # type: ignore[arg-type] # numpy stubs incomplete for **kwargs | ||
| buffer.seek(0) | ||
| return buffer.read() | ||
|
|
||
|
|
||
| def compute_checksum(arrays: dict[str, np.ndarray]) -> str: |
There was a problem hiding this comment.
The checksum is computed from the NPZ container bytes produced by np.savez(). Even with sorted keys, the resulting bytes (and therefore checksum) are not guaranteed deterministic across calls because the underlying ZIP metadata (e.g., timestamps) can vary. Additionally, other code paths serialize NPZ differently (e.g., compressed vs uncompressed), which can cause checksum mismatches depending on which serializer is used. Consider computing the checksum from a deterministic representation (e.g., sorted keys + dtype/shape + raw array bytes) and standardize NPZ serialization (compressed vs uncompressed) across the project.
| logging.basicConfig( | ||
| level=getattr(logging, settings.log_level.upper(), logging.INFO), | ||
| format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", | ||
| ) |
There was a problem hiding this comment.
Settings supports log levels like TRACE/VERBOSE/FATAL (see Settings defaults), but Python's standard logging module doesn't define those levels, so getattr(logging, ...) will silently fall back to INFO. This makes config values misleading and can cause unexpected logging behavior. Either restrict Settings.log_level to standard logging levels (DEBUG/INFO/WARNING/ERROR/CRITICAL) or explicitly register custom levels and map them consistently for both uvicorn and stdlib logging.
|
|
||
|
|
||
| _JUNIPER_DATA_API_KEYS_LIST_EMPTY: list[str] | None = None | ||
| _JUNIPER_DATA_API_KEYS_LIST_VALUES: list[str] | None = [] |
There was a problem hiding this comment.
Avoid using a mutable list literal as a module-level default (_JUNIPER_DATA_API_KEYS_LIST_VALUES = []). Even if it's not currently used as the default, it can be accidentally referenced later and mutated, causing shared state across imports. Prefer a tuple, or set it to None and use default_factory at the model field when you actually want an empty collection.
| _JUNIPER_DATA_API_KEYS_LIST_VALUES: list[str] | None = [] | |
| _JUNIPER_DATA_API_KEYS_LIST_VALUES: list[str] | None = None |
Pull Request: JuniperData Service Extraction & Integration Infrastructure
Date: 2026-02-06
Version(s): 0.1.0 → 0.4.0
Author: Paul Calnon
Status: READY_FOR_MERGE
Summary
This PR extracts JuniperData as a standalone dataset generation microservice from the Juniper monorepo, replacing the legacy JuniperCanopy codebase with a purpose-built FastAPI service. The branch delivers the complete JuniperData application (v0.1.0–v0.4.0), including 8 dataset generators, 7 storage backends, REST API with 16 endpoints, Docker containerization, a shared client library, dataset lifecycle management, comprehensive CI/CD pipeline with security scanning, and 699 tests across service and client.
SemVer Impact: MINOR (cumulative across v0.1.0–v0.4.0)
Breaking Changes: YES — Complete repository transformation from JuniperCanopy to JuniperData
Context / Motivation
JuniperData was designed as a standalone microservice to provide dataset generation capabilities to the Juniper ecosystem. Previously, dataset generation was embedded within JuniperCascor's
SpiralProblemclass. This extraction enables:X_train,y_train,X_test,y_test,X_full,y_full)JUNIPER_DATA_URLenvironment variable enables JuniperData mode in consumersRelated Documentation:
Priority & Work Status
Priority Legend
Changes
Added
Core Application (
juniper_data/)__init__.py— Package init with version (v0.3.0)__main__.py— CLI entry point with argparse (host, port, storage-path, log-level, reload)core/models.py— 11 Pydantic v2 models (DatasetMeta, CreateDatasetRequest/Response, GeneratorInfo, PreviewData, DatasetListFilter, BatchDeleteRequest/Response, UpdateTagsRequest, DatasetStats)core/split.py— Dataset shuffle and split utilities (shuffle_data(),split_data(),shuffle_and_split())core/dataset_id.py— Deterministic SHA-256 hash-based dataset ID generationcore/artifacts.py— NPZ artifact handling with checksumsGenerator Framework (
juniper_data/generators/) — 8 generators registered inGENERATOR_REGISTRYspiral/— N-spiral classification dataset generator (modern + legacy_cascor algorithms)xor/— XOR classification dataset generator (4-quadrant diagonal class assignment)gaussian/— Gaussian blobs classification dataset generator (mixture-of-Gaussians)circles/— Concentric circles classification dataset generator (binary inner/outer)checkerboard/— Checkerboard pattern classification dataset generator (2D alternating grid)csv_import/— CSV/JSON import generator for custom datasetsmnist/— MNIST and Fashion-MNIST dataset generatorarc_agi/— ARC-AGI visual reasoning tasks generator (requiresarc-agi>=0.9.0)Storage Layer (
juniper_data/storage/) — 7 backend implementationsbase.py— AbstractDatasetStorewith lifecycle methodsmemory.py—InMemoryDatasetStorefor testing and ephemeral uselocal_fs.py—LocalFSDatasetStorefor file-based persistence (JSON metadata + NPZ artifacts)cached.py—CachedDatasetStorecaching layer with error logginghf_store.py—HuggingFaceDatasetStorefor HuggingFace Hub storagekaggle_store.py—KaggleDatasetStorefor Kaggle Datasets storagepostgres_store.py—PostgresDatasetStorefor PostgreSQL storageredis_store.py—RedisDatasetStorefor Redis storageREST API (
juniper_data/api/)app.py— FastAPI application factory with lifespan management, CORS middleware, exception handlerssettings.py— Pydantic-settings configuration withJUNIPER_DATA_env prefixroutes/health.py— Health endpoints:GET /v1/health,GET /v1/health/live,GET /v1/health/readyroutes/generators.py— Generator discovery:GET /v1/generators,GET /v1/generators/{name}/schemaroutes/datasets.py— 11 dataset endpoints:POST /v1/datasets— Create dataset (deterministic caching)GET /v1/datasets— List with pagination (limit/offset)GET /v1/datasets/filter— Advanced filtering (generator, tags, date range, sample count)GET /v1/datasets/stats— Aggregate statisticsPOST /v1/datasets/batch-delete— Bulk delete (up to 100 IDs)POST /v1/datasets/cleanup-expired— TTL-based cleanupGET /v1/datasets/{id}— Metadata retrievalGET /v1/datasets/{id}/artifact— NPZ download (StreamingResponse)GET /v1/datasets/{id}/preview— JSON sample previewDELETE /v1/datasets/{id}— Single dataset deletionPATCH /v1/datasets/{id}/tags— Tag management (add/remove)Shared Client Library (
juniper_data_client/)client.py—JuniperDataClientwith all API methods, automatic retry logic, connection poolingexceptions.py— Exception hierarchy (ConnectionError, TimeoutError, NotFoundError, ValidationError)pyproject.toml— Standalone pip-installable package configurationpy.typed— PEP 561 type marker for mypy strict modeREADME.md— Comprehensive documentation with usage examplesDocker Containerization
Dockerfile— Multi-stage build (python:3.11-slim), non-root user (juniper:1000), port 8100.dockerignore— Build context exclusions (tests, docs, notes, caches)HEALTHCHECKinstruction (30s interval, 10s timeout, 5s start period, 3 retries)CI/CD Pipeline
.github/workflows/ci.yml— Full pipeline: pre-commit (Python 3.12-3.14), unit tests (80% coverage gate), integration tests, security scanning (Gitleaks, Bandit SARIF, pip-audit), slow tests (weekly), build verification.github/workflows/codeql.yml— Weekly semantic code analysis.github/dependabot.yml— Automated dependency updates (pip + GitHub Actions).pre-commit-config.yaml— 16+ hooks: black, isort, flake8 (with bugbear, comprehensions, simplify), mypy, bandit, pyupgrade (py311+), shellcheck, YAML/TOML/JSON validationTest Suite (658 tests + 41 client tests)
tests/unit/— 589 unit tests across 25 test filestests/integration/— 69 integration tests across 5 test filestests/fixtures/— Golden dataset fixtures (2-spiral, 3-spiral) for parity testingtests/conftest.py— Shared fixtures (TestClient, storage, sample datasets)juniper_data_client/tests/test_client.py— 41 client tests usingresponsesHTTP mockingDocumentation
docs/api/JUNIPER_DATA_API.md— 622-line API reference with versioning strategy, endpoint documentation, NPZ schema, client examples (Python, curl, Docker Compose)notes/INTEGRATION_DEVELOPMENT_PLAN.md— 20 work items compiled from 5 documentation sourcesnotes/test_suite_audit/— Test suite audits and CI/CD enhancement plansnotes/mcp_tools/— MCP/Serena setup documentationAGENTS.md— Complete project guide (updated from JuniperCanopy)CLAUDE.md— Symlink to AGENTS.md for Claude Code integrationInfrastructure
conf/juniper_data.conf— Application launch configuration (renamed from juniper_canopy.conf)conf/juniper_data_functions.conf— Shell functions (renamed from juniper_canopy_functions.conf)conf/conda_environment.yaml— Updated Conda environment for JuniperDataconf/script_util.cfg— Script utility configurationconf/logging_config-CASCOR.yaml— Cascor logging configurationutil/juniper_data.bash— Application launch scriptpyproject.toml— Complete project configuration (dependencies, tool settings, test markers)Changed
src/package structure withjuniper_data/for proper package discoveryFixed
# type: ignore[arg-type]for negative tests in test_spiral_generator.pygetattr()pattern for dynamic attributes in test_api_app.pyAnyandDictfrom datasets.py# noqa: E402for late imports, converted bare f-stringsjuniper_datavsjuniper-data), which causedpip-audit --strictto fail with "Dependency not found on PyPI"_load_and_preprocesscalledds.with_format("numpy")for bulk column access, but test mocks didn't configurewith_format(), returning a genericMagicMockthat produced empty arrays onnp.array(). Fixed by addingformatted_dsmocks returning proper numpy data. Also added missingn_samplessupport viads.select()in the generator.# nosec B615from the comment line abovehf_load_dataset()to inline on the call itself; bandit only honors# nosecdirectives on the same line as the flagged codeRemoved
src/) — Entire Dash dashboard application (185 files)src/backend/— Cascor integration, Redis, Cassandra, statistics, training monitor/state machinesrc/frontend/— Dashboard manager, all UI components (metrics, network visualizer, dataset plotter, etc.)src/communication/— WebSocket managersrc/demo_mode.py,src/config_manager.py,src/main.py— Application entry pointssrc/tests/— All JuniperCanopy tests (~2,900 tests)src/assets/— Logo images and iconssrc/logger/— Custom logging module.codecov.yml,.coveragerc,.markdownlint.json,.yamllint.yaml,conftest.py(root)snapshots/,demosymlink,src/logssymlinkSecurity
juniper(UID 1000)Impact & SemVer
src/) is removed and replaced with JuniperData (juniper_data/)JUNIPER_DATA_URLenvironment variable enables JuniperData mode in consumersTesting & Results
Test Summary
Coverage
Environments Tested
Verification Checklist
algorithm="legacy_cascor"matches JuniperCascor SpiralProblem statisticsn_points/noise_levelaccepted for consumer compatibilityAPI Changes
New Endpoints (All New)
/v1/health/v1/health/live/v1/health/ready/v1/generators/v1/generators/{name}/schema/v1/datasets/v1/datasets/v1/datasets/filter/v1/datasets/stats/v1/datasets/batch-delete/v1/datasets/cleanup-expired/v1/datasets/{id}/v1/datasets/{id}/artifact/v1/datasets/{id}/preview/v1/datasets/{id}/v1/datasets/{id}/tagsNPZ Artifact Schema (Data Contract)
X_train(n_train, n_features)float32y_train(n_train, n_classes)float32X_test(n_test, n_features)float32y_test(n_test, n_classes)float32X_full(n_samples, n_features)float32y_full(n_samples, n_classes)float32Files Changed
Total: 451 files changed, 19,150 insertions(+), 94,825 deletions(-):
New Components
Core Application:
juniper_data/__init__.py— Package init with versionjuniper_data/__main__.py— CLI entry point (argparse + uvicorn)juniper_data/core/models.py— 11 Pydantic data modelsjuniper_data/core/split.py— Shuffle and split utilitiesjuniper_data/core/dataset_id.py— Deterministic ID generationjuniper_data/core/artifacts.py— NPZ artifact handlingGenerators:
juniper_data/generators/spiral/generator.py— Spiral dataset generator (modern + legacy algorithms)juniper_data/generators/spiral/params.py— Spiral params with consumer aliasesjuniper_data/generators/spiral/defaults.py— Default constantsjuniper_data/generators/xor/generator.py— XOR classification generatorjuniper_data/generators/xor/params.py— XOR parameter modelStorage:
juniper_data/storage/base.py— Abstract store with lifecycle methodsjuniper_data/storage/memory.py— In-memory storejuniper_data/storage/local_fs.py— Filesystem storeAPI:
juniper_data/api/app.py— FastAPI application factoryjuniper_data/api/settings.py— Pydantic-settings configurationjuniper_data/api/routes/health.py— Health probe endpointsjuniper_data/api/routes/generators.py— Generator discovery endpointsjuniper_data/api/routes/datasets.py— Dataset CRUD + lifecycle endpointsClient Library:
juniper_data_client/client.py— REST client with retries and connection poolingjuniper_data_client/exceptions.py— Exception hierarchyjuniper_data_client/pyproject.toml— Package configurationInfrastructure:
Dockerfile— Multi-stage Docker build.dockerignore— Build context exclusions.github/workflows/ci.yml— CI/CD pipeline.github/workflows/codeql.yml— CodeQL analysis.github/dependabot.yml— Dependency automationconf/juniper_data.conf— Application launch configutil/juniper_data.bash— Launch scriptTests:
juniper_data/tests/unit/test_spiral_generator.py— 43 spiral generator testsjuniper_data/tests/unit/test_split.py— 18 data split testsjuniper_data/tests/unit/test_dataset_id.py— 16 ID generation testsjuniper_data/tests/unit/test_storage.py— 44 storage testsjuniper_data/tests/unit/test_artifacts.py— 11 artifact testsjuniper_data/tests/unit/test_api_app.py— 17 app factory testsjuniper_data/tests/unit/test_api_routes.py— 18 route testsjuniper_data/tests/unit/test_api_settings.py— 11 settings testsjuniper_data/tests/unit/test_main.py— 10 entry point testsjuniper_data/tests/unit/test_lifecycle.py— 27 lifecycle testsjuniper_data/tests/unit/test_xor_generator.py— 18 XOR generator testsjuniper_data/tests/integration/test_api.py— 18 API integration testsjuniper_data/tests/integration/test_e2e_workflow.py— 14 E2E workflow testsjuniper_data/tests/integration/test_lifecycle_api.py— 17 lifecycle API testsjuniper_data/tests/integration/test_storage_workflow.py— 8 storage workflow testsjuniper_data_client/tests/test_client.py— 35 client testsDocumentation:
docs/api/JUNIPER_DATA_API.md— 622-line API referencenotes/INTEGRATION_DEVELOPMENT_PLAN.md— Integration work itemsnotes/test_suite_audit/*.md— Test suite audit reportsAGENTS.md— Updated project guideModified Components
Configuration:
pyproject.toml— Reconfigured for JuniperData (dependencies, tools, markers).pre-commit-config.yaml— Updated hooks and settings.gitignore— Updated for JuniperData structureconf/conda_environment.yaml— Updated for JuniperData dependenciesconf/*.conf— Updated shell configuration filesReports:
reports/junit/results.xml— Updated test resultsRemoved Components
JuniperCanopy Application (185
src/files):src/backend/— cascor_integration.py, data_adapter.py, redis_client.py, cassandra_client.py, statistics.py, training_monitor.py, training_state_machine.pysrc/frontend/— dashboard_manager.py, 10 component files (metrics_panel, network_visualizer, dataset_plotter, decision_boundary, about_panel, etc.)src/communication/— websocket_manager.pysrc/main.py— 2,030-line application entry pointsrc/demo_mode.py— 1,004-line demo modesrc/config_manager.py— 492-line configuration managersrc/tests/— ~2,900 tests across unit, integration, regression, and performance suitessrc/assets/— 15 logo/icon filesLegacy Infrastructure (86 files):
conf/docker-compose.yaml,conf/Dockerfile,conf/app_config.yamlconf/requirements.txt,conf/requirements_ci.txtconf/setup_environment.conf,conf/setup_environment_functions.confRisks & Rollback Plan
GET /v1/healthJUNIPER_DATA_URL) allow graceful degradation when JuniperData is unavailableRelated Issues / Tickets
What's Next
Remaining Items
Notes for Release
Release: JuniperData v0.4.0 — Integration Infrastructure & Client Library:
Key highlights:
Review Notes
Repository Transformation: This PR replaces the entire JuniperCanopy codebase with JuniperData. The
src/directory (185 files) is removed andjuniper_data/(82 new files) takes its place. This is a planned extraction, not an accidental deletion.Version History: The CHANGELOG documents the complete development history from v0.1.0 (initial release) through v0.4.0 (this PR). Each version built incrementally on the previous one.
Client Library: The
juniper_data_client/package is designed to be published to PyPI and consumed by both JuniperCascor and JuniperCanopy, replacing their current duplicate client code.Legacy Parity: The
algorithm="legacy_cascor"mode in SpiralGenerator reproduces the statistical properties of JuniperCascor's original SpiralProblem implementation, ensuring backward compatibility.New Modules: 6 new generators and 5 new storage backends added after the initial PR description was written.
B008 Warnings: The 9 B008 flake8 warnings in
datasets.pyare intentional FastAPI patterns (usingQuery(),Depends()in function defaults) and should not be "fixed."Test Count Growth: From 0 tests (main branch had JuniperCanopy tests) to 699 JuniperData-specific tests (658 service + 41 client).