Aditya-feat: Add Material Cost Correlation API endpoint #1957
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…correlation Implement comprehensive date utilities supporting multiple input formats: - parseDateInput: handles ISO, MM-DD-YYYY, MM/DD/YYYY formats and Date objects - normalizeStartDate: normalizes to start of day in UTC - normalizeEndDate: normalizes to end of day UTC with special 'today' handling (now-5min) - isDateToday: helper to check if date matches today in UTC - parseAndNormalizeDateRangeUTC: main orchestrator with default handling and validation All functions use UTC timezone and throw structured error objects following existing patterns. Replaces inline date handling patterns found in existing controllers.
Implement generic query parameter parser supporting multiple input formats: - Handles undefined, single values, comma-separated strings, and arrays - Optional ObjectId validation using mongoose.Types.ObjectId.isValid - Filters empty strings and whitespace-only values - Throws structured error objects consistent with date parsing utilities - Collects all invalid ObjectIds before throwing (not on first invalid) Reuses ObjectId validation pattern from existing controllers and comma-splitting pattern from materialCostController. Designed to be generic and reusable across endpoints.
Implement MongoDB aggregation helpers for material usage and cost calculations: - getEarliestRelevantMaterialDate: finds earliest date from updateRecord/purchaseRecord with filters - aggregateMaterialUsage: aggregates quantityUsed from updateRecord arrays by date range - aggregateMaterialCost: aggregates totalCost from approved purchaseRecord arrays by date range - buildBaseMatchForMaterials: shared helper to build base match conditions (prevents duplication) Reuses aggregation patterns from materialCostController (unwind, match, group, project stages). Only counts purchases with status='Approved' matching existing endpoint behavior. All functions return safe defaults (null/empty arrays) on error, logging exceptions for debugging.
Group related parameters into objects to comply with linter max-parameters rule: - aggregateMaterialUsage: group projectIds/materialTypeIds into filters object, effectiveStart/effectiveEnd into dateRange object - aggregateMaterialCost: same parameter grouping applied - Reduces parameter count from 5 to 3 (BuildingMaterial, filters, dateRange) - Functions destructure objects internally, preserving existing behavior
Implement buildCostCorrelationResponse function to merge usage and cost data: - Merges usage and cost data by composite key (projectId-materialTypeId) - Performs parallel lookups for project names and material type names/units - Computes derived values (totalCostK, costPerUnit) with division-by-zero handling - Groups data by project with per-material-type breakdowns - Handles projects with explicit selection but no data (empty arrays) - Sorts projects alphabetically by name - Builds comprehensive meta object with request, range, and units info - Returns structured response with meta and data arrays Includes helper functions to prevent duplication: - calculateCostPerUnit: handles division-by-zero, returns null when quantityUsed is 0 - calculateTotalCostK: converts cost to thousands - objectIdToString: safely converts ObjectIds to strings All edge cases handled: missing lookups use fallbacks, empty results return valid structure, errors return safe defaults.
Add bmGetMaterialCostCorrelation controller function that orchestrates the entire request flow: - Parses and validates multi-select query parameters (projectId, materialType) using utility - Computes default start date from earliest material record if not provided - Parses and normalizes date range with UTC handling and validation - Runs usage and cost aggregations in parallel for performance - Builds structured response with merged data, lookups, and computed values - Comprehensive error handling with appropriate HTTP status codes (400, 422, 500) - Logs all errors with request context for debugging - Returns structured JSON response with meta and data arrays Follows existing controller patterns and reuses all utility functions to prevent code duplication.
…alsController Extract magic numbers to named constants for better maintainability: - DECIMAL_PRECISION = 4 for quantity calculations (.toFixed) - DAYS_IN_WEEK = 7 and DAYS_IN_TWO_WEEKS = 14 for date calculations Replace console.error with logger.logException in bmGetMaterialSummaryByProject: - Use standard logger utility with request context (method, path, params, query) - Replace magic number 500 with HTTP_STATUS_INTERNAL_SERVER_ERROR constant - Maintains consistency with other controller error handling patterns
Register GET /materials/cost-correlation route in bmMaterialsRouter: - Route placed before /materials/:projectId to ensure proper Express route matching - Full path: /api/bm/materials/cost-correlation - Binds to controller.bmGetMaterialCostCorrelation method - Follows existing route pattern and formatting style Route order is critical - specific routes must come before parameterized routes to prevent Express from matching cost-correlation as a projectId parameter.
Replace magic number 400 with HTTP_STATUS_BAD_REQUEST constant in date range error handling for consistency.
Error handling is already comprehensively implemented:
- Query parameter validation returns 400 for invalid ObjectIds
- Date parsing errors return 422 (Unprocessable Entity)
- Date range validation errors return 400 (Bad Request)
- Aggregation errors return 500 with safe defaults (empty arrays)
- Response building errors return 500 with fallback handling
- All errors are logged with request context using logger utility
- Division-by-zero handled gracefully (returns null)
- Missing lookups use fallback values (ID as name)
- Consistent error response format: { error: message }
Extract repeated ObjectId conversion pattern into convertStringsToObjectIds helper: - Eliminates 4 instances of idStrings.map((id) => new mongoose.Types.ObjectId(id)) - Used in buildBaseMatchForMaterials and buildCostCorrelationResponse - Reduces code duplication and improves maintainability Add comprehensive duplication review document verifying: - All utility functions are reusable and centralized - No business logic duplication between utilities and controller - Error handling follows consistent patterns - All calculations use shared helpers - Code duplication is below 3% target - All checklist items from implementation plan verified
Move test file from src/utilities/queryParamParser.test.js to src/utilities/__tests__/queryParamParser.test.js to follow repository convention: - Matches pattern used by bmdashboard controllers (__tests__ subdirectories) - Keeps test files organized and separate from source files - Updates import path from './queryParamParser' to '../queryParamParser' All 40 tests pass with 100% coverage. This convention will be followed for all future utility test files.
Implement 88 unit tests covering all date utility functions: - parseDateInput: valid/invalid formats, error structure - normalizeStartDate: UTC/local normalization, edge cases - isDateToday: UTC/local comparison, boundary conditions - normalizeEndDate: today capping, non-today normalization, edge cases - parseAndNormalizeDateRangeUTC: date range validation, default handling, error cases Achieves 94% statement coverage, 86% branch coverage, 100% function coverage. All tests use fixed date mocking for consistent results. Tests follow repository convention in __tests__ subdirectory.
Implement 86 unit tests covering all helper functions: - convertStringsToObjectIds: array conversion, edge cases - buildBaseMatchForMaterials: project/material filtering, combined filters - getEarliestRelevantMaterialDate: parallel queries, result comparison, error handling - aggregateMaterialUsage: pipeline structure, filtering, return format - aggregateMaterialCost: status filtering, cost calculation, field validation - calculateCostPerUnit: division handling, edge cases, rounding - calculateTotalCostK: cost scaling calculations - objectIdToString: conversion handling, null/undefined cases - buildCostCorrelationResponse: lookup maps, data merging, project totals, sorting, meta construction, error handling Exports helper functions for testing. Achieves 97% statement coverage, 76% branch coverage, 95% function coverage. All tests use mocked Mongoose models and logger.
Implement 35 unit tests covering all request flow scenarios: - Successful flows: all parameters, default dates, no filters - Query parameter validation: invalid ObjectIds, empty parameters - Date parsing errors: invalid formats, invalid ranges - Aggregation errors: usage/cost failures, database errors - Response building errors: lookup failures, logic errors - Default date computation: earliest date, fallbacks, error handling - Parallel execution: Promise.all verification - Response structure: meta/data validation - Edge cases: empty results, missing properties - Logging verification: all error paths logged with context All tests use mocked utilities and models. Tests follow repository convention and extend existing test file.
Implement 10 unit tests covering route registration and structure: - Route registration: correct path, HTTP method, controller binding - Route ordering: cost-correlation before parameterized route - Router structure: Express Router instance, all routes registered Tests verify that /materials/cost-correlation route is registered before /materials/:projectId to ensure correct route matching. All tests use mocked controller and verify route structure.
Add support for projectName and materialName query parameters in addition to existing ID-based parameters. Implement name resolution functions to convert names to ObjectIds with proper error handling. Maintain backward compatibility with existing projectId and materialType parameters.
- Add explicit date existence checks in purchase record aggregation - Extract helper functions to reduce buildCostCorrelationResponse complexity - Add defensive null/undefined checks in data merging logic - Ensure totalCost is properly converted to number type - Improve error handling and data validation
Fix critical bug where cost correlation endpoint was querying wrong MongoDB collection. Changes: - routes.js: Switch from buildingMaterial (buildingInventoryItems collection) to buildingMaterialModel (buildingMaterials collection) which contains unitPrice data - materialCostCorrelationHelpers.js: - Add stage to handle missing unitPrice values in aggregation - Improve cost value validation (NaN checks) in mergeUsageAndCostData - Add defensive validation in buildProjectsMap for data integrity - Enhance error handling with console.error logging - bmMaterialsController.js: Add response validation check for debugging.
Clean up debugging code and improve logging practices: - Remove console.error debugging statement from aggregateMaterialCost - Remove logger.logException calls from validation error handlers (handleDateRangeError, handleQueryParamError) - these are expected validation errors that return proper HTTP responses, not exceptions - Remove unused debugging code block (costDataTotal calculation)
- Fix mock response chaining (json returns this for proper flow) - Change mockRejectedValueOnce to mockImplementationOnce for sync functions (parseAndNormalizeDateRangeUTC is synchronous, not async) - Update tests to reflect that validation errors are no longer logged as exceptions (query param errors, date errors) - Fix pipeline stage assertions in helpers test for updated aggregation pipeline structure (Stage 3.5/3.6 for unitPrice handling) - Update error logging assertion to match correct function name (buildLookupMaps instead of buildCostCorrelationResponse) - Remove unused commented-out import (MongoMemoryServer)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Implements a new Material Cost Correlation API endpoint that aggregates material usage data from
updateRecordarrays and cost data from approvedpurchaseRecordentries to analyze consumption vs cost efficiency across projects.Related PRS (if any):
This PR is the backend for the frontend PR 4587.
Main changes explained:
Created/Updated Files:
src/controllers/bmdashboard/bmMaterialsController.js
bmGetMaterialCostCorrelationmethod with helper functions for query parameter extraction, date handling, and error handlingsrc/utilities/materialCostCorrelationHelpers.js
src/utilities/materialCostCorrelationDateUtils.js
src/utilities/queryParamParser.js
src/routes/bmdashboard/bmMaterialsRouter.js
GET /materials/cost-correlationroute (positioned before/materials/:projectId)Test files (5 test files)
API Endpoint:
GET /api/bm/materials/cost-correlationQuery Parameters:
projectId/projectName(optional, multi-select): Filter by project(s)materialType/materialName(optional, multi-select): Filter by material type(s)startDate(optional): YYYY-MM-DD, MM-DD-YYYY, MM/DD/YYYY, or ISO 8601 (defaults to earliest material record)endDate(optional): Same formats (defaults to current time minus 5 minutes)Response Structure:
{ "meta": { "request": { "projectIds": [], "materialTypeIds": [], "startDateInput": "...", "endDateInput": "..." }, "range": { "effectiveStart": "...", "effectiveEnd": "...", "defaultsApplied": {...} }, "units": { "currency": "USD", "costScale": { "raw": 1, "k": 1000 } } }, "data": [ { "projectId": "...", "projectName": "Project A", "totals": { "quantityUsed": 2500, "totalCost": 5000, "totalCostK": 5, "costPerUnit": 2 }, "byMaterialType": [ { "materialTypeId": "...", "materialTypeName": "Cement", "unit": "tons", "quantityUsed": 2500, "totalCost": 5000, "totalCostK": 5, "costPerUnit": 2 } ] } ] }Key Implementation Details:
BuildingMaterial.updateRecordarraysBuildingMaterial.purchaseRecordarrays (status='Approved' only)How to test:
Aditya-feat/material-cost-correlation-chartrm -rf node_modules package-lock.json && npm cache clean --forcenpm installto install dependencies, then run the backend locally (npm run dev)npm test -- bmMaterialsController materialCostCorrelation queryParamParser bmMaterialsRouterScreenshots or videos of changes:
TestVideo.mov
Note: