Workflow-based project serialization and parallel execution#1229
Merged
Conversation
ThomasKroes
added
enhancement
Introduce populateDataBufferFromVariantMapToRawBufferSync() (declared in Serialization.h, implemented in Serialization.cpp) to decode variant-map payloads directly into caller-provided raw buffers. Replace numerous populateDataBufferFromVariantMap(...) calls with the new sync helper across PointData, Points, Images and ClusterData to ensure decoded sizes match expected buffer sizes. Refactor ClusterData loading to use populateDataBufferFromVariantMapAsync() with a QFutureWatcher and read clusters via QDataStream from the decoded bytes, removing the previous WorkflowPlan-based synchronous decoding path.
Move raw read/write helpers into ClusterData and refactor cluster serialization to separate metadata and a contiguous indices blob (bump ClustersFormatVersion to 2). Comment out the old QDataStream cluster operators and remove the friend write operator from Cluster to prepare for the new format. Update ClusterData::toVariantMap to emit metadata and indices as separate raw blocks and update fromVariantMap to consume the indices blob (reconstruction logic staged / partially commented). Switch several PointData loads from synchronous blocking buffer population to asynchronous variants, add debug logging callbacks, and adjust captures to use the instance. Extend util/Serialization with new APIs to populate a raw destination buffer via the workflow executor (populateDataBufferFromVariantMapToRawBuffer and async variant), add a PopulateDoneCallback type, and increase the default max worker thread count to 64 to support parallel decoding. These changes enable non-blocking data loading and a more compact cluster storage layout.
Add ClusterSerializer.h/.cpp and register them in CMakeLists. Implements conversion between QVector<Cluster> and QVariantMap using a versioned binary format (FormatVersion = 2). Uses QDataStream (Qt_6_5) to serialize header metadata and a contiguous index buffer; includes helpers to write/read raw numeric vectors, build an index buffer from clusters, and rebuild clusters from headers+indices. Adds size/version checks and throws runtime_error on format or I/O errors. Median/mean/stddev are serialized but currently not restored into Cluster objects.
Refactor cluster serialization to perform metadata and index loading as WorkflowPlan stages: ClusterSerializer is now a QObject, adds ClustersLoadContext, and uses a WorkflowPlan with parallel preprocessing jobs (headers and indices) followed by a sequential rebuild stage. Change WorkflowPlanExecutor APIs to use std::optional<WorkflowExecutionContext> for the parent context, propagate a copied parent context into async execution, and ensure per-job contexts wait for pending async work before finishing. Also fix a variable name typo in ProjectOpenWorkflowPlan and apply minor formatting/cleanup.
Remove several qDebug() statements from ClusterSerializer::fromVariantMap to reduce noisy debug output. Deleted logs that printed header and index data sizes and the 'Rebuilt X clusters' message after rebuilding clusters.
Adds PropertiesSerializer.h/.cpp that provide optimized QVariantMap serialization (toVariantMap/fromVariantMap), include a format version, and enable optional verbose debug logging. Removes many low-level cluster raw-data helpers from PropertiesSerializer.cpp and switches DatasetImpl (Set.cpp) to use PropertiesSerializer::toVariantMap/fromVariantMap when storing/loading dataset properties. Also fixes WorkflowPlanExecutor to register pending async work on the current context (currentContext) instead of parentContextCopy. These changes simplify properties handling and centralize optimized serialization logic.
Remove a leftover qDebug in Dataset::fromVariantMap, adjust dataset load weighting to pass rawBlockSize directly, and refactor async workflow execution to create the child execution/report node immediately when a parent context exists. executeAsync now builds an executionContext early and forwards it to executeAsyncImpl; nested workflows are executed synchronously under a WorkflowExecutionScope with start/finish info logs and return an empty future. Also tidy up a small conditional formatting in job progress handling. These changes ensure child progress/report nodes are created while the parent job is still active and improve nested workflow lifecycle visibility.
Add a new properties serializer (PropertiesSerializer) implemented in DimensionsSerializer.h/.cpp to provide optimized QVariant serialization with a format version (FormatVersion = 2). The serializer exposes toVariantMap and fromVariantMap, performs version checks, and uses a WorkflowPlan to run preprocessing during deserialization. Register the new files in PointData's CMakeLists (sources and headers). Also, uncomment the _dimensionsPickerAction->fromParentVariantMap(...) call and apply a small whitespace cleanup in PointData.cpp.
Introduce DimensionsSerializer (replacing the old PropertiesSerializer usage) to serialize/deserialize point dimension names as optimized raw data. The serializer provides toVariantMap and fromVariantMap, uses a FormatVersion and WorkflowPlan/DimensionsLoadContext for async, thread-aware loading and sets names on PointData. PointData now calls DimensionsSerializer for read/write of DimensionNames and delays dataset-change notifications to ensure dimensions are applied. Misc: remove several debug/prettyPrintVariantMap calls (Project.cpp, ProjectMetaAction.cpp, ProjectSaveWorkflowPlan.cpp), update includes and headers, and add basic error/version checks when loading dimensions.
Rename the DimensionsSerializer class and files to DimensionNamesSerializer, updating all references and build rules. Added DimensionNamesSerializer.h and removed the old DimensionsSerializer.h; renamed the implementation file and updated CMakeLists.txt to use the new names. Adjusted API to operate on Points* (instead of PointData*), refactored the load context (DimensionNamesLoadContext, no longer carrying a QVariantMap), and updated call sites in PointData.cpp. Also updated debug macro name and a QStringList reserve cast. FormatVersion remains unchanged.
Initialize _completionMatchMode to Qt::MatchContains in OptionAction constructor to ensure a defined completer matching behavior and avoid an uninitialized member value.
Call getProjectSaveParameters() with the current project's file path when a project is open, so save operations default to the actual project path. Refactor getProjectSaveParameters() to only construct and show the FileSaveDialog when no filePath is provided, and populate ProjectSaveParameters from the dialog (including parallel settings and thread count) when used. Persist directory and MaxNumberOfThreads settings and update the Parallel setting only when the dialog is shown. This avoids unnecessary prompts when saving to a known path and fixes incorrect default save behavior.
Switch dataset load jobs to JobProgressMode::Nested and remove the explicit stage weight so progress is aggregated via nested children. In WorkflowExecutionContext::createChild normalize weights with effectiveWeight = max(1.0, weight), use effectiveWeight for creating progress nodes and in diagnostics, and only warn if progress already started and effectiveWeight > 0. This prevents zero/negative child weights and makes progress reporting more consistent.
Introduce preliminary async serialization API and thread-safety fixes. - util/Serializable.h: Add AsyncToVariantMapResult and AsyncFromVariantMapResult structs, declare virtual fromVariantMapAsync() and toVariantMapAsync(), and expand documentation for synchronous vs asynchronous serialization/deserialization. Comment out WorkflowPlan include. - util/Serializable.cpp: Provide default stub implementations for fromVariantMapAsync() and toVariantMapAsync(). - private/DataHierarchyManager.cpp: Make creation of item map jobs thread-safe by using a shared QMutex and QMutexLocker when writing into the shared workflow state; change job progress mode to Automatic and add a debug log message. Also adjust lambda captures to avoid referencing mutated locals. - private/DataManager.cpp: Temporarily comment out a large block in toVariantMap() that builds savedDataList/data save jobs (wrapped with /* ... */). These changes prepare an asynchronous serialization workflow, add basic synchronization to prevent concurrent writes to shared state, and disable an existing save-job block for now. Further work is expected to implement full workflow integration and error/progress handling for the async APIs.
Introduce a new header (src/util/AsyncVariantMapResult.h) that defines AsyncToVariantMapResult and AsyncFromVariantMapResult (with a sharedVariantMap alias and CORE_EXPORT), and add it to PUBLIC_UTIL_HEADERS. Update Serializable.cpp to include the new header and simplify Serializable.h by replacing the previous inline struct definitions with forward declarations. This decouples async variant-map result types from Serializable, reducing header dependencies and allowing shared reuse of the result types.
Introduce asynchronous serialization helpers and adapt callers, and remove the password parameter from the saveProject API. - Serializable: add forward declarations for AsyncToVariantMapResult/AsyncFromVariantMapResult, fix include path, and implement fromVariantMapAsync() and toVariantMapAsync() to return WorkflowPlan-based async results (toVariantMapAsync returns a shared QVariantMap result). The default async implementations schedule the existing synchronous functions into a WorkflowPlan stage. - DataHierarchyManager: include AsyncVariantMapResult and rewrite item map creation to use dataHierarchyItem->toVariantMapAsync(), execute the resulting workflow plan, wait for completion, rethrow exceptions if any, then store the resulting variant map (with SortIndex) into the shared state. Adjust job progress mode and thread-affinity and increment sort index per item. - Project API: remove the encryption password parameter from AbstractProjectManager::saveProject, ProjectManager::saveProject, and corresponding declaration in ProjectManager.h. These changes enable structured async serialization workflows and simplify the project save API by removing the password argument.
Replace the async toVariantMap workflow with a direct synchronous call to toVariantMap(), removing the WorkflowExecutionOptions, future execution/wait and exception rethrow logic. Also change the job's progress mode from Nested to Automatic. This simplifies item map creation by avoiding a nested workflow execution and unnecessary synchronization.
Always pass the incoming filePath to getProjectSaveParameters instead of conditionally using the current project's file path. This prevents the current project's path from overriding the intended target when saving (e.g., for new projects or 'Save As') and ensures save parameters are derived from the caller-provided path.
Adjust modal task UI and clean up logging: remove "Finished" from the modal task status filter so finished tasks aren't included; cast getMinimumDuration() to int when setting the timer interval; add braces around the dialog-close branch and remove the QCoreApplication::processEvents() call. Also remove noisy qDebug() statements from Task::setFinished() and Task::setProgress() to reduce log spam.
Remove the old mv::util::BaseException and related Exception types and switch callers to use ManiVaultException or std::runtime_error. Delete WorkflowRunner and WorkflowAsyncLauncher and remove them from CMake sources. Update ManiVaultException declaration/definition and adjust namespaces/includes accordingly. Replace catch clauses and thrown exception types across multiple files (models, help, project manager, page actions, learning center, file downloader) and simplify FileDownloader async error propagation. Also update WorkflowPlanExecutor signature to reference the new ManiVaultException type.
…le with the HMBA release branch) Introduce utilities to detect and map datasets within selection groups. Added areDatasetsPartOfSelectionGroup to AbstractEventManager and implemented it in EventManager to scan KeyBasedSelectionGroup instances. Extended SelectionGroup with BiMap::getKeys(), KeyBasedSelectionGroup::areDatasetsPartOfGroup(), and getMappingBetweenDatasets() (plus corresponding header declarations) to enable retrieving keys and deriving index mappings between datasets in the same group. These additions enable selection sync and membership checks between datasets.
Add optional parent-context overloads and propagate execution contexts through the workflow executor API. - Extend AbstractWorkflowPlanExecutor with overloads that accept an optional WorkflowExecutionContext for executeBlocking/executeAsync/executeOnCurrentThread and adjust executeAsyncImpl signature. - Update WorkflowPlanExecutor to implement the new overloads: pass parent context into executeChild/executeRoot, create child contexts for nested async jobs, and register pending async work on the parent context. - Propagate the executionContext into executeAsyncImpl and use WorkflowExecutionScope when present so nested workflows inherit progress/reporting/thread-pool behavior. - Refactor several method signatures to use unqualified types (WorkflowPlan, WorkflowExecutionContext) and expose getThreadPool as public. - Add matching WorkflowPlan API overloads and corresponding (currently empty) stubs in WorkflowPlan.cpp for blocking/async/current-thread calls that accept a parentContext. - Minor formatting/cleanup around the blocking execution event-loop and future handling. The changes enable nested workflows to correctly report progress and be tracked by parent contexts; some newly added WorkflowPlan.cpp methods remain to be implemented.
Use std::move for WorkflowExecutionOptions (and parentContext where applicable) when forwarding to executor calls to avoid unnecessary copies. Add null-checks with qWarning and early returns in async/on-current-thread overloads that accept a parentContext to guard against a missing executor. Update header to remove the default nullptr for the Task* parameter on executeOnCurrentThread, requiring the caller to pass the task explicitly.
Replace the old 'source' concept with a dedicated 'emitter' field and add a new 'location' field to workflow messages and models. Update AbstractWorkflowMessagesModel to expose Emitter and Location columns/items (rename SourceItem -> EmitterItem and add LocationItem, adjust header/data logic). Change WorkflowMessage struct to include _emitter and _location (remove _source/_nodeName) and add docs. Refactor WorkflowReportNode to use SharedWorkflowReportNode/SharedWorkflowReportNodes (std::shared_ptr and std::vector), update createChild/addMessage/getters and adjust message parameters. Propagate type/signature changes to WorkflowExecutionState and WorkflowExecutionContext, update recursive collectors and getters, and make executeJobOnWorkerThread explicitly override in WorkflowPlanExecutor. Minor API and doc improvements (QString move in ctor, use std::int32_t for counts). These are breaking API changes that require corresponding updates where these types and function signatures are used.
Consolidate string-based sorting for message columns and clean up switch indentation. - WorkflowMessagesFilterModel: remove the separate Source comparison and have Emitter, Location and Text share the same EditRole string comparison (renamed lhsText/rhsText to lhsString/rhsString) to reduce duplication and ensure consistent ordering. - AbstractWorkflowMessagesModel: fix indentation/formatting of SeverityLevel switch cases in LevelItem::data. These changes simplify sorting logic and correct code style/formatting in the models.
Update `CMakeMvSourcesPublic.cmake` to stop listing action files that are no longer part of the public source set. This removes `ActionOperation` and the ActionRecipe/ActionHooks entries from the public header/source lists so CMake reflects the current code layout.
Protect the shared Taskflow executor with a mutex to ensure thread-safe access from multiple code paths. Add null checks for workflow plans and remove the unused handleStageException() method. Introduce DepthGuard RAII helper for automatic depth counter management.
Delete the unused `runStagesRoot` helper from `TaskflowWorkflowPlanExecutor`. This trims redundant code around root-stage execution and keeps stage execution routed through the remaining Taskflow-based helpers.
Update `CMakeMvSourcesPublic.cmake` to drop `WorkflowExecutionNotifier.h/.cpp` from the public workflow header/source lists. This keeps the notifier out of the public source set and aligns the CMake export list with the current workflow API surface.
Removes unused/empty workflow translation units (`WorkflowContextBase.cpp`, `WorkflowExecutionOptions.cpp`, `WorkflowMessage.cpp`) and updates `CMakeMvSourcesPublic.cmake` accordingly. Also refactors `WorkflowGuiThreadDispatcher` to use an out-of-line constructor in its `.cpp` file and adds clearer API/class documentation about GUI-thread affinity and intended usage.
Replace `WorkflowProgressNode::_parent` with `std::weak_ptr` and update parent accessors accordingly. `getParent()` now locks the weak pointer and `isRoot()` checks `expired()`, preventing strong reference cycles between parent and child nodes while preserving existing behavior.
Guarded registry access with a QMutex in add(), get(), and remove() to prevent concurrent read/write races when workflow results are accessed across threads. The header was also expanded with API/class documentation and updated member declarations to include the mutex used for synchronization.
Replace `using namespace mv::workflow` with explicit `mv::workflow::` qualifications in TaskflowWorkflowPlanExecutor.h to avoid namespace pollution. Also fix include cleanup: remove unused QReadWriteLock from WorkflowProgressNode.h, swap QFuture for QMutex in WorkflowResultFuture.h, and qualify AbstractWorkflowPlanExecutor call in Main.cpp.
Clean up workflow header files by replacing `} // namespace mv::util` with plain `}` at namespace close. This standardizes formatting across the workflow module without changing behavior.
Rename private-style `_prefixed` fields in `WorkflowExecutionOptions` and `WorkflowExecutionMetrics::AtomicMetric` to plain names, and update all usage sites accordingly.
Updated workflow stage batch-size lookups in `DataHierarchyManager` and `Serialization` to use the current `workflowBatchingOptions` member names (`datasetLoadingBatchSize`, `dataBlockEncodingBatchSize`, and `dataBlockDecodingBatchSize`) instead of the old underscored fields. This aligns these workflows with the updated execution options API.
Drop leading underscores from WorkflowBatchingOptions batch-size fields to align with naming conventions and improve API clarity. Also clean up the namespace closing line in the header.
Update `PointData::toVariantMapWorkflow()` to store the serialized `rawMap` in the `Raw` field instead of `storeRawStage`. This keeps the workflow output consistent with the expected raw payload and prevents incorrect data from being written.
Adjust workflow execution context setup to avoid assigning a new UUID in `makeRoot` and enforce stricter validity checks in `createChild`. Child contexts now only get created when report node, progress node, and state are all present, preventing partial/invalid execution contexts.
Update `WorkflowConsoleDashboard` to stop more responsively by notifying a condition variable before joining the worker thread. The render loop now runs in a non-const `run()` method, uses a mutex/condition wait between renders, and explicitly sets `_running` when entering the loop. The header was also expanded with API/class documentation, made `final`, and extended with synchronization members for the new wait/notify behavior.
Update `WorkflowConsoleDashboard::run()` to use `_conditionMutex` exclusively for `wait_for` and evaluate `_running` via `load()`. This removes the lock/unlock churn around `render()` and aligns condition-variable waiting with the correct mutex and atomic state checks.
Refactors project operation parameter structs to use consistent non-underscore member names (`filePath`, `parallel`, `maxParallelThreads`) and updates all corresponding `ProjectManager` call sites. Also adds a `verbosity` field to `ProjectOperationParameters` for future operation-level logging control.
Replaced `maxConsoleLogDepth`/`verbosity` with `maxLoggingDepth` across project and workflow execution options, and passed it through open/save project execution paths. Added a user-configurable "Maximum logging depth" setting to project open/save dialogs with persisted preferences. Also fixed minor numeric type issues by casting KDE values to float and using `std::fabs` in MeanShift bounds calculation.
Set `maxLoggingDepthAction` to use the spinbox widget in both project open and save parameter dialogs, ensuring consistent numeric input behavior for this setting.
Add a Windows-only `NOMINMAX` define in `DensityComputation.cpp` and `MeanShift.cpp` so platform headers do not introduce `min`/`max` macros that can interfere with standard C++ code.
Define `NOMINMAX` at the top of `DensityComputation.cpp` and `MeanShift.cpp` so it is set before any headers are included. This prevents `Windows.h` from introducing `min`/`max` macros and avoids related compile or symbol conflicts.
Switch workflow message items to shared message objects and rename `WorkflowMessage` fields to cleaner public members. This updates the list/filter models, fixes filtering to read severity from the underlying item safely, resets the list model before loading new results, and simplifies the result dialog to use a direct tree view with the severity filter toolbar. It also streamlines details tooltip HTML rendering for workflow message metadata.
Removes eager JSON file loading and error swallowing from `Project` construction so initialization no longer depends on immediate disk parsing. Also updates the open-workflow plan name to include the target file path (`Open project <path>`), improving traceability when opening projects.
ThomasKroes
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Introduce a Taskflow-based Workflow Execution Architecture
This PR introduces ManiVault's first dedicated workflow execution architecture, built on top of Taskflow.
Previously, project loading, saving, and other long-running operations were orchestrated through imperative code paths, nested function calls, futures, and ad-hoc parallelization. While functional, there was no unified execution model, making complex workflows increasingly difficult to extend, optimize, and reason about.
The new architecture compiles workflows into dependency graphs that are executed by Taskflow, providing a scalable, composable, and highly parallel execution model. Besides significantly simplifying the implementation, this results in order-of-magnitude performance improvements for large projects and establishes a solid foundation for future workflow features.
✨ Highlights
New workflow execution architecture
⚡ Massive performance improvements
The new architecture enables substantially better utilization of modern multi-core systems, resulting in dramatic reductions in project loading and saving times.
Example project:
Large projects (20–70 GiB) that previously required several minutes to load now complete in only tens of seconds.
Project saving also benefits significantly from the new parallel serialization and blob encoding pipeline.
🧵 Parallelized project serialization
Large binary datasets are automatically partitioned into blocks, enabling efficient parallel compression and decompression while reducing peak memory usage.
🏗️ Modern workflow API
Introduced a cleaner and more expressive workflow API supporting:
This makes complex workflows easier to implement, extend, and maintain.
📈 Improved progress reporting
📦 New block-based blob storage
Introduced a new block-based serialization format for large binary data.
Features include:
🔄 Modernized project I/O
Project loading and saving have been restructured into composable workflow plans, making the complete I/O pipeline easier to extend, maintain, profile, and optimize.
🛠️ Internal improvements
User-visible benefits
Projects saved with ManiVault Studio 1.5.0 and newer use the new serialization architecture.
Projects created with earlier versions remain fully supported and are automatically upgraded when opened. Saving the project again converts it to the new format, allowing future loads and saves to benefit from the improved storage format and significantly improved performance.
Foundation for future development
The new workflow execution architecture provides a strong foundation for future enhancements, including:
This PR represents one of the largest architectural changes to ManiVault to date.
Rather than introducing isolated performance optimizations, it establishes a unified execution architecture for workflows throughout the application. Besides significantly simplifying the implementation, it delivers order-of-magnitude performance improvements for large projects while providing a scalable foundation for future workflow features.