Development roadmap

Jump to bottom Edit New page

Damien Irving edited this page Jun 5, 2015 · 8 revisions

Here are the features that we're working on developing. It's an open source project, so feel free to fork our GitHub repos and contribute!

Sanity / attribute checking for input files

The most common cause of workflow failure is input files with problems such as duplicate values in the time axis
See this issue for discussion on developing a Sanity Check module

Create an observations Dataset (see this issue)
Most users currently use the CWSLab workflow tool via the NCI virtual desktop. We therefore need to talk to NCI about:

Usage monitoring
A place for users to write output data to which is (a) persistent (i.e. not scratch space like the /local/ directory) but (b) not a formal dataset you would want to keep in /g/data/
How to connect Vistralis to the Raijin queuing system so that workflows can be run in parallel (this should be technically possible by July 2015)

Handling of modules that take more than one input files

This issue discusses the mapping of conflicting constraints
This issue discusses the handling of metadata

Hide the wrapper process from the user

See this issue
Eventually we may simply have a graphical user interface where the user just enters a few details relating to their command line program and the wrapper is automatically generated. Such functionality already exists in generic VisTrails, we just had to edit the source code for our purposes.
Another option may be to create wrappers automatically from JSON files - i.e. a user specifies the inputs, outputs, any added constraints and the command line program to run.

Use THREDDS for data access instead of depending on cdscan

Since cdscan is a CDAT function, the merging of different files can only be done with Python at the moment

Make it easier for people to use the plugin on their own machine and with their own data

At the moment the documentation mainly caters for people who are analysing CMIP data on the NCI machines
As well as improving the documentation, we need to make it easier for people to describe their own filename patterns / filesystem

Use the UV-CDAT version of VisTrails

They've taken the generic VisTrails and built it into UV-CDAT