Home
Drona Composer is also a Passenger Phusion app using React/Flask. Before you can use Drona Composer, you need to run setup to install the dependencies. After cloning the repo inside OOD (either sys or dev directory), enter the Drona Composer directory and run the following command:
./setup
This will install all the Python dependencies in requirements.txt. It will also create the environments directory. After running the setup script, the app should be ready to use. The first time you run the app, it will show the typical "Initialize App" screen.
Drona Composer will check a specific user directory for environments. On HPRC clusters, this location is /scratch/users/$USER. If the cluster you are installing on uses a different location, you will need to update functions def add_environment() and def _get_environments() in file: views/job_composer.py and replace:
user_envs_path = f"/scratch/user/{os.getenv('USER')}/drona_composer/environments"
with the correct location on your cluster.
Drona Composer will use the /tmp dir to store temporary information. On HPRC clusters, this is just /tmp, hard coded in functions:
machine_driver_scripts/utils.py: drona_add_additional_file(additional_file, preview_name = "", preview_order = 0)
machine_driver_scripts/utils.py: drona_add_warning(warning)
machine_driver_scripts/utils.py: drona_add_mapping(key, evaluation_str)
machine_driver_scripts/engine.py: set_dynamic_additional_files(self, env_path, params)
machine_driver_scripts/engine.py: get_dynamic_map(self)
machine_driver_scripts/engine.py: get_warnings(self, params)
If you use a different location for tmp, you need to update these functions with the correct tmp location.
The config.yml file contains the entry modules_db_path: "/sw/hprc/sw/dor-hprc-tools-dashboard-utils/bin/" This variable is used by the module form element. This is the only element that depends on an external script that retrieves available modules. The script will be available through another git repository.
The setup script retrieves the cluster name by calling a custom script named clustername. On non-HPRC clusters, you might need to update setup.sh to set the cluster name. The clustername is mainly used internally and in the form title. The name doesn't really matter.
Drona Composer requires at least Python version 3.8. For clusters that still have versions 3.6, we provide an older requirements file (requirements.txt.Python2.6.8). Copy this file to requirments.txt and follow the usual setup process. You might need to remove file package-lock.json.
Drona Composer will check for system environments (an environment is the building block to create/generate jobs) in the environments directory. When you first install Drona Composer, this directory will be empty. We recommend adding at least the Generic environment. You can find example Generic envs in directory environments-repo/clustername/ You can copy this Generic env, update the various declaration files to match your particular situation, and then copy the update env to the environments directory.
Drona Composer will check for system environments (an environment is the building block to create/generate jobs) in the environments directory. When you first install Drona Composer, this directory will be empty. We recommend adding at least the Generic environment. You can find example Generic envs in directory environments-repo/clustername/ You can copy this Generic env, update the various declaration files to match your particular situation, and then copy the update env to the environments directory.
Drona Composer provides a 100% graphical interface to create and run any workflow without the need to write a Slurm script or even be aware of Slurm syntax. It guides you to provide the relevant information to generate and submit a job for many different workflows and environments.
Drona provides a number of environments to choose from. The image below shows a screenshot of the Drona composer interface with a dropdown menu of some of the environments.
Drona composer checks for environments in the following two locations.
- Drona system directory
- $SCRATCH/drona_composer/environments
The screenshot above shows system environments in black and environments in your SCRATCH directory in blue. HPRC provides environments for numerous research areas. In a later section, we show how to import additional environments to your SCRATCH directory.
Once you select an environment, additional fields relevant to that environment will become visible. In this case, we will show the Generic environment. The Generic environment is the most flexible, allowing you to create any job script. The next screenshot shows the expanded form to guide you in providing all the relevant information.
Some of the form's fields include a module element (to add all the modules needed in your job script easily), a field to upload data and scripts, a field where you can specify what kind of accelerator (if any) you want to use, and fields to specify common job requirements and advanced Batch scheduler-specific options. For other environments, these fields would be very different and specific to that particular environment.
Hover over the little question mark to get additional information for any of the input fields. It will show further information and help.
NOTE: You are not required to enter values for all (or any) fields. When a nu value is specified, Drona composer will use default values. Once you are finished entering values, click the "preview" button.
The following image shows a screenshot of the preview window.
The preview window contains various components,
Drona composer will analyze the provided input values to ensure there are no mistakes or inconsistencies. The warning pane can also show environment-specific messages to assist you. For example, in the screenshot, the warning shows that a GPU was requested for 4 days. However, the maximum wall time to request a GPU on ACES is two days.
Drona Composer will generate the job scripts based on your input. It will insert code for requested modules and changes to the correct directory. For most environments, it will also set up and execute that environment. All the job script windows are fully editable. For the Generic profile, this is the place where you enter the commands you want to execute in the job.
HPRC provides environments from a variety of research areas. These environments are available in a repository that you can import. Next to the environments dropdown, you will find a button with a + button. If you click this button, a new window with available environments you can import will pop up. The following image shows a screenshot of the import window.
To import an environment, click the add button, and Drona Composer will copy the environment to $SCRATCH/drona_composer/environments.
To see Drona Composer in action, please check out a Drona Composer Demo on HPRC's YouTube channel. Note that this is an older version of Composer with a number of current features missing.
One feature of Drona Composer is that researchers can create their environments. An environment consists of a few specification files. The following sections will discuss how to create a custom environment.
Aside from the system directory, Drona composer will search $SCRATCH/drona_composer/environments for additional environments. Every directory inside $SCRATCH/drona_composer/environments is considered an environment, and Drona Composer will show all of them in the environment list in the form. Once an environment is selected, the Drona composer will read various files in the environment directory to populate the form and generate the job script. Drona will look for the following files:
- schemas.json
- maps.json
- template.txt
- driver.sh (potentially optional in future release)
- utils.py (optional)
The composer engine orchestrates the whole process. It will process file schemas.json (the file declaring all the form elements) and generates the expanded form. Once the researcher clicks the preview button, the composer engine maps input values to placeholder strings using the mapping in file maps.json. Finally, the engine replaces the placeholders in the template and driver files with actual string values.
The following sections will detail how Drona Composer processes these files to generate and submit a job script.
The form contents are declared in file schema.json, in the environment's directory. This file contains all the individual elements shown in the form. Every element will have the following minimum structure.
"<ElementName>": {
"type": "<type>",
"label": "<labelName>",
"name": "<varName>",
"help": "Here you can provide some extra information to assist the researcher"
}
ElementName identifies the element. It's recommended that you provide a descriptive name. The type field designates the type of element. Every element has an assigned label that will be shown in the form together with the component. The name field designates the variable name that will store the value the researcher enters in the form. For example, in a number element (discussed next), the researcher might enter a value of 100. That value will be stored in variable varName. The help field provides an option to show additional information when hovering over the label. The help field is optional (in theory, the name and label fields are also optional)
Every Drona Composer Form element will have custom fields. Check the form elements page for a list of all available form elements.
For an actual production example, check out schema.json from the Generic environment that is part of ACES Drona composer
The above section showed all the currently available elements. By default, Drona composr will show all of these elements. However, many environments might only want to show certain elements depending on the selected values in other elements. For example, Drona composer provides the condition field for this. This field can be added to any element.
"condition": "<ElementName>.value1
In the above example, the element containing the condition will only be shown if another element ElementName has been selected with a value op value1.
The template.txt and driver.sh files contain placeholder values that will be replaced with actual string values in the final step. To generate the placeholder values, Drona uses a mapping from input values (that the researcher provided in the form) to placeholder values. The mappings are declared in the mapping file mapping.json. This is a JSON-formatted text file with a list of mappings. The syntax for every element in the list is as follows;
"placeholder" : "[STRINGELEM ]+"
where STRINGELEM can be any of the following
- a variable, designated by $variable
- a function call, designated by !some_function($var1, ... , $varN)
- a string literal
$variable is the name field in an element defined in schemas.json. When evaluating, this will store the value the researcher provided in the form. The function call is a call to a function that is defined in file utils.py
For example, suppose schema.json defined elements with names cores, time, mem, and version. The snippet below shows a maps.json file with two placeholders.
{
"batchopts": “!retrieve_batch_opts($cores, $time, $memory)”,
“MODULE”: “module load Matlab/$version”
}
For an actual production example of a mapping file, check out the maps.json and utils.py files from the Generic environment that is part of ACES Drona composer
File template.txt contains a template of the job script that will be generated. In the final step, Drona composer will iterate over template.txt and driver.sh and replace any placeholder with the generated value.
For an actual production example check out the template.txt and driver.sh files from the Generic environment that is part of ACES Drona composer
Drona composer provides an option to add warnings to the preview window. This can be used to provide feedback to the researcher based on values entered in the form, for example, on the ACES clusters the maximum walltime for PVC jobs (i.e. jobs where the researcher requests a PVC GPU) is 2 days. The screenshot below shows the warning when the researcher requests one or multiple PVC GPUs and a five-day walltime.
To add the warning, you can call the following convenience function inside any Python function definition in utils.py
drona_add_warning(String)
For example, above, we saw a screenshot of a warning when a researcher requests a PVC and a wall time of more than two days. The snippet below is actual code from utils.py in the Generic environment that adds the warning:
if gpu == "PVC" and total_hours > (24*2):
drona_add_warning("Requested walltime "+walltime+" (hh:mm). Reducing to max of 48 hours in GPU/PVC queues")
The Drona engine will iterate over all the warnings and show them in the preview window. In future versions of the Composer, we might add options to add severity levels for warnings.
By default Drona composer will process (substitute placeholders, preview, and copy to the job directory) the following two files:
- template.txt (which typically is the job script that is being created)
- driver.sh
This is sufficient for most environments, but for more advanced environments, it's possible that additional files need to be processed. For example, some environments might require different templates, depending on the input the researcher provided in the form. Drona Composer provides two ways to accomplish this.
You can specify a list of additional files Drona composer will process in a JSON file named additional_files.json. Suppose we have two Python scripts that will be executed in our environment, named pre_process.py and post_process.py. The format of this JSON file will be as follows:
{
"files" : ["pre_process.py","post_process.py"]
}
Drona composer will look for File additional_files.json in the environment directory and include all the entries as additional files to be processed. In the example, pre_process.py and post_process.py will be handled like driver.sh and template.txt. Any placeholders will be substituted and shown in the fully editable preview window. Once the researcher clicks the submit button, those files will be copied to the job directory.
Another way to add additional files is to call the following convenience function inside any Python function definition in utils.py
drona_add_additional_file(String)
As you can see, this is very similar to the convenience function to add warnings, discussed above. As an example, Let's suppose we have an environment that needs to run a pre-processing step that depends on the values the researcher provided in the form.
if gpu == "PVC":
drona_add_additional_file("pre_process_pvc.py")
else:
drona_add_additional_file("pre_process.py")
In the above example, if the researcher selected a PVC GPU, it will include a specialized preprocessing script (pre_process_pvc.py). The files that are added dynamically will be handled exactly the same as driver.sh and template.txt. Any placeholders will be substituted and they will be shown in the fully editable preview window. Once the researcher clicks the submit button, those files will be copied to the job directory.
A previous section discussed the mapping from input values the researcher provided in the form to placeholders that will be substituted in the template files (e.g., template.txt and driver.sh). The mapping is defined in file maps.json. Drona also provides a function to add dynamic mappings. This is very useful for cases where multiple placeholders depend on similar input variables and conditions. To add mappings dynamically, use the following function
drona_add_mapping(String,String)
where the first String represents the left-hand side of the mapping and the second String represents the righthand side of the mapping.