Add capability to perform multiple runs, possibly with a parameter sweep

[jopetty]

Currently, there is a one-to-one correspondence between issuing the python main.py ... command and producing a single model. But ideally, an experiment should encapsulate both a parameter sweep (i.e., train SRN-SRN, GRU-GRU, Transformer-Transformer, etc. models) and allow for multiple runs of any given parameter combination (i.e., for each set of hyperparameters do 10 runs so we can average performance). This means we need a way to launch multiple jobs and specify parameter sweeps in a config file.

For sweeping: Hydra includes an Ax plugin, which seems to allow for parameter sweeps to be defined in the YAML config.

For multiruns: Maybe look at the JobLib launcher plugin

[jopetty]

Also of note, there is a plugin for the Submitit Launcher which automatically runs sessions with SLURM jobs. It would be useful to see if this could be used to submit jobs on the GRACE cluster.

[jopetty]

It looks like multirun support is somewhat provided out-of-the-box; using the -m flag allows one to sweep over parameters like:

python train.py model=model1,model2

but this has some problems with the custom output directory structure we've created.

[jopetty]

Multirun directory structure has been fixed in 33f9ce5. Now it looks like:

outputs/
  experiment/
    model/
      DATE_TIME/
        RUN_1/
        RUN_2/
        ....