Skip to content

Commit

Permalink
Improve the documentation
Browse files Browse the repository at this point in the history
	new file:   docs/MdVGumbyOverview.png
	modified:   docs/index.rst
	new file:   docs/quickoverview.rst
	modified:   docs/tutorials/experiment_tutorial_3.rst
	renamed:    docs/tutorials/synchronized_instances_das5.conf -> docs/tutorials/synchronized_instances_das.conf
	renamed:    scripts/das4_node_run_job.sh -> scripts/das_node_run_job.sh
	renamed:    scripts/das4_reserve_and_run.sh -> scripts/das_reserve_and_run.sh
	renamed:    scripts/das4_setup.sh -> scripts/das_setup.sh

Resolves: #501
  • Loading branch information
pvanita committed Mar 28, 2022
1 parent c9d29d1 commit 5ac576e
Show file tree
Hide file tree
Showing 8 changed files with 74 additions and 22 deletions.
Binary file added docs/MdVGumbyOverview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 6 additions & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@
Welcome to Gumby's documentation!
=================================

.. toctree::
:maxdepth: 1
:caption: Overview:

quickoverview.rst

.. toctree::
:maxdepth: 1
:caption: Concepts:
Expand Down
46 changes: 46 additions & 0 deletions docs/quickoverview.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
***********************
A Quick Overview
***********************

Gumby is an experiment runner framework to run local and distributed experiments.


The current supported DAS environments are the DAS5 and DAS6 clusters.
More information on the DAS environments can be found at:
`DAS5 <https://www.cs.vu.nl/das5/clusters.shtml>`_.
`DAS6 <https://www.cs.vu.nl/das6/clusters.shtml>`_.

The interaction between the researcher and the gumby framework can be presented als follows:

.. image:: MdVGumbyOverview.png
:width: 600


Installation

Prior to installing Gumby install the required dependencies for basic tests.
Those are the software packages:
python-psutil
python-configobj
r-base

The way to install packages is specific for your Linux installation.
As an example on the Ubuntu or Debian based systems you may use:

.. code-block:: bash
$ sudo apt-get install python-psutil python-configobj r-base
These dependencies can also be installed using the `pip` utility.
Please note that more elaborate experiments might require additional dependencies.

Next, clone the Gumby repository from GitHub by running the following command:

.. code-block:: bash
$ git clone https://github.com/tribler/gumby
In the Tutorials section you will find information on how to create your first Gumby experiment.

28 changes: 14 additions & 14 deletions docs/tutorials/experiment_tutorial_3.rst
Original file line number Diff line number Diff line change
@@ -1,41 +1,41 @@
Running Distributed Experiments on the DAS5
Running Distributed Experiments on the DAS
===========================================

In the previous tutorials we have devised experiments that spawn multiple instances on a single computer.
In this tutorial, we will show how to run an experiment on the `DAS5 compute cluster <https://www.cs.vu.nl/das5/home.shtml>`_.
The DAS5 is a Dutch nation-wide compute infrastructure that consists of multiple clusters, managed by different
In this tutorial, we will show how to run an experiment on the `DAS compute cluster <https://www.cs.vu.nl/das5/home.shtml>`_.
The DAS is a Dutch nation-wide compute infrastructure that consists of multiple clusters, managed by different
universities.
This tutorial assumes that the reader has access to the DAS5 head nodes.
This tutorial assumes that the reader has access to the DAS head nodes.

The experiment we will run on the DAS5 is the same experiment that we described in the previous tutorial.
The experiment we will run on the DAS is the same experiment that we described in the previous tutorial.
In this experiment, we will spawn multiple (synchronized) instances and each instance will write its ID to a file five
seconds after the experiment starts.
The experiment is started on the DAS5 head node and before the instances spawn, Gumby automatically reserves a certain
The experiment is started on the DAS head node and before the instances spawn, Gumby automatically reserves a certain
number of compute nodes.
Each compute node then spawns a certain number of instances, depending on the experiment configuration.
When the experiment ends, all data generated by instances is collected by the head node.

The configuration file for this DAS5 experiment looks as follows:
The configuration file for this DAS experiment looks as follows:

.. literalinclude:: synchronized_instances_das5.conf
.. literalinclude:: synchronized_instances_das.conf

The new configuration options are annotated with some explanation.
It includes a ``local_setup_cmd`` configuration option that is executed before the experiment starts.
The ``das4_setup.sh`` script checks the user quote on the DAS5 and invokes the ``build_virtualenv.sh`` script that
The ``das_setup.sh`` script checks the user quote on the DAS and invokes the ``build_virtualenv.sh`` script that
prepares a virtual environment with various Python packages.
To use this virtual environment, the ``use_local_venv`` option is set.

Additionally, there are a few configuration options that are specific to DAS5 experiments.
The ``node_amount`` configuration option indicates how many DAS5 compute nodes are used.
The maximum number of compute nodes in each cluster can be found `here <https://www.cs.vu.nl/das5/clusters.shtml>`_.
Additionally, there are a few configuration options that are specific to DAS related experiments.
The ``node_amount`` configuration option indicates how many DAS compute nodes are used.
The maximum number of compute nodes in each cluster can be found at the DAS sites listed in the Quick Overview.
In our experiment, we spawn 16 instances and use 2 compute nodes.
Gumby automatically balances instances over compute nodes and in our experiment, each compute node hosts 8 instances.
The ``node_timeout`` configuration option indicates the timeout of the experiment.
To prevent premature termination of an experiment, we recommend to set this value a bit higher than the time of the
latest event in the scenario file.

To run this experiment, execute the following command on one of the DAS5 head nodes:
To run this experiment, execute the following command on one of the DAS system head nodes:

.. code-block:: bash
$ gumby/run.py gumby/docs/tutorials/synchronized_instances_das5.conf
$ gumby/run.py gumby/docs/tutorials/synchronized_instances_das.conf
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
experiment_name = synchronized_instances_das5
instances_to_run = 16
local_instance_cmd = das4_reserve_and_run.sh
local_instance_cmd = das_reserve_and_run.sh
post_process_cmd = post_process_write_ids.sh
scenario_file = write_ids.scenario
sync_port = __unique_port__

# The command that is executed prior to starting the experiment. This script prepares the DAS5 environment.
local_setup_cmd = das4_setup.sh
# The command that is executed prior to starting the experiment. This script prepares the DAS environment.
local_setup_cmd = das_setup.sh

# We use a venv on the DAS5 since installing packages might lead to conflicts with other experiments.
# We use a venv on the DAS since installing packages might lead to conflicts with other experiments.
use_local_venv = TRUE

# The number of DAS5 compute nodes to use.
# The number of DAS compute nodes to use.
node_amount = 2

# The experiment timeout after which the connection with the compute node is closed.
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash
# %*% A simple script to run experiments on the DAS4 trough prun.
# %*% Have in mind that this script uses das4_node_run_job.sh, so you will need to set its config options too.
# %*% A simple script to run experiments on the DAS trough prun.
# %*% Have in mind that this script uses das_node_run_job.sh, so you will need to set its config options too.
EXIT_CODE=-15
cancel_reservation () {
# Cancel all our dr jobs in the queue
Expand Down Expand Up @@ -48,7 +48,7 @@ WORKING_DIR=$PWD
cd ~

echo "Reserving $NODE_AMOUNT nodes for $DAS4_RESERVE_DURATION secs."
prun -t $DAS4_RESERVE_DURATION -v -np $NODE_AMOUNT das4_node_run_job.sh &
prun -t $DAS4_RESERVE_DURATION -v -np $NODE_AMOUNT das_node_run_job.sh &
PID=$!

sleep 1
Expand Down
File renamed without changes.

0 comments on commit 5ac576e

Please sign in to comment.