-
Notifications
You must be signed in to change notification settings - Fork 36
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
new file: docs/MdVGumbyOverview.png modified: docs/index.rst new file: docs/quickoverview.rst modified: docs/tutorials/experiment_tutorial_3.rst renamed: docs/tutorials/synchronized_instances_das5.conf -> docs/tutorials/synchronized_instances_das.conf renamed: scripts/das4_node_run_job.sh -> scripts/das_node_run_job.sh renamed: scripts/das4_reserve_and_run.sh -> scripts/das_reserve_and_run.sh renamed: scripts/das4_setup.sh -> scripts/das_setup.sh Resolves: #501
- Loading branch information
Showing
8 changed files
with
74 additions
and
22 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
*********************** | ||
A Quick Overview | ||
*********************** | ||
|
||
Gumby is an experiment runner framework to run local and distributed experiments. | ||
|
||
|
||
The current supported DAS environments are the DAS5 and DAS6 clusters. | ||
More information on the DAS environments can be found at: | ||
`DAS5 <https://www.cs.vu.nl/das5/clusters.shtml>`_. | ||
`DAS6 <https://www.cs.vu.nl/das6/clusters.shtml>`_. | ||
|
||
The interaction between the researcher and the gumby framework can be presented als follows: | ||
|
||
.. image:: MdVGumbyOverview.png | ||
:width: 600 | ||
|
||
|
||
Installation | ||
|
||
Prior to installing Gumby install the required dependencies for basic tests. | ||
Those are the software packages: | ||
python-psutil | ||
python-configobj | ||
r-base | ||
|
||
The way to install packages is specific for your Linux installation. | ||
As an example on the Ubuntu or Debian based systems you may use: | ||
|
||
.. code-block:: bash | ||
$ sudo apt-get install python-psutil python-configobj r-base | ||
These dependencies can also be installed using the `pip` utility. | ||
Please note that more elaborate experiments might require additional dependencies. | ||
|
||
Next, clone the Gumby repository from GitHub by running the following command: | ||
|
||
.. code-block:: bash | ||
$ git clone https://github.com/tribler/gumby | ||
In the Tutorials section you will find information on how to create your first Gumby experiment. | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,41 +1,41 @@ | ||
Running Distributed Experiments on the DAS5 | ||
Running Distributed Experiments on the DAS | ||
=========================================== | ||
|
||
In the previous tutorials we have devised experiments that spawn multiple instances on a single computer. | ||
In this tutorial, we will show how to run an experiment on the `DAS5 compute cluster <https://www.cs.vu.nl/das5/home.shtml>`_. | ||
The DAS5 is a Dutch nation-wide compute infrastructure that consists of multiple clusters, managed by different | ||
In this tutorial, we will show how to run an experiment on the `DAS compute cluster <https://www.cs.vu.nl/das5/home.shtml>`_. | ||
The DAS is a Dutch nation-wide compute infrastructure that consists of multiple clusters, managed by different | ||
universities. | ||
This tutorial assumes that the reader has access to the DAS5 head nodes. | ||
This tutorial assumes that the reader has access to the DAS head nodes. | ||
|
||
The experiment we will run on the DAS5 is the same experiment that we described in the previous tutorial. | ||
The experiment we will run on the DAS is the same experiment that we described in the previous tutorial. | ||
In this experiment, we will spawn multiple (synchronized) instances and each instance will write its ID to a file five | ||
seconds after the experiment starts. | ||
The experiment is started on the DAS5 head node and before the instances spawn, Gumby automatically reserves a certain | ||
The experiment is started on the DAS head node and before the instances spawn, Gumby automatically reserves a certain | ||
number of compute nodes. | ||
Each compute node then spawns a certain number of instances, depending on the experiment configuration. | ||
When the experiment ends, all data generated by instances is collected by the head node. | ||
|
||
The configuration file for this DAS5 experiment looks as follows: | ||
The configuration file for this DAS experiment looks as follows: | ||
|
||
.. literalinclude:: synchronized_instances_das5.conf | ||
.. literalinclude:: synchronized_instances_das.conf | ||
|
||
The new configuration options are annotated with some explanation. | ||
It includes a ``local_setup_cmd`` configuration option that is executed before the experiment starts. | ||
The ``das4_setup.sh`` script checks the user quote on the DAS5 and invokes the ``build_virtualenv.sh`` script that | ||
The ``das_setup.sh`` script checks the user quote on the DAS and invokes the ``build_virtualenv.sh`` script that | ||
prepares a virtual environment with various Python packages. | ||
To use this virtual environment, the ``use_local_venv`` option is set. | ||
|
||
Additionally, there are a few configuration options that are specific to DAS5 experiments. | ||
The ``node_amount`` configuration option indicates how many DAS5 compute nodes are used. | ||
The maximum number of compute nodes in each cluster can be found `here <https://www.cs.vu.nl/das5/clusters.shtml>`_. | ||
Additionally, there are a few configuration options that are specific to DAS related experiments. | ||
The ``node_amount`` configuration option indicates how many DAS compute nodes are used. | ||
The maximum number of compute nodes in each cluster can be found at the DAS sites listed in the Quick Overview. | ||
In our experiment, we spawn 16 instances and use 2 compute nodes. | ||
Gumby automatically balances instances over compute nodes and in our experiment, each compute node hosts 8 instances. | ||
The ``node_timeout`` configuration option indicates the timeout of the experiment. | ||
To prevent premature termination of an experiment, we recommend to set this value a bit higher than the time of the | ||
latest event in the scenario file. | ||
|
||
To run this experiment, execute the following command on one of the DAS5 head nodes: | ||
To run this experiment, execute the following command on one of the DAS system head nodes: | ||
|
||
.. code-block:: bash | ||
$ gumby/run.py gumby/docs/tutorials/synchronized_instances_das5.conf | ||
$ gumby/run.py gumby/docs/tutorials/synchronized_instances_das.conf |
10 changes: 5 additions & 5 deletions
10
...utorials/synchronized_instances_das5.conf → ...tutorials/synchronized_instances_das.conf
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.