Skip to content

Deployment steps

Jump to bottom
Hans Permana edited this page Nov 6, 2018 · 17 revisions

Before installation

Make sure that you have a look at the hardware and software specifications in which the following instruction is valid. They are available here.

Install Docker

If old Docker installation exists

sudo apt-get remove docker.io docker-engine
sudo rm -r /var/lib/docker/

Installation

sudo apt-get update
sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88 # to verify that the fingerprint is 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install docker-ce

First time only

sudo groupadd docker
sudo usermod -aG docker <username>

(optional) Enable on start-up

sudo systemctl enable docker

Environment specific configuration

EODC

EODC Openstack seems to use lower Maximum Transmission Unit (MTU) values and Docker container does not infer this value. As a result, connection to https sites always failed. Extra configuration when starting up Docker daemon is needed (link1 link2)

sudo cp /lib/systemd/system/docker.service /etc/systemd/system/docker.service
sudo vim /etc/systemd/system/docker.service
	# modify "ExecStart=/usr/bin/dockerd -H fd://" to "ExecStart=/usr/bin/docker daemon -H fd:// --mtu=1450"
sudo systemctl daemon-reload
sudo service docker restart

Unfortunately, that is not enough. The default docker_gwbridge(which is used as a default swarm network) had been created with the default MTU value (1500). This has to be deleted and re-created.

sudo apt-get install bridge-utils
sudo service docker stop
sudo ifconfig docker_gwbridge down
sudo brctl delbr docker_gwbridge
brctl show docker_gwbridge # to see if docker_gwbridge has been removed, should have this message "can't get info No such device"
sudo service docker start
# at this point, the docker_gwbridge is somehow automatically created. Try removing it
# First remove the attached container(s) to this network. See: https://docs.docker.com/engine/swarm/networking/#customize-the-docker_gwbridge
docker network disconnect -f docker_gwbridge gateway_ingress-sbox
docker network rm docker_gwbridge
docker network create --subnet 172.18.0.0/16 --opt com.docker.network.bridge.name=docker_gwbridge --opt com.docker.network.bridge.enable_icc=false --opt com.docker.network.bridge.enable_ip_masquerade=true --opt com.docker.network.driver.mtu=1450 docker_gwbridge
sudo service docker restart

Finally, the following options needs to be added to docker-compose.yml.

networks:
   swarmnet:
   docker_gwbridge:
      driver: bridge
      driver_opts:
         com.docker.network.driver.mtu: 1450

Firewall

For environments protected by Firewall, the following port numbers need to be allowed.

  • Port 7946 TCP/UDP for container network discovery.
  • Port 4789 TCP/UDP for the container overlay network.
  • TCP port 2377 for cluster management communications.
  • TCP and UDP port 2049 for NFS server.
  • TCP and UDP port 111 for rpcbind/sunrpc (for NFS).
  • TCP 8081 presumably for the notebook server to communicate with the hub (need more information)
  • TCP 8888 presumably to communicate with the notebook server (need more information)
  • TCP 443 for the HTTPS connection to the hub

Swarm configuration

On manager node

docker swarm init --advertise-addr <manager host machine IP address>

A response like the following will be returned.

Swarm initialized: current node (bvz81updecsj6wjz393c09vti) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-3pu6hszjas19xyp7ghgosyx9k8atbfcr8p2is99znpy26u2lkl-1awxwuwd3z9j1z3puu7rcgdbx \
    172.17.0.2:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

As stated, the given command is to be executed on worker node(s).

On worker node

If, for some reason, the token information is lost, run the following command:

docker swarm join-token worker

and then, run the following command to join the Swarm network

docker swarm join --token <token generated by the manager> <manager host machine IP address>:2377

Jupyterhub configuration

Create GitHub OAuth app

now create a new GitHub OAuth app to get the Client ID and Client Secret. This is required for authentication using GitHub. To do that, go to https://github.com/organizations/esa-esdl/settings/applications and create new OAuth app. Once created, the information of this new app shall be entered in .env file.

Build and start Jupyterhub

On the Server node, run the following:

cd ~
git clone https://github.com/esa-esdl/jupyterhub-swarm.git
cd jupyterhub-swarm
mv .env.example .env
vim .env
# generate certs --> see https://github.com/esa-esdl/jupyterhub-swarm/wiki/Create-new-Letsencrypt-SSL-certiticates
# generate cookie_secret (A 64-byte cryptographically secure randomly generated string used to sign values of secure cookies set by the hub)
openssl rand -hex 32 > cookie_secret
vim userlist
    # example: 
    # user1 admin
    # user2
    # user3
docker build -f Dockerfile.jupyterhub -t jupyterhub .
#docker volume create jupyterhub_data
docker stack deploy --compose-file docker-compose.yml jupyterhub

Notebook images

Run the following on worker node to download notebook container (it will take a while)

docker pull quay.io/esdc/singleuser-r-nb:0.5.2
Clone this wiki locally