developwithcrane.md

Use crane in development

This guide shows how crane can be used to help in the service migration process.

Overview and general concepts

Token

Almost every operation is asynchronous. You have to do the request and after check the status.

The request gives you an operation identifier. This is call a token.

Tokens also identifies the resource, so it can be used as parameter in other operations.

The are several types:

• context token: identifies a context. Starts with CT.
• build token: identifies a build process. Follows the format <context token>-<imagename>.
• cluster token: identifies a cluster. Starts with CL.
• composition token: identifies a composition deploy process. Starts with CP

Base image

CloudOpting provide several base images. Those can be created with crane and are supposed to be stored on the local registry.

Context

Environment with a common set of puppet modules which are described in a puppetfile.

Images are associated to a context.

A context can have a group identifier. This is a custom id to set as image repo. (will see this later)

Image

An image is the result of executing a Dockerfile and maybe apply a puppet manifest.

There are several ways to build that from:

• Dockerfile + puppet manifest
• base image name + puppet manifest
• Dockerfile

Puppetfile

File that list all the puppet modules needed and where to get them.

Take a look at the official reference or see the example.

Don't forget to install also the dependencies (you can see them on the main page of each module. e.g. https://forge.puppetlabs.com/puppetlabs/mysql/dependencies).

Puppet manifest

File that describes how the system will be configured.

Dockerfile

Docker recipe for a container.

docker-compose.yml

Describes a set of containers, the relation between them, the command to run each one, the volumes to mount, and some other configuration parameters.

cluster

Resource that wraps one or several machines (do not try to add several because it is not supported for the moment) under a identifier and allows you to deploy on them.

composition

Made adding a docker-compose file to a context and linking to a cluster.

Setup you environment

Clone this repo

Using:

git clone https://github.com/CloudOpting/cloudopting-crane

Start crane

Start crane using the specific compose file prepared for testing pilots, pilot-dev.yml:

docker-compose -f pilot-dev.yml up

Enter the UI

Browse to http://<docker machine ip>:8888.

You will see several groups: builder, cluster, composition and extra.

Clicking on each group you can expand the list of methods and selecting a method you can use it introducing the parameters and so on. We'll see later the main methods to help in developing applications.

Example of use

NOTE 1: in crane is possible to add base images to the internal private registry and after use just it name and a puppet manifest to build a image (/builder/images/bases routes). For this guide, in order to keep things simple, we'll skip that.

NOTE 2: all files in this example are available in /examples/migration-guide-1.

Example description

We are going to build a simple application based on two services: a redis and a python flask web server.

The redis container will be built using the official redis image.

The web will use cloudopting/ubuntubase:14.04 and a manifest.

Step 1: build context

Build a context with the puppet modules that your entire application needs.

POST /builder/context with:

• puppetfile: a file like puppetfile with all the modules and dependencies
• group (optional): common label to gather several images. This will become the repo name on the private dockerhub. Left it blank to use default.

The response will be something like:

{
  "status": "building",
  "images": [],
  "token": "CTJnnfqI",
  "group": "default",
  "description": "Under creation"
}

It's important the token parameter, save it.

You can check the status with GET /builder/context/{token}.

{
  "status": "finished",
  "images": [],
  "token": "CTJnnfqI",
  "group": "default",
  "description": "Build finished without errors"
}

Wait until status == finished.

Step 2: build images

Build an image.

It is possible to use one of the following combinations:

• a base image name and a puppet manifest. As we said in NOTE 1, we are not using that on this guide.

• a Dockerfile and a puppet manifest.

• just a Dockerfile.

POST /builder/images with:

• contextReference: the context token from the previous step.
• imageName: leave it blank (NOTE 1).
• dockerfile: the image Dockerfile

First for the redis image:

you'll get something like:

{
  "status": "building",
  "description": "Under creation",
  "tag": "default/redis",
  "token": "CTJnnfqI-IMredis",
  "imageName": "redis",
  "context": "CTJnnfqI"
}

With that token you can use: GET /builder/images/CTJnnfqI-IMredis to retrieve the status.

{
  "status": "finished",
  "description": "Build finished without errors",
  "imageName": "redis",
  "token": "CTJnnfqI-IMredis",
  "tag": "default/redis",
  "context": "CTJnnfqI"
}

Wait until status == finished.

Now, let's do the same for the web image:

Notice this time we're using a Dockerfile (from a cloudopting base) and also a manifest.pp. Take a look at both files to see the details.

This should be the response.

{
  "status": "building",
  "description": "Under creation",
  "tag": "default/redis",
  "token": "CTJnnfqI-IMweb",
  "imageName": "web",
  "context": "CTJnnfqI"
}

Do GET /builder/images/CTJnnfqI-IMweb to retrieve the status.

{
  "status": "finished",
  "description": "Build finished without errors",
  "imageName": "web",
  "token": "CTJnnfqI-IMweb",
  "tag": "default/web",
  "context": "CTJnnfqI"
}

Wait until status == finished.

IMPORTANT: check build.log, specially the apply manifest part, because puppet agent may write some errors but return 0, so crane do not detect the problem.

Step 3: fake a cluster

In crane cluster is a generic entity which represents any place where to deploy containers.

For testing, there is a special container with a docker engine inside where you can try to deploy. Crane resolve the name emulatedhost to that fake machine.

Create a cluster entity with:

POST /cluster/provisionedSingleMachine

• endpoint: url where docker engine will be listening. For this example: http://emulatedhost:4243
• apiVersion: it is possible to force a version, but leave it blank.

The response should be:

{
  "status": "joining",
  "description": "Ready to use",
  "token": "CLo0BKFI",
  "nodes": [
    {
      "status": "joining",
      "endpoint": "http://emulatedhost:4243"
    }
  ],
  "type": "simple-preprovisioned",
  "numberOfNodes": 1
}

Again, we get a token. This cluster token can be used as identifier to say crane where to deploy the application (next step).

Step 4: deploy

Now we can use crane to deploy our application on this emulated host.

We need a docker-compose.yml file like this.

Notice images are named like group/imagename.

Deploy the app with:

POST /composer

• clusterToken: the token we obtained in the previous step.
• composefile: a valid docker-compose.yml

The application should now being deployed on the host.

For this example the web application listen to the port 5000 which is redirected to the port 80 on the docker host (see the emulatedhost configuration on pilot-dev.yml )

While testing your application you may need to change that value and maybe add others.

EXTRA: debug errors

You can debug build errors seeing the log and error files.

On the root of this project, go to application-data/commander/.

There will be a folder called builds, where you will find a folder for each context (named with the context token) and inside all the documents related with that. Inside the subfolder images there will be a folder for each image, and inside that some logs. The most useful are: build.log, build_err.log, pull.log and pull_err.log.