Skip to content

Commit f63a7a7

Browse files
martinpittlarskarlitski
martinpitt
authored and
larskarlitski
committed
tutorial: Using Cockpit test VMs with your own test framework
Closes cockpit-project#158
1 parent c0eac52 commit f63a7a7

File tree

2 files changed

+296
-0
lines changed

2 files changed

+296
-0
lines changed

‎_posts/2018-03-28-puppeteer-tests.md

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
---
2+
title: Using Cockpit test VMs with your own test framework
3+
author: pitti
4+
date: 2018-03-28
5+
category: tutorial
6+
tags: cockpit starter-kit tests puppeteer
7+
slug: cockpit-custom-test-framework
8+
comments: true
9+
---
10+
11+
The [Cockpit Starter Kit](https://github.com/cockpit-project/starter-kit/) provides the scaffolding for your own Cockpit
12+
extensions: a simple page (in React), build system (webpack, babel, eslint, etc.), and an integration test using
13+
Cockpit's own Python test API on top of the
14+
[Chromium DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/). See the recent
15+
[introduction](http://cockpit-project.org/blog/cockpit-starter-kit.html) for details.
16+
17+
But in some cases you want to use a different testing framework; perhaps you have an already existing project and tests.
18+
Then it is convenient and recommended to still using Cockpit's test VM images: they provide an easy way to test your
19+
project on various Fedora/Red Hat/Debian/Ubuntu flavors; and they take quite a lot of effort to maintain! Quite
20+
fortunately, using the test images is not tightly bound to using Cockpit's test API, and not even to tests being written
21+
Python. They can be built and used entirely through command line tools from Cockpit's [bots/
22+
directory](https://github.com/cockpit-project/cockpit/tree/master/bots/), so you can use those with any programming
23+
language and test framework.
24+
25+
## Building and interacting with a test VM
26+
27+
To illustrate this, let's check out the Starter Kit and build a CentOS 7 based VM with cockpit and the starter kit
28+
installed:
29+
30+
```sh
31+
$ git clone https://github.com/cockpit-project/starter-kit.git
32+
$ cd starter-kit
33+
$ make vm
34+
```
35+
36+
Ordinarily, the generated `test/images/centos-7.qcow2` would be used by
37+
[test/check-starter-kit](https://github.com/cockpit-project/starter-kit/blob/master/test/check-starter-kit). But let's
38+
tinker around with the VM image manually. Cockpit's
39+
[testvm.py](https://github.com/cockpit-project/cockpit/blob/master/bots/machine/testvm.py) module for using these VMs
40+
can be used as a command line program:
41+
42+
```sh
43+
$ bots/machine/testvm.py centos-7
44+
ssh -o ControlPath=/tmp/ssh-%h-%p-%r-23253 -p 2201 [email protected]
45+
http://127.0.0.2:9091
46+
RUNNING
47+
```
48+
49+
It takes a few seconds to boot the VM, then it prints three lines:
50+
51+
* The SSH command to run something inside the VM
52+
* The URL for the forwarded Cockpit port 9090
53+
* A constant `RUNNING` flag that test suites can poll for to know when to proceed.
54+
55+
You can now open that URL in your browser to log into Cockpit (user "admin", password "foobar") and see the installed
56+
Starter Kit page, or run a command in the VM through the given SSH command:
57+
58+
```sh
59+
$ ssh -o ControlPath=/tmp/ssh-%h-%p-%r-23253 -p 2202 [email protected] head -n2 /etc/os-release
60+
NAME="CentOS Linux"
61+
VERSION="7 (Core)"
62+
```
63+
64+
The VM gets shut down once the `testvm.py` process gets a `SIGTERM` (useful for test suites) or `SIGINT` (useful for
65+
just pressing Control-C when interactively starting this in a shell).
66+
67+
## Using testvm.py in your test suite
68+
69+
Let's use the above in a [Puppeteer](https://github.com/GoogleChrome/puppeteer) test.
70+
[check-puppeteer.js](../files/starter-kit/check-puppeteer.js) is a straight port of
71+
[check-starter-kit](https://github.com/cockpit-project/starter-kit/blob/master/test/check-starter-kit); of course it is
72+
a little longer than the original as we don't have the convenience functions of
73+
[testlib.py](https://github.com/cockpit-project/cockpit/blob/master/test/common/testlib.py) to automatically start and
74+
tear down VMs or do actions like "log into Cockpit", but it is still fairly comprehensible. Download it into the tests/ directory of
75+
your starter-kit checkout, then install puppeteer:
76+
77+
```sh
78+
$ PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1 npm install [email protected]
79+
```
80+
81+
This will avoid downloading an entire private copy of chromium-browser and use the system-installed one. (Of course you
82+
can also just call `npm install puppeteer` if you don't care about the overhead). Now run the tests:
83+
84+
```sh
85+
$ DEBUG="puppeteer:page,puppeteer:frame" PYTHONPATH=bots/machine test/check-puppeteer.js
86+
```
87+
88+
This will run them in a verbose mode where you can follow the browser queries and events.
89+
90+
Let's walk through some of the code:
91+
92+
```js
93+
function startVm() {
94+
return new Promise((resolve, reject) => {
95+
let proc = child_process.spawn("bots/machine/testvm.py", [testOS], { stdio: ["pipe", "pipe", "inherit"] });
96+
let buf = "";
97+
proc.stdout.on("data", data => {
98+
buf += data.toString();
99+
if (buf.indexOf("\nRUNNING\n") > 0) {
100+
let lines = buf.split("\n");
101+
resolve({ proc: proc, ssh: lines[0], cockpit: lines[1] });
102+
}
103+
});
104+
proc.on("error", err => { throw `Failed to start vm-run: ${err}` });
105+
});
106+
}
107+
```
108+
109+
This uses `testvm.py` as above to launch the VM. In a "real" test suite this would go into the per-test setup code.
110+
`check-puppeteer.js` does not use any test case organization framework (like [jest](https://www.npmjs.com/package/jest)
111+
or [QUnit](https://qunitjs.com/)), but if you have more than two or three test cases it's recommended to use one.
112+
113+
114+
```js
115+
async function testStarterKit() {
116+
const vm = await startVm();
117+
118+
const browser = await puppeteer.launch(
119+
// disable sandboxing to also work in docker
120+
{ headless: true, executablePath: 'chromium-browser', args: [ "--no-sandbox" ] });
121+
```
122+
123+
This is the actual test case. Here we start Puppeteer, and here you can change various
124+
[options](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteerlaunchoptions)
125+
to influence the test. For example, set `headless: false` to get a visible Chomium window and follow live what the test
126+
does (or where it hangs).
127+
128+
```js
129+
const page = await browser.newPage();
130+
131+
try {
132+
// log in
133+
await page.goto(vm.cockpit);
134+
await page.type('#login-user-input', 'admin');
135+
await page.type('#login-password-input', 'foobar');
136+
await page.click('#login-button');
137+
```
138+
139+
This is the equivalent of Cockpit test's `Browser.login_and_go()`. In a real test you would probably factorize this into
140+
a helper function.
141+
142+
```js
143+
await page.waitFor('#host-nav a[title="Starter Kit"]');
144+
await page.goto(vm.cockpit + "/starter-kit");
145+
let frame = getFrame(page, 'cockpit1:localhost/starter-kit');
146+
147+
// verify expected heading
148+
await frame.waitFor('.container-fluid h2');
149+
await frame.waitForFunction(() => document.querySelector(".container-fluid h2").innerHTML == "Starter Kit");
150+
151+
// verify expected host name
152+
let hostname = vmExecute(vm, "cat /etc/hostname").trim();
153+
await frame.waitFor('.container-fluid span');
154+
await frame.waitForFunction(
155+
h => document.querySelector(".container-fluid span").innerHTML == ("Running on " + h),
156+
{}, hostname);
157+
}
158+
```
159+
160+
This is a direct translation of what check-starter-kit does: Assert the expected heading and host name message.
161+
162+
```js
163+
catch (err) {
164+
const attachments = process.env["TEST_ATTACHMENTS"];
165+
if (attachments) {
166+
console.error("Test failed, taking screenshot...");
167+
await page.screenshot({ path: attachments + "/testStarterKit-FAIL.png"});
168+
}
169+
throw err;
170+
}
171+
```
172+
173+
This part is optional, but very useful for debugging failed tests. If any assertion fails, this creates a PNG screenshot
174+
from the current browser page state. Run the test with `TEST_ATTACHMENTS=/some/existing/directory` to enable this. The
175+
Cockpit CI machinery will export any files in this directory to the http browsable test results directory.
176+
177+
```js
178+
finally {
179+
await browser.close();
180+
vm.proc.kill();
181+
}
182+
};
183+
```
184+
185+
This is a poor man's "test teardown" which closes the browser and VM.
186+
187+
## Feedback
188+
189+
starter-kit and external Cockpit project tests are still fairly new, so there are for sure things that could work more
190+
robustly, easier, more flexibly, or just have better documentation. If you run into trouble, please don't hesitate
191+
telling us about it, preferably by [filing an issue](https://github.com/cockpit-project/starter-kit/issues).
192+
193+
Happy hacking!
194+
195+
The Cockpit Development Team

‎files/starter-kit/check-puppeteer.js

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
#!/usr/bin/node
2+
/* Usage:
3+
* 1. Build a Cockpit starter kit test VM:
4+
* $ git clone https://github.com/cockpit-project/starter-kit.git
5+
* $ cd starter-kit
6+
* $ make vm
7+
*
8+
* 2. Install puppeteer, using the already installed chromium browser:
9+
*
10+
* $ PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1 npm install [email protected]
11+
*
12+
* (You can also just do "npm install puppeteer" if you don't mind
13+
* downloading an internal copy of Chromium and want to use the latest
14+
* version.)
15+
*
16+
* 3. Run this test, with some debug output:
17+
*
18+
* $ DEBUG="puppeteer:page,puppeteer:frame" PYTHONPATH=bots/machine test/check-puppeteer.js
19+
*/
20+
21+
const testOS = process.env["TEST_OS"] || "centos-7";
22+
23+
const child_process = require('child_process');
24+
const puppeteer = require("puppeteer");
25+
26+
// start virtual machine and return VM object; call vm.proc.kill() to stop it again
27+
function startVm() {
28+
return new Promise((resolve, reject) => {
29+
let proc = child_process.spawn("bots/machine/testvm.py", [testOS], { stdio: ["pipe", "pipe", "inherit"] });
30+
let buf = "";
31+
proc.stdout.on("data", data => {
32+
buf += data.toString();
33+
if (buf.indexOf("\nRUNNING\n") > 0) {
34+
let lines = buf.split("\n");
35+
resolve({ proc: proc, ssh: lines[0], cockpit: lines[1] });
36+
}
37+
});
38+
proc.on("error", err => { throw `Failed to start vm-run: ${err}` });
39+
});
40+
}
41+
42+
// run shell command in VM object and return its output
43+
function vmExecute(vm, command) {
44+
let cmd = vm.ssh + ' ' + command;
45+
let out = child_process.execSync(cmd, { encoding: 'utf8' });
46+
console.log(`vmExecute "${cmd}" output: ${out}`);
47+
return out;
48+
}
49+
50+
// return Puppeteer frame object for given frame ID
51+
function getFrame(page, id) {
52+
return page.mainFrame().childFrames().find(f => f.name() == id);
53+
}
54+
55+
async function testStarterKit() {
56+
const vm = await startVm();
57+
58+
const browser = await puppeteer.launch(
59+
{ headless: true, executablePath: 'chromium-browser', args: [ "--no-sandbox" /* to work in docker */ ] });
60+
61+
const page = await browser.newPage();
62+
63+
try {
64+
// log in
65+
await page.goto(vm.cockpit);
66+
await page.type('#login-user-input', 'admin');
67+
await page.type('#login-password-input', 'foobar');
68+
await page.click('#login-button');
69+
70+
await page.waitFor('#host-nav a[title="Starter Kit"]');
71+
await page.goto(vm.cockpit + "/starter-kit");
72+
let frame = getFrame(page, 'cockpit1:localhost/starter-kit');
73+
74+
// verify expected heading
75+
await frame.waitFor('.container-fluid h2');
76+
await frame.waitForFunction(() => document.querySelector(".container-fluid h2").innerHTML == "Starter Kit");
77+
78+
// verify expected host name
79+
let hostname = vmExecute(vm, "cat /etc/hostname").trim();
80+
await frame.waitFor('.container-fluid span');
81+
await frame.waitForFunction(
82+
h => document.querySelector(".container-fluid span").innerHTML == ("Running on " + h),
83+
{}, hostname);
84+
} catch (err) {
85+
const attachments = process.env["TEST_ATTACHMENTS"];
86+
if (attachments) {
87+
console.error("Test failed, taking screenshot...");
88+
await page.screenshot({ path: attachments + "/testStarterKit-FAIL.png"});
89+
}
90+
throw err;
91+
} finally {
92+
await browser.close();
93+
vm.proc.kill();
94+
}
95+
};
96+
97+
testStarterKit()
98+
.catch(err => {
99+
console.error(err);
100+
process.exit(1);
101+
});

0 commit comments

Comments
 (0)