-
Notifications
You must be signed in to change notification settings - Fork 1
/
index.html
281 lines (164 loc) · 19.9 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
<!DOCTYPE html>
<html xmlns='http://www.w3.org/1999/xhtml' lang='en'>
<head>
<meta charset='utf-8'/>
<title>Coremob Test Approach</title>
<script class='remove'>
var respecConfig = {
specStatus: "CG-DRAFT"
, shortName: "coremob-testing-plan"
, editors: [
{ name: "Tobie Langel"
, company: "Facebook"
, companyURL: "http://facebook.com/" }
]
, edDraftURI: "http://tobie.github.com/coremob-test-approach"
, copyrightStart: 2012
, noIDLIn: true
, wg: "Core Mobile Web Platform Community Group"
, wgURI: "http://www.w3.org/community/coremob/"
, wgPublicList: "public-coremob"
, wgPatentURI: "XXX"
, isRecTrack: false
, isNoTrack: true
, format: 'markdown'
};
</script>
<script src='http://www.w3.org/Tools/respec/respec-w3c-common' class='remove' async></script>
</head>
<body style="max-width:45em">
<section id='sotd'>
Development of this document happens on <a href='https://github.com/coremob/level-1-rec'>GitHub</a>. Please file well-scoped <a href='https://github.com/coremob/level-1-rec/issues'>issues</a> and send <a href='https://github.com/coremob/level-1-rec/pulls'>pull requests</a> directly on GitHub. Please send comments to the mailing list as described below.
</section>
<section id='abstract'>
This document describes the testing plan for providing a test suite for the Coremob specification.
</section>
Introduction
------------
As per it's [charter](http://www.w3.org/community/coremob/charter/), the Coremob CG is due to produce test suites for the specifications it publishes. Because Coremob specs do not create new technology but instead curates existing ones, the role of the CG is limited, in theory at least, to finding pre-existing test suites for each of the specifications it references and developing software which enables to run them all together. In practice however, many of the referenced specifications don't have proper test suites yet and CG participants will either have to contribute tests themselves or enable the broader Web developer community to do so, notably via efforts like [Test the Web Forward](http://testthewebforward.org/).
This document will therefore endeavor to look at these two aspects--the test runner and the actual test themselves--and provide guidance to the CG on how to best move forward.
Terminology
-----------
Testing _outside_ of the browser predates testing _within_ the browser by a number of years. Consequently, the architecture and terminology of testing wasn't really designed with the Web in mind. While Web pages provide a simple way to run tests in a clean environment, they also introduce a significant layer of technological complexity (they're asynchronous, and opaque to the test runner until loaded). They also really muddle concepts. Whereas in a standard programming environment the relationship between test and test suites are straightforward, on the Web, a test suite can be either a collection of pages or a collection of tests depending on who you're talking to. Similarly, test runners are exist both at the page level, where they iterate over tests (or sometimes, groups of tests), and at the top level, where they iterate over pages (or, you've guessed it, groups of pages).
For the purpose of this document, the following terminology will be used.
A <dfn title="testrunner">test runner</dfn> is a program which is able to run multiple, potentially nested <a title="testsuite">test suites</a> and collect the results of running these suites for display through a UI or further processing. It is composed of a client-side component written in JavaScript and of an optional server-side component.
A <dfn title="testsuite">test suite</dfn> consists of a collection of <a title="testpage">test pages</a> or of other, nested test suites.
A <dfn title="testpage">test page</dfn> is a single HTML document and, optionally, additional resources such as images or CSS files. Test pages contain one or more <a title="test">tests</a>. They also reference a <a title=testframework>test framework</a> which auto-runs the test on page load.
A <dfn title="test">test</dfn> is a single JavaScript function. Its role is to setup the test environment, run a number of <a title="assertion">assertions</a>, report the result of running those assertions to a <a title="testframework">test framework</a>, and tear down its environment. Tests can be synchronous or asynchronous depending on the needs.
An <dfn title="assertion">assertion</dfn> is the smallest building block of testing. It allows to verify predetermined conditions are met and throws an exception when that is not the case.
A <dfn title="testframework">test framework</dfn> is a JavaScript program which is in charge of running the <a title="test">tests</a> found on a _single_ <a title="testpage">test page</a> and reporting the results of running these tests to the <a title="testrunner">test runner</a>.
<dfn title="testharness.js">testharness.js</dfn> is the W3C's <a title="testframework">test framework</a> of choice. The source code can be found on [GitHub](https://github.com/jgraham/testharness.js/) and [excellent documentation](http://darobin.github.com/test-harness-tutorial/docs/using-testharness.html) is available.
A test framework <dfn title="shim">shim</dfn> is a JavaScript program that enables tests written using a different <a title="testframework">test framework</a> than <a title="testharness.js">testharness.js</a> to run and report their results to the <a title="testrunner">test runner</a>.
The <dfn title="testserver">W3C test server</dfn> is a web server that is fully dedicated to testing. It hosts the <a title="w3ctestsuiteframework">W3C Test Suite Framework</a>. Its URL is [http://w3c-test.org](http://w3c-test.org).
The <dfn title="w3ctestsuiteframework">W3C Test Suite Framework</dfn> is the combination of all <a title="testsuite">test suites</a> written by W3C groups and versioned in a Mercurial repository stored on the <a title="testserver">W3C test server</a>, the <a title="jsonapi">JSON API</a>, <a title="testharness.js">testharness.js</a> and code which allows to run the <a title="testsuite">test suites</a> and store their results in a database.
The <dfn title="jsonapi">W3C Test Suite Framework JSON API</dfn>'s essential role is to smooth out the somewhat chaotic organization of the <a title="testserver">W3C test server</a> by providing a unified API to find the <a title="testsuite">test suite</a> that corresponds to each specification. It also enables posting the test results back to the W3C database. The API is [fully documented](http://w3c-test.org/framework/docs/api/).
Requirements
------------
The requirements are
There are two sets of requirements. Those stated in Coremob CG's [charter](http://www.w3.org/community/coremob/charter/) which are must-haves and those expressed during the F2F meetings which are nice-to-haves.
### Must-haves ###
> The CG will compile accompanying test suites for each specification it releases. Where appropriate, the test suites will draw on pre-existing tests developed for the feature’s original specification. Otherwise, CG participants will endeavor to fill in the gaps. Since test case development may drive normative changes or clarifications into the original specifications, any new tests will be contributed back to the appropriate W3C (or non-W3C) group by their original author.
With that in mind the CG has a single requirement to produce a test suite to accompany [[Coremob 2012]].
The CG MUST produce a test suite for [[COREMOB-2012]].
The CG MUST produce a test framework that can run testharness.js based tests.
### Nice-to-Haves ###
The CG MAY produce shims that allow non-W3C tests to be run and their results collected.
The CG MAY produce a test framework that enables running W3C tests across different origins.
The CG MAY produce a test framework that's able to run on a non public network.
Tests
-----
- Look at the existing W3C test suites and assess the test coverage of specs referenced by [[COREMOB-2012]].
- Identify the gaps and prioritize them.
- Involve developers to help close the gap where it matters most, notably by piggybacking on [Test the Web Forward](http://testthewebforward.org/) and similar efforts.
- Identify existing test suites outside of W3C specifications (notably for HTTP and ECMAScript).
- Assess their coverage.
- Assess compatibility of their <a title=testframework>test framework</a> with the different <a title=testrunner>test runner</a> options we are considering.
Structure of the Test Runner
----------------------------
We propose four solutions of increasing complexity.
### Host the test runner directly on the W3C test server ###
<img src="./img/coremob_1.png" width="471" height="517" alt="Diagram of option 1">
1. The test device navigates to the <a title=testserver>W3C test server</a> which responds with the <a title=testrunner>test runner</a>.
2. The <a title=testrunner>test runner</a> queries the <a title="jsonapi">W3C JSON API</a> to obtain the URLs of the <a title=testpage>test pages</a> it wants to run.
3. The <a title=testrunner>runner</a> then sequentially passes the URL of each <a title=testpage>test page</a> to the iframe.
4. Each <a title=testpage>test page</a> and its resources is fetched from the <a title=testserver>W3C test server</a> and loaded into the iframe. This auto-runs the <a title=testframework>test framework</a>.
5. Once it has finished running the <a title=testpage>test page</a>, the <a title=testframework>test framework</a> updates the <a title=testrunner>test runner</a>.
6. When all <a title=testpage>test pages</a> have been run, the <a title=testrunner>test runner</a> transfers the results to browserscope.org and the <a title=testserver>W3C test server</a> (via its <a title="jsonapi">JSON API</a>).
7. The results are publicly available on browserscope.org and the <a title=testserver>W3C test server</a>.
#### Pros ####
- This solution satisfies all of our Must-have requirements.
- It is fast and simple to build as it consists of a simple client-side runner.
- There are no known issues.
- implies
#### Cons ####
- This solution does not satisfy most of our Nice-to-Have requirements.
- Including tests for non W3C specification implies adding them directly to the W3C repository.
### Host the test runner on coremob.org but run the tests from the W3C test server ###
<img src="img/coremob_2.png" width="482" height="507" alt="Coremob 2">
1. The test device navigates to coremob.org which responds with the <a title=testrunner>test runner</a>.
2. The <a title=testrunner>test runner</a> queries the <a title="jsonapi">W3C JSON API</a> using JSON-P to obtain the URLs of the <a title=testpage>test pages</a> it wants to run.
3. The <a title=testrunner>runner</a> then sequentially passes the URL of each <a title=testpage>test page</a> to the iframe.
4. In turn each <a title=testpage>test page</a> is fetched along with its resources from the <a title=testserver>W3C test server</a> and loaded into the iframe. Other, non W3C <a title=testpage>test pages</a> can also be run in the same way, provided the <a title=testframework>test framework</a> used is compatible with the <a title=testrunner>runner</a>, or can be made compatible via a <a title="shim">shim</a>. For example, [test262](http://test262.ecmascript.org/), the conformance test suite for [[ECMA-262-51]], would be run that way.
5. Once it has finished running the <a title=testpage>test page</a>, the <a title=testframework>test framework</a> updates the <a title=testrunner>test runner</a> through the [[POSTMSG]] API.
6. When all <a title=testpage>test pages</a> have been run, the <a title=testrunner>test runner</a> transfers the results to browserscope.org and the <a title=testserver>W3C test server</a> (via its <a title="jsonapi">JSON API</a>).
7. The results are publicly available on browserscope.org and the <a title=testserver>W3C test server</a>.
#### Pros ####
- Satisfies all of our Must-have requirements and some of our Nice-to-Have ones too.
- Relatively easy to implement.
- Could enable pulling running tests on other domains in a breeze, provided they sent test results using [[POSTMSG]] and we agreed on a format for test results, e.g. [TAP](http://testanything.org/wiki/index.php/Main_Page).
#### Cons ####
- requires modification of W3C code to enable cross-origin communication.
- doesn't enable testing of unreleased devices on closed networks.
### Host the test runner on coremob.org or locally, proxy all requests through it ###
There a number of alternative scenarios here depending on whether we prefer more code to run on the client or on the server. The basic idea stays the same, however: the URLs of the <a title=testpage>test pages</a> are fetched through the <a title="jsonapi">JSON API</a> and their content (and resources) proxied through a server-side component. This avoids same origin policy constraints and also allows running the <a title=testpage>test suite</a> inside of a closed network.
Below are two example architecture for this. The first example targets the current use case we have with Coremob, that is, being able to host a publicly accessible <a title="testsuite">test suite</a>, run it and collect the results on browserscope.org or elsewhere. The second example caters for the needs of device manufactures or carriers, which want to test unreleased devices on a closed network.
#### Example architecture of a publicly runnable implementation ####
<img src="img/coremob_3.png" width="516" height="511" alt="Coremob 3">
1. The test device navigates to coremob.org which responds with the <a title=testrunner>test runner</a>.
2. The <a title=testrunner>runner</a> queries the <a title="jsonapi">W3C JSON API</a> using JSON-P to obtain the URLs of the <a title=testpage>test pages</a> it wants to run. Note this could also be done by a server-side component.
3. The <a title=testrunner>runner</a> then sequentially passes the URL of each <a title=testpage>test page</a> as a query parameter of the iframe.
4. Instead of being fetched directly from the <a title=testserver>W3C test server</a>, each <a title=testpage>test page</a> along with its resources is instead proxied through coremob.org. Other, non W3C <a title=testpage>test pages</a> could also be similarly proxied and run.
5. Once it has finished running the proxied <a title=testpage>test page</a>, the <a title=testframework>test framework</a> updates the <a title=testrunner>test runner</a>.
6. Once all <a title=testpage>test pages</a> have been run, the <a title=testrunner>test runner</a> transfers the results to browserscope.org and the <a title=testserver>W3C test server</a> (via its <a title="jsonapi">JSON API</a>).
7. The results are publicly available on browserscope.org and <a title=testserver>W3C test server</a>.
#### Example architecture for an implementation hosted on a closed network ####
<img src="img/coremob_3_firewall.png" width="510" height="539" alt="Coremob 3 Firewall">
1. The test device navigates to a local server within the closed network.
2. This local server queries the <a title="jsonapi">W3C JSON API</a> and obtain the URLs of the <a title=testpage>test pages</a> it wants to run.
3. The local server responds to the test device with the <a title=testrunner>test runner</a> and the URLs of the proxied <a title=testpage>test pages</a> it wants to run.
4. The <a title=testrunner>runner</a> then sequentially passes the URL of each proxied <a title=testpage>test page</a> the iframe.
5. Instead of being fetched directly from the <a title=testserver>W3C test server</a>, each <a title=testpage>test page</a> along with its resources is instead proxied through the local server. Other, non W3C <a title=testpage>test pages</a> could also be similarly proxied and run.
6. Once it has finished running the proxied <a title=testpage>test page</a>, the <a title=testframework>test framework</a> updates the <a title=testrunner>test runner</a>.
7. When all <a title=testpage>test pages</a> have been run, the <a title=testrunner>test runner</a> transfers the results back to the local server.
8. The results are only available locally.
#### Pros ####
- satisfies all of our Must-have requirements
#### Cons ####
- Some uncertainty as to how that can be done given the SOP problem
- does not satisfy all of our Nice-to-Have requirements.
### Build a custom solution ###
This option completely forgoes the <a title="jsonapi">W3C JSON API</a> and goes directly for the W3C's repository. While this greatly simplifies the design, it puts the burden of understanding the structure of the W3C repository and keeping up with changes on us.
Other test repositories can be treated similarly.
<img src="img/coremob_4.png" width="494" height="505" alt="Coremob 4">
1. The test repositories are copied to coremob.org on a regular basis (maybe via a cron job).
2. When test device navigates to coremob.org it responds with the <a title=testrunner>test runner</a> and the URLs of the <a title=testpage>test pages</a> it wants to run.
3. The <a title=testrunner>runner</a> sequentially passes the URL of each <a title=testpage>test page</a> to the iframe.
4. The <a title=testpage>test pages</a> and their resources are fetched directly from the locally cloned repository on coremob.org.
5. Once it has finished running the <a title=testpage>test page</a>, the <a title=testframework>test framework</a> updates the <a title=testrunner>test runner</a>.
6. When all <a title=testpage>test pages</a> have been run, the <a title=testrunner>test runner</a> transfers the results to browserscope.org or elsewhere.
7. The results are publicly available on browserscope.org or can be kept private if required.
#### Pros ####
- satisfies all requirements
#### Cons ####
- Not relying on the <a title=jsonapi>JSON API</a> exposes us to maintenance costs.
Conclusion
----------
To decide between the different solutions. it depends on resources available and motivation of W3C to provide (and maintain) certain features into their API.
1. liaise with W3C about the different requirements we have for the <a title=jsonapi>JSON API</a>:
* support for cross-origin messages,
* support listing of <a title=testpage>test page</a> sub-resources.
2. Determine test coverage of W3C specs. Create a doc about this. Liaise with W3C.
3. Determine test coverage, licensing terms, and runner compat of non W3C specs. Notably test262.
4. define CG participant resources to build runner.
</body>
</html>