WPT CLI: --headless / --no-headless

[foolip]

Spinoff from web-platform-tests/results-collection#603.

As a user of ./wpt run, it would be great to be able to not have a lot of windows be created, as it makes it hard to do other work while the tests are running.

I would propose that --headless is the eventual default with --no-headless as the opt-out, but having the ability to enable it is the first step.

[Hexcles]

Or Xvfb

[jugglinmike]

Resolved via gh-13076

[whimboo]

@foolip or @jugglinmike, at which screen size do we run the headless tests? Might it be set to 800x600 or larger?

I ask because I wonder about our test failures for Firefox here:
https://wpt.fyi/results/webdriver/tests/maximize_window/maximize.py?label=experimental

Webdriver tests start with a window of 800x600 in size, so maximizing it won't change the size of the window and as such the tests aren't correct.

[foolip]

@whimboo I don't know, I suppose the default size in headless mode is browser-specific, but 800x600 sounds like a possible default. But whatever the size is, isn't the problem that maximizing doesn't make any sense in headless mode? What would the size become?

[whimboo]

You can fake the screen size at least for Chrome (--window-size) and Firefox (--headless-width, --headless-height). I don't know about other browsers.

So if we run the web-platform-tests only in headless mode as it looks like when I check the results of webdriver tests for Firefox (https://wpt.fyi/results/webdriver/tests/maximize_window?label=experimental), we won't even get any qualified test results for window manipulation tests of those browsers which don't have headless support yet.

[foolip]

If it wasn't clear from this issue and linked issues, we currently don't use headless, --no-headless is passed in tools/ci/taskcluster-run.py and in the Buildbot setup.

But it sounds like if we want to test using headless, we have command line arguments that make that possible.

[whimboo]

Oh ok. Then some other environmental settings might cause those problems. Sorry for bothering you on this issue then, and thanks for clarifying!

[whimboo]

Well, it's actually not headless but xvfb which is used. And I don't see that a resolution is specified. @foolip which resolution is actually used for wpt and specifically webdriver tests? I always struggle to find log files. :/

[foolip]

@jgraham can you help @whimboo find log files? Maybe we're not logging enough, or linking them clearly enough?

@gsnedders do you know the screen resolution and browser size we use in all cases? Is it only 600x600 as the browser size which we set or reftests, and everything else is haphazard?

[jgraham]

The logs are https://tools.taskcluster.net/groups/Zr_bW3aPRDKtdc48BidQ7Q

[jgraham]

(for example)

[whimboo]

Ok, sorry for the noise here. This will clearly be the last comment to close out the discussion. Given that xvfb is used to run the wdspec tests, I finally know why we are failing. Currently I'm reworking all the window manipulation commands, which should hopefully make us be green for the upcoming Firefox 65 release.