jasmine_web_test_suite()

[dgp1130]

The rules_webtesting workspace currently supports several different languages, but one place it is lacking is JavaScript / TypeScript. Since these are the required languages for working with browsers, many developers would prefer to work within them instead of having their end-to-end tests written in a different language. rules_nodejs already provides a lot of the core Bazel functionality for working with JavaScript / TypeScript, so I think this should be fairly straightforward to support.

Currently if users want to do end-to-end testing in JS/TS, their options mainly include:

1. Use cypress_web_test().
   • This is a bit more opinionated test setup as I understand it.
   • I believe Cypress also versions it's browsers directly via NPM, which is not very Bazel aware.
2. Use Puppeteer with any kind of nodejs_test().
   • Since Puppeteer can be treated as a simple library, this mostly works out of the box.
   • I assume Playwright would work similarly, but I'm not very familiar with it.
   • Puppeteer versions it's browsers directly via NPM, which is not very Bazel aware.
3. Use Protractor.
   • It's official support status is unclear.
4. Use WebDriverJs or WebDriverIO with any kind of nodejs_test().
   • This requires setting up a WebDriver server and downloading the relevant browsers and drivers, which can be quite involved and is the main problem rules_webtesting solves.

The first two are probably the most attractive, but both of these ship browsers and their drivers via NPM, so any edit to the package.json file causes them to be redownloaded. Add in the need to run a bare npm install to bring package-lock.json up to date with any changes, and npm ci in certain situations, plus the fact that rules_nodejs WORKSPACE rules also run npm install, means that even minor, unrelated package.json changes can result in 3-4 downloads of the relevant browsers. That can easily exceed 500MB. Even in Silicon Valley this can still take 15+ minutes for me from a trivial package.json edit on my home network (because American ISPs are highly competitive with each other). I'm sure others feel this pain even worse.

rules_webtesting already solves this problem since browsers are versioned via Bazel and not dependent on package.json. We just need a way to author tests in JS/TS.

I propose a new jasmine_node_test_suite() rule, similar to the existing *_web_test_suite() rules, but just using jasmine_node_test(). It would simply set up WebDriver in the same fashion and set the relevant environment variable. We would leave it up to users what library they want to use to connect to that instance (WebDriverIO, WebDriverJs, etc.).

I implemented this strategy in one of my projects recently and I think it worked out pretty well TBH. Most of the work went into re-implementing wrap_web_test_suite() and figuring out how to use WebDriverIO like a simple library (the latter would the user's problem, not rules_webtesting's problem, so nothing we need to worry about there). In particular, it makes editing package.json a much less painful experience.

rules_nodejs also supports a few other testing systems, so I want to call out some of the alternative options:

• nodejs_test() - This is more generic than jasmine_node_test() and arguably a better candidate to avoid tying rules_webtesting to any one particular JS/TS testing framework. Unfortunately, it's also less useful and requires manually integrating the testing framework, which is a lot of extra work to ask for a user.
• Jest - This appears to be possible in rules_nodejs, but does not have an official implementation. A Jest user would most likely be able to use it with rules_webtesting given a similar amount of effort as was required for my experiment, and I think that's fairly reasonable given its support status in rules_nodejs.
• Karma - Tests run in the browser, which doesn't make sense with WebDriver.
• Protractor - Already provides integration with WebDriver, so rules_webtesting is not useful. Also it's support status is currently unclear.

I personally think jasmine_web_test_suite() is the best option here, but I'm interested to hear other opinions on this. Arguably we could have a more direct integration with an actual JS/TS implementation of WebDriver, but I don't thinks that's strictly necessary or would improve the developer experience much. I think it's easier and more flexible for users if rules_webtesting just sets up the WebDriver server and lets the user figure out how to connect to it.

If this seems like a good direction to take, I would be happy to take a stab at the implementation. I already did most of the same work for my project, so upstreaming it here would make things simpler on my end and allow others to more easily use the trick.