Fixing CI

[bsamuel-ui]

The biggest blocker to timely releases is the state of the CI.

Issues with CI

• Builds are failing on windows
• Builds take 30 minutes
• Maintainers have limited access to CI machines

Options

Sticking with what we've got

• Effort: 0
• Con: All of the above

Set up new appveyor account and fix it there

• Effort: probably less than github actions
• Risk: I haven't used appveyor
• Con: Only addresses windows issues

Github actions

• Effort: rewrite the CI config
• Pro: runners for MacOS, Windows, Linux
• Possible benefit: forks can run their own CI? (Can we confirm this?)
• Risk: Can we get prebuilt pythons? If not, building is still sloooow
• Warm embrace of: github's vertical integration

[bsamuel-ui]

Turns out @dschep was already investigating this: dschep@dbd2717

[miketheman]

How would we feel cutting out the failing appveyor tests for now while we figure the rest out? Circle is pretty good, albeit lengthy, and gives is the green confidence.

I was looking at how we might split the large mega test file into smaller chunks that could be parallelized, but the nature of the test suite is currently very end-to-end.

[bsamuel-ui]

I think that makes sense, it's a false negative at the moment. (Edit: changed it.)

[miketheman]

Builds take 30 minutes

I've been thinking about this a little more, and here's some thoughts:

We could pursue reducing ~4-5 minutes via using a different base image that contains both python & nodejs that we want, as well as adding dependency caching for things like pipenv, poetry, npm deps.
That still leaves us with the large 28-30 minutes of the test suite, so I figured it might prove useful to do something like this in CircleCI using Pipelines.

First, split the large test.js file up based on some categorization, maybe "pip, pipenv, poetry, docker, slim"?
The tests/ directory already has some fixture projects there, we might want to move some files around.
There's a fair amount of overlapping testing though, so I'm not certain where it's best to draw these lines yet. Very open to advice here!

Then run a pipeline that has these workflow steps:

1. create the base environment for testing (install deps)
2. run lint (fast)
3. run tests in parallel based on the splits

This ought to help with the overall test suite duration on CircleCI - it won't do much for people trying to run it locally, other than provide the facility to run a subset of the test suite locally, which is also good.

[bsamuel-ui]

I'm going to try a branch using Github actions; this may wind up doing more work since we'll be running the tests against multiple environments.

I think breaking out the tests is very worthwhile, but I want to limit this specific issue to getting a solid CI base to work from.

[bsamuel-ui]

Progress: Github actions are green on Ubuntu, Mac and Windows.