Cypress: Vanilla or Cucumber?

[varl]

Based on the discussion in the meeting #134 we need more time to properly evaluate if using Cucumber together with Cypress is something we want to heavily invest in.

This topic is to share experiences, discuss pros and cons, and see if we can figure out a way forward from here.

First off, let's state some questions we would like to answer. Post them below and I will add them to the list here.

To start off I will note a few from the frontend meeting.

Questions

• What do we want to use the feature files for?
• Is it possible to use Cucumber for some things but Vanilla for others in the same suite?
• What is the actual overhead of Cucumber?
• How can we report our test results automatically to Zephyr in JIRA?
• Have we had experiences where the feature files themselves have been really valuable?

Current usages

These are the usages I am currently aware of, let me know if an app or library is missing and I will amend the list.

Team Platform

Cucumber

• UI
• Import/Export
• SMS Configuration
• WHO Data Quality
• Cache Cleaner

Team Analytics

Vanilla

• Data Visualizer
• Maps

Cucumber

• Dashboards

Team Tracker

Cucumber

• Capture

[Mohammer5]

What do we want to use the feature files for?

• Replace writing zephyr test steps, include a link to a feature file (/w lines to the relevant scenarios if applicable) instead
• Automated detailed test failure report to QA in JIRA

Is it possible to use Cucumber for some things but Vanilla for others in the same suite?

• We could use feature files for DHIS2-XXXX jira issues and use vanilla cypress for all TECH-, LIB-, and other XXXX issues
  • DHIS2-XXXX feature files could use the same tagging mechanism that QA is already using to tag which scenario/feature belongs to which issue

What is the actual overhead of Cucumber?

• Parsing time of feature files is a biggie!
• Onboarding: Cognitive overhead when new to feature files
  • It takes a bit of experience to figure out how to structure feature files so that the tests actually work
  • Feature files could be proposed by non-techies, but they'd never be in their final form (see point above)
• For some people the mental overhead of having to think in different structures instead of just writing plain vanilla cypress tests causes an increase in time to write these tests
  • (For me it's the other way, it takes a bit more time to write the feature files first, but then I have a very clear structure and just need to implement the individual steps without having to think about the "bigger picture" (the full scneario))
• We have a higher maintenance burden when it comes to support libraries (the cli utils tool) as we have to maintain code related to cypress AND cucumber

How can we report our test results automatically to Zephyr in JIRA?

• Regardless of whether we want to use feature files or not, cypress has a mature reporting support. We could leverage on existing reporters or write a custom one that sends the data to the appropriate apis of our Jira

The README of cypress-cucumber-preprocessor library states the following:

The cypress-cucumber-preprocessor can generate a cucumber.json file output as it runs the features files. This is separate from, and in addition to, any Mocha reporter configured in Cypress.

These files are intended to be used with one of the many available Cucumber report generator packages. Seems to work fine with both https://github.com/jenkinsci/cucumber-reports-plugin and https://github.com/wswebcreation/multiple-cucumber-html-reporter

Have we had experiences where the feature files themselves have been really valuable?

• I didn't have to write zephyr test steps
  • This is of course a personal experience, but everytime I'm writing a manual zephyr test, it's really frustrating. The UI is just bad and it eventually contains exactly the same stuff that I would've put into a feature file anyways, just without being connected to actual test code
• I find writing tests this way a lot easier.
  • It definitely took some time to get used to it. But my workflow now is quite nice:
    • I don't group steps by scenario mentally anymore, I look at each step independently when looking at the test code.
    • When writing feature files, I "just" have to think about what the app should be capable of and I write that down.
      I don't have to think about test code whatsoever while thinking about how the app works.
      When it comes to writing the test code after I finished the feature files, I first copy all the individual steps and create the empty execution functions. I then just go from top to bottom, from step to step and implement that step, regardless of its context
    • => To me it feels it actually speeds up writing tests
      • Test code is chunked into logical parts which are easy to find
      • The title of the steps is included in the cypress GUI, I know exactly how to quickly find the responsible code
• It's very easy to understand what a test should do - when not having worked on a feature yet - without having to read through actual code

[awgaan]

Have we had experiences where the feature files themselves have been really valuable?

I rewrote the import-export app, and I found the feature files very helpful to read when I started looking at the app and tried to determine what it was/(should be) capable of.

When I added some new features (like tracked entity instance import/export) I wrote feature files to help me understand the exact requirements.

Background

No Cucumber or Cypress experience before working on the import-export app. For me, the Cucumber feature files were not a cognitive overhead, but rather made it easier to see the whole picture.

What is the actual overhead of Cucumber?

@Mohammer5 mentions that the parsing time of feature files is significant. I don't get why. A parser cannot possibly be that slow. It's just matching our humanly readable Cucumber syntax into a list of already defined functions. Or am I missing something?

Stubbed Cypress responses

I think we should be careful trusting passed Cypress results where the tests use stubbed responses (at least for apps). I worked on something else for a while before the 2.35 freeze and came back to a ton of broken functionality in the import-export app that I thought worked because the Cypress tests passed, but forgot to consider that the 2.35 API responses did not match the 2.34 stubbed responses.

[Mohammer5]

@Mohammer5 mentions that the parsing time of feature files is significant. I don't get why. A parser cannot possibly be that slow.

@awgaan (copied from slack:) I assume it's the additional IO processes of reading files files. In my experience, that's always the slowest part when having to deal with parsing

• The files have to be read
• The plugin has to
  • parse the individual steps, taking scenarios with multiple examples into account
  • create adhoc test files with describe and it, re-arranging the scenarios
  • Has to find relevant test code (files in the feature's folder, common code), which requires checking some configs

And the above is happening for every single feature file. It doesn't matter too much with a couple of feature files. But with 100+ feature files and 500+ scenarios, this can take a while (a minute or two).

I think, while this is a nuisance while waiting for CI to run the tests, I don't mind the extra time here too much, I prefer having the feature files, at least when just comparing this trade-off vs feature files in isolation.

Ideally, I think our test suite should resemble the distribution mentioned here: https://kentcdodds.com/blog/unit-vs-integration-vs-e2e-tests. That means that the more realistic tests should make up the smallest part of our test suite (like cypress tests), because they're expensive. You also don't paint a wall with a tiny brush. It might be accurate, but it's not practical.

Currently, repos like UI have a ton of cypress tests, and some unit tests. That should be the other way around. That is my core consideration for all our discussions about tests. So I think we should test most functionality in jest, and a (much) smaller portion with cypress. That is the practical, common approach. Whether we use feature files or not. If we want to guard all our features in cypress it's not going to be a practical test suite, but a slow, costly to maintain one.

It might be precise to paint an entire wall with a tiny brush, but it's going to lead to fatigue, extra cost, and noone is going to want to paint anymore in the end. So realistically we can't guard all our features with cypress.

So if we can't guard all our features with our cypress tests, then what we have left is a slightly more human readable syntax for a couple of integration tests. To me, that isn't worth it.

[Mohammer5]

@ismay what about apps? Would you also integrate most of the tests that currently leverage on cypress with jest instead?

In regards to UI I 100% agree that we overshot with writing all the tests with cypress. But testing apps is a completely different scenario, which includes forms, "complex" interaction between user/app and app/backend, etc. This seems to warrant using browser-based tests.

In that scenario I'm currently in favor of using feature files. The experience @awgaan had is exactly what I was hoping for as well (someone new to the app has little to no problems understanding how the app is supposed to function)

[varl]

I think the workflow that has emerged from using Cucumber is quite good, but we don't have to use Cucumber itself to use the workflow:

1. Write documentation for how the feature should work, and get feedback from stakeholders.
2. Write tests for how the feature should work.
3. Implement the feature.
4. Run tests to verify the implementation.

The end result is that we end up with documentation that our users can read.

What we end up with Cucumber/Gherkin is documentation that a computer can read.

---

In practice, I concede that I would write the feature first, then the tests. But I would use the documentation as guide for how the feature is implemented.