Update Section: Testing

[MelSumner]

This section is already pretty up to date, looks like someone went over it recently with the RFC-232 changes.

Some things to add:

• explanation for why the types of tests introduced in the guides go in directories with legacy names (acceptance, integration).
• explanation of why both Unit Tests and Container Tests both go in the unit directory.
• examples of nesting modules. (A module within a module).
• explanation of how to stub/mock a method or object in a unit or container test.
• mention addons that could help with testing
• testing routes: include how to handle testing the asynchronous beforeModel, model, or afterModel hooks.
• how to test Query Params
• how to test Native ES Classes
• What to do if you get the "autorun is disabled, wrap in a run loop" error message?

[kovalchik]

I was reading through the Octane testing section the other day and noticed that there's still a lot of usages of this.set. For example, in the components testing section:

// set the outer context to red
this.set('colorValue', 'red');
await render(hbs`<PrettyColor @name={{this.colorValue}} />`);
assert.equal(this.element.querySelector('div').getAttribute('style'), 'color: red', 'starts as red');

I know there's some instances where using set is required, but when setting up my tests, I was able to directly assign outer context values without any issues. Should updating the examples in the testing section to not use get and set also be a checklist item?

// set the outer context to red
this.colorValue = red;
await render(hbs`<PrettyColor @name={{this.colorValue}} />`);
assert.equal(this.element.querySelector('div').getAttribute('style'), 'color: red', 'starts as red');

[ijlee2]

@MelSumner @jenweber If it's ok with you and the Ember Learning Team, I'd like to help improve the Octane documentation for testing. I am planning to take a break from work on July 26-31 and work on personal projects.

I looked at how the Testing section currently reads from the perspective of a beginner. Some of the contents are great, but I thought the Testing section, overall, can use more polish.

I would like to reorganize existing content, create more diverse test examples (many examples only show testing what happens after get or set), and provide additional information. By doing so, I think I can address several of the items that @MelSumner listed above.

What are your thoughts?

Best,

---

Proposed structure for Testing:

• Introduction

  • Why do we need testing?
  • What tools are available to help me?
    • Ember CLI (generates blueprints)
    • QUnit, QUnit DOM
    • Mocha, Chai DOM
    • Ember Test Selectors
    • Ember Exam
    • Ember CLI Mirage
    • Ember Percy
  • How to run tests
    • --filter
  • How to debug tests

• Different types of tests

  • Unit
    • Container
  • Rendering/Integration
  • Application/Acceptance

• How to write tests

  • Good practices for writing tests
    • Writing custom assertions
  • Test helpers
    • Built-in (@ember/test-helpers)
    • Custom
  • Submodules
  • Stubs

• Testing adapter (currently does not exist)

• Testing application

• Testing component

• Testing controller

• Testing helper

• Testing model

• Testing route

• Testing serializer (currently does not exist)

• Testing service

• Testing utility

[toddjordan]

Thanks for being willing to take on making tests more accessible to new users <3

I have a few comments...

1. I think probably 3rd party tools should be a separate section at the end. Installing many of these tools change the way you write tests and since the rest of the guides are written assuming out of the box tooling I think it could cause a new learner some grief. Also, what is the criteria for including test tools?
2. The first thing in "How to write tests" should not be "writing custom assertions". In fact I've found that to be a very rare task. Also maybe that section should be more like "Advanced Topics", and also moved to the end.
3. Stubs is potentially a big topic with divergent views and should be written carefully if we want to broach in its own section.
4. in "How to run" we should go over all flags (--server, --reporter, etc)
5. The container is a common concept across all tests, not just unit. this.owner is now implicit in all tests I think. Also this.element should be doc'd. Probably "Testing Basics" is still needed

[toddjordan]

The other thing we'll need to be careful for is not providing a bunch of advanced examples interleaved with beginner focused examples. We want to make a clear path for a beginner to get bootstrapped and writing tests, and not get them bogged down in trying to understand scenarios that they may not need, or may not even be prudent for them to do at first.

[ijlee2]

@toddjordan, much thanks for your feedback. I agree with many of your points and can clarify my ideas further (number scheme matching yours):

1. 3rd party tools

• To ask for clarification, where did you vision the 3rd party tools to be? Are we looking at the end of the Introduction page, or the end of the Testing section (i.e. in my list above, after Testing Utility)?

• I'd like to show code examples that show an assertion in QUnit and, side-by-side (on the next line), the same assertion in QUnit DOM. Showing both options would document the official way as well as the more productive, easier-to-understand way.

  I think asking a developer to type this.element.querySelector(...).textContent.trim() to test a value may discourage them to write tests.

• I selected addons that, from my experience, can increase a developer/team's productivity and improve their way of writing tests. Some testing tools (e.g. Mocha, Percy), I selected because I haven't used them and wanted to learn a bit more by working on this issue.

2. How to write tests section

• Yep, listing writing custom assertions first is completely my bad. I didn't write down what I had originally had in mind for the beginning of this section.

  I want the beginning of this section to carry an inspirational message, that new developers can be productive from day one. One of the things that they can do is poke around the app and describe what they saw. The question is, how should they write tests? What are some good practices?

• There are 3 good practices that I wanted to suggest (writing custom assertion is not one of them, my bad). Because they are based on my own experience and not necessarily those of others, I wasn't sure if they belong to a personal blog more than the Ember documentation.

• Definitely, writing a custom assertion is an advanced topic, a practice that I have yet to try out myself and wanted to learn more about.

3. Stubs

• I'm not too familiar with stubs and would like to know more about the different perspectives that developers may have. Can I ask you and others for help with this section?

  ("Testing Components" and "Testing Routes" in the current Octane documentation are the only places that mention writing a stub, for a service and for a window.alert.)

4. Flags in How to run

• I think documenting all (or the important) flags can be a good idea (but can also be overwhelming to look at if formatting isn't good). An alternative may be to suggest typing ember help to see the flag options?

5. Container

• Before reading the current Octane documentation, I wasn't aware of the notion of a "container test." I always thought we have 3 types of tests: unit, integration, and acceptance.

• Where can I go/whom can I ask to learn more about this.owner and this.element? How they work is still a bit of mystery to me.

6. Advanced examples

• Yep, the simpler the better, for sure.

Thanks again!

[Gaurav0]

@ijlee2 @toddjordan @MelSumner @jenweber Can we update this issue to clarify what remaining tasks still need to be done? Thanks!

[ijlee2]

Hi, @Gaurav0. Thanks for asking.

I don't have edit permission to update Melanie's comment, so let me copy-paste it here:

• explanation for why the types of tests introduced in the guides go in directories with legacy names (acceptance, integration).
• explanation of why both Unit Tests and Container Tests both go in the unit directory.
• examples of nesting modules. (A module within a module).
• explanation of how to stub/mock a method or object in a unit or container test.
• mention addons that could help with testing
• testing routes: include how to handle testing the asynchronous beforeModel, model, or afterModel hooks.
• how to test Query Params
• how to test Native ES Classes
• What to do if you get the "autorun is disabled, wrap in a run loop" error message?

Regarding the first 2 points: In the updated Introduction section and the new Test Types section, I think we explained enough that application = acceptance and rendering = integration, that the reader will implicitly know why a certain test belongs to a certain folder.

I'm not sure about what Melanie had had in mind with regards to the points 6, 7, 8, and 9.

---

With regards to the list of to-dos that I had had in mind, I completed the Introduction and Different Types of Tests, but have been slacking off on writing How to Write Tests. I hope to have something written about good practices as a result of my upcoming EmberFest talk. I can definitely use your help with talking about test helpers, submodules, and stubs.

The remaining sections—from Testing Adapter to Testing Utility in my list—I think we could review what has been written already and see how we need to create/update/delete content. I personally would like to see sections on testing adapters and serializers because the Guides currently don't talk about them and provide examples. As a new developer, I have ideas of what I would test but don't know how I would write tests for them.

---

What are your thoughts?

Best,

[Gaurav0]

Sounds good. I put some stubs stuff in unit testing basics for now, when you get to "How to write tests" we can see if we can move it there.

[Gaurav0]

Melanie copied the issue from my analysis. 6 has to do with using async/await on functions that return promises. 7 is about query parameters, currently shoehorned into controllers. 8 is about importing a native class rather than using this.owner. 9 is all about the run loop, what it is and why it needs attention during tests.

[ijlee2]

@Gaurav0 Thanks for your explanation. Yeah, I need help from more people with writing the testing doc, so it'd be great to have your help.

I'm not sure what time zone you're in (I'm in central), but maybe we can do a video meeting to talk about how to divide the remaining tasks? I can be available at most times starting tomorrow (Sep 18, Wed). Would that be all right with you?