Hit testing and layer ordering

[jakearchibald]

The current API has a somewhat high-level model for resolving hit testing, by giving you a transform that you can apply to your elements that puts them in the correct place.

However, this doesn't cater for layer order, which is obviously an important part of getting hit-testing right.

It feels like the high-level model should cater for this somehow.

At the very least, changing z-index of items in the subtree shouldn't trigger an additional paint event.

[schenney-chromium]

Yes, this issue was raised on the spec PR I think. The children in the canvas subtree need to be in the right order, I think reversed to their draw order, so that hit testing goes to the right place in the present of overlap. We could automate this but for now we're relying on developers to manage it.

[chrishtr]

Hit testing follows regular DOM order. That's great, because html-in-canvas will not have different hit testing or a11y behavior than other DOM. If a developer draws two canvas children in an overlapping manner, they need to make sure that the z-indexing of them in the DOM agrees with the order in which they call the canvas APIs to draw the elements. I think that's fine.

I also think it's fine for the paint callback to be called when z-indexes on the DOM children of the canvases change--after all, DOM content also repaints when that happens.

[jakearchibald]

My point is, it seems weird that the API gives you a transform to help manage the hit testing, but when it comes to layer ordering it's like "nah you're on your own".

It's a pretty inconsistent thing to do.

[jakearchibald]

I also think it's fine for the paint callback to be called when z-indexes on the DOM children of the canvases change--after all, DOM content also repaints when that happens.

I'm not sure this argument makes sense since you can say the same about transform, and that doesn't (for similarly good reasons) trigger a paint event.

[chrishtr]

My point is, it seems weird that the API gives you a transform to help manage the hit testing, but when it comes to layer ordering it's like "nah you're on your own".

It's a pretty inconsistent thing to do.

The way I see it, the API doesn't "give you a transform". It merely says that hit testing follows the DOM hit testing algorithm (which takes into account transform, z-index and layout.)

Ordering of children drawn into the canvas inherently needs developer work to draw it in the right order when executing a WebGL/WebGPU/Canvas2D program, because the commands are not sequential and are embedded in a larger scene graph only the

[chrishtr]

I'm not sure this argument makes sense since you can say the same about transform, and that doesn't (for similarly good reasons) trigger a paint event.

Transform is special, because the API expects transforms to be set during the paint callback. That's because we expect that in practice these programs compute the transform at the time of executing the WebGL/WebGPU/Canvas2D program. If transform then caused paint to be called, it'd be an infinite loop of redundant paint callbacks. The API's solution is to not invalidate when transform alone changes.

[schenney-chromium]

There are two things that make automated handling of hit test ordering very hard to implement in a way that will work for all use cases.

The first is that we don't really know what is actually on the screen in a canvas, because the author can clear some or all of the canvas at any time, or paint something on top, etc. We would need a fully featured canvas hit testing system to address this, which has been proposed (I think) but never moved far enough forward.

And there may be children of the canvas intended to provide a11y fallback content and we do not want to make that any harder than it currently is.

Having said all that, we might get the right result much of the time if calling drawElement causes the element to move to be first of the children. We would need to do so without triggering relayout etc, though to get correct hit testing we would need updated layout. Maybe an option could be provided to perform such actions.

[jakearchibald]

Transform is special, because

I know. I was suggesting z-index could also be special for similar reasons. Although @schenney-chromium's suggestion would also work.

[chrishtr]

I know. I was suggesting z-index could also be special for similar reasons. Although @schenney-chromium's suggestion would also work.

Ok. If it makes sense overall, making z-index special and not trigger onpaint could be added.

[jakearchibald]

Although, now I think of it, changing the element order might mess with accessibility, eg tab order.

The UA stylesheet could make children of the canvas position relative, then the paint command could return an object that could be Object-assigned to el.style. It would include transform and z-index.

[progers]

Small clarification: z-index changes wouldn't cause paint to fire today because it doesn't change the painted output of an individual element.

An option is for the canvas to maintain an ordered list of which elements have been drawn last, and then provide an API for updating the z-index of an element so that the hit test order matches the drawn order. Would this approach solve this issue?

Using a similar pattern as the current transform approach, that would look like:

  canvas.onpaint = () => {
    ctx.reset();
    let transform = ctx.drawElementImage(element, 10, 20);
    element.style.transform = transform.toString();
    element.style.zIndex = canvas.getChildDrawElementImageOrder(element);
  };

I considered having getChildDrawElementImageOrder(element) take into account the current z-index values of all of the children of the canvas, but that could lead to an ever-increasing z-index when drawing two elements every frame.

@jakearchibald 's approach would be a small change to the current API shape, as it would change the return type of drawElementImage from a DOMMatrix to a json object that looks like { transform: 'matrix(...)', zIndex: '5' }. The example would then look like:

  canvas.onpaint = () => {
    ctx.reset();
    let updatedStyle = ctx.drawElementImage(element, 10, 20);
    Object.assign(element.style, updatedStyle);
    // Alternatively:
    // element.style.transform = updatedStyle.transform;
    // element.style.zIndex = updatedStyle.zIndex;
  };

@jakearchibald , have I captured your thoughts correctly?

Both of these options do seem reasonable to me. I prefer the the less-prescriptive approach of adding the lower-level APIs, but Object.assign is elegant.

[jakearchibald]

Small clarification: z-index changes wouldn't cause paint to fire today because it doesn't change the painted output of an individual element.

Ohh, good to know. Bad assumption on my part. I thought that since transform was called out specifically, then other things would trigger a paint.

Using a similar pattern as the current transform approach, that would look like:

  canvas.onpaint = () => {
    ctx.reset();
    let transform = ctx.drawElementImage(element, 10, 20);
    element.style.transform = transform.toString();
    element.style.zIndex = canvas.getChildDrawElementImageOrder(element);
  };

The thing I don't like about that is that drawElementImage returns you something that you can use to put your element in the 'right' place. Except it isn't the right place, since it doesn't handle layer ordering. Developers would need to know this, and know there's some separate API they also have to use to achieve what they're trying to achieve.

In my proposal, drawElementImage returns you the things you need to apply to put the element in the right place.

@jakearchibald , have I captured your thoughts correctly?

Yes.

There may be a better solution out there though. I'm trying to avoid the current case where the developer is lulled into a false sense of security, where the API looks like it's giving you the details you need to position the element, but in reality the z-index is missing.

An alternative could be an additional API, ctx.placeElementImage(el, …), which does the same as drawElementImage, but also mutates el.style.

[progers]

could lead to an ever-increasing z-index when drawing two elements every frame.

This is an issue for any approach that sets z-index at the time of drawing an element. When not drawing one element repeatedly, the updated z-index will need to be higher than any previously-set z-index.

We could add syncDrawElementStyles() to allow for an explicit sync at the end of painting, when global state is known, like the following (example requires Chromium with this patch to work):

<!doctype html>
<style>
@keyframes anim {
  from { background-color: pink; }
  to { background-color: lightblue; }
}
#bg {
  width: 100px;
  height: 100px;
  animation: anim 2s linear alternate-reverse infinite;
}
</style>
<canvas id="canvas" style="width: 200px; height: 200px;" layoutsubtree>
  <div id="bg"></div>
  <div id="fg">hello world</div>
</canvas>

<script>
  const drawElementHistory = new Map();
  function didDrawElement(element, drawTransform) {
    drawElementHistory.delete(element);
    drawElementHistory.set(element, drawTransform);
  }
  function syncDrawElementStyles() {
    let zIndex = 0;
    for (const [element, drawTransform] of drawElementHistory) {
      element.style.transform = drawTransform.toString();
      element.style.zIndex = (++zIndex);
    }
  }

  const ctx = canvas.getContext('2d');
  canvas.onpaint = () => {
    let transform = ctx.drawElementImage(bg, 0, 0);
    didDrawElement(bg, transform);
    transform = ctx.drawElementImage(fg, 0, 0);
    didDrawElement(fg, transform);

    syncDrawElementStyles();
  };
</script>