The object model

[almarklein]

To define the protocol, we must first agree on an object model. The part that we seem to (more or less) agree about can be illustrated like this:

The canvas is the toplevel window to draw something to. It can be a physical window on screen, or a virtual (off-screen) canvas. The space on the canvas is divided to accommodate for multiple visializations (what Matlab calls subplots). Note that there is one region for the actual scene, and another for the ticks, labels etc. Layout is tricky.

A visual is an object that describes one "thing" being visualized. E.g. a line, markers, an image, volume, mesh, etc. The actual data for a visual is stored in buffers and textures, so that data can easily be shared. Visuals have properties that affect their appearance. Each visual also has a transform.

A proposal: the view also defines the camera, the interaction model (orbit, trackball), and the toplevel visual (or a list of visuals if we don't have a scengraph).

Points to address:

• Name for canvas/window/figure (my fav is canvas).
• Name for view/subplot (my fav is view, since we may want to use "plot" to create line+markers).
• Do we leave layout for the client lib, or does it specify a high-level layout and leave the hard work for the backend?
• Do we support a scenegraph? I suspect that our target audience won't care much, and a client lib can implemented on top. Plus we can add it later, relatively easily, I think.
• Do we collect the appearance-props of a visual in a "material" as in TheeJS and PyGfx? That way you can have multiple objects with the same material that can be changed using one uniform update (if the backend supports this). Though it also complicates the protocol, and not a typical use-case.

[rougier]

╭────────────────────────────────────╮                 
│ ○ System window                  × │                 
├────────────────────────────────────┤                 
│ Canvas                      ●──────┼─▶ Top margin   
│     ┌──────────────────────────┐   │                 
│     │                          │   │                 
│     │                          │   │                 
│     │     Canvas viewport      │ ●─┼─▶ Right margin 
│  ●  │                          │   │                 
│  │  │                          │   │                 
│  │  └──────────────────────────┘   │                 
│  │                    ●────────────┼─▶ Bottom margin
└──┼─────────────────────────────────┘                 
   └───▶ Left margin                                       

┌───────────────────────────────────────────────────┐                    
│ Canvas viewport                                  ●┼─▶ Padding (canvas)
│ ┌───────────────────────┬───────────────────────┐ │                    
│ │ ┌ Figure viewport ──┐ │ ┌ Figure viewport ──┐ │ │                    
│ │ │ ┌───────────────┐ │ │ │ ┌───────────────┐ │ │ │                    
│ │ │ │ Data viewport │ │ │ │ │ Data viewport │ │ │ │                    
│ │ │ │               │ │ │ │ │               │ │ │ │                    
│ │ │ │               │ │ │ │ │               │ │ │ │                    
│ │ │ │               │ │ │ │ │               │ │ │ │                    
│ │ │ │               │ │ │ │ │               │●┼─┼─┼─▶ Padding (figure)
│ │ │ └───────────────┘ │ │ │ └───────────────┘ │●┼─┼─▶ Margin (figure) 
│ │ └───────────────────┘ │ └───────────────────┘ │ │                    
│ └───────────────────────┴───────────────────────┘ │                    
└───────────────────────────────────────────────────┘                    

[rougier]

At the lowest level, we can deal only with canvas and viewport I think. And only with native units (physical or logical pixels for canvas/viewports).

[djhoese]

The canvas is the toplevel window to draw something to.

Be careful of using window here. I think most people would consider a window the top level system (title bar, etc) window like in @rougier's diagram. A widget may be more universally understood.

One other thing is that you may want to look into using the new github mermaid functionality supported in markdown: https://github.com/mermaid-js/mermaid#class-diagram-docs---live-editor
The above diagrams probably can't be done with this, but something to consider in future READMEs and issue posts.

[rossant]

At the lowest level, we can deal only with canvas and viewport I think. And only with native units (physical or logical pixels for canvas/viewports).

Agreed. I like your terminology.

Visuals, props, buffers, textures should also be part of the object model. Still unsure about scene graph, views, cameras, interaction model: I would say these could be part of the client. However, I think we do need an abstract model for data transformations. I'll try to propose something before the next meeting.

[rossant]

• Name for canvas/window/figure (my fav is canvas).

+1 for canvas

• Name for view/subplot (my fav is view, since we may want to use "plot" to create line+markers).

viewport, as suggested by Nicolas?

• Do we leave layout for the client lib, or does it specify a high-level layout and leave the hard work for the backend?

Maybe, yes.

• Do we support a scenegraph? I suspect that our target audience won't care much, and a client lib can implemented on top. Plus we can add it later, relatively easily, I think.

Maybe not, but transforms would be important to support GPU-accelerated interaction.

• Do we collect the appearance-props of a visual in a "material" as in TheeJS and PyGfx? That way you can have multiple objects with the same material that can be changed using one uniform update (if the backend supports this). Though it also complicates the protocol, and not a typical use-case.

I'd say no.

[almarklein]

viewport, as suggested by Nicolas?

+1

Be careful of using window here.

Good point. A canvas is the toplevel rectanglular region, from the point of view of the viz. It can be provided by an offscreen buffer, a widget embedded in an app, or as the content of a toplevel window.

[almarklein]

At the lowest level, we can deal only with canvas and viewport I think.

I'm not sure what you mean by at the "lowest level" here.

[rougier]

I meant we can have a "layered" communication protocol like the OSI mode. At the lowest level, we communicate only in terms of elementary shapes (markers, lines, polygons, images, text) and viewports (clipping & camera). At a second level, we can add transforms, colormaps, etc. But even the first level would be nice to have if it is fast enough, and it is theoretically sufficient to produce any figure even though most of the work would be left to the client.

[almarklein]

I'm not sure if having multiple levels in the protocol is a good idea, tbh. If I understand what you mean corerctly, it adds complexity for both client and backend, because they must support each layer. A client needs to do more work with the lower level, and a backend will need to do more with a higher level. So if we have two levels, some functionality will need to be implemented both on the client and backend.

[rossant]

Alternatively, we could have a single "layer", but optional features. A backend would or not support transforms, for example.

Anyway, here are some of my notes about the object model:

• Protocol
• YAML code example
• Proof of concept of a Datoviz renderer running the YAML code example

Beyond canvas and viewport, we should add visuals, props, arrays, and transforms to the model. Then I think we cover almost everything. Buffer and textures should actually be renderer-specific objects and not part of the protocol (GPU renderers would map arrays to buffers/textures depending on the number of dimensions).

[djhoese]

Feel free to tell me I'm too out of the loop to understand, but if every other component is an abstract concept, why can't buffers/textures ("data containers"?) be abstract concepts too?

[rossant]

They could, but "buffer" and "texture" are typically used in the context of GPU rendering, whereas the protocol is meant to be independent from the underlying rendering technology. It wouldn't make much sense to talk about buffers and textures with a matplotlib backend, for example. But it's just a matter of terminology. If we call them "arrays", then we're not making any assumption about how they will actually be stored on hardware.

[rougier]

In my mind the layer does require extra work from the server or client. But if a server only do "layer 1", it will be advertised as "layer 1 server" and this is the bare minimum we can ask. It's mostly a matter of qualifying things.

[rougier]

I think buffer / texture might be a bit too GPU specific. Data + data type might be a better abstraction.

[almarklein]

As I understand it, we're targeting mostly GPU-based renderers. So the terminology for buffer and texture should be fine. Besides, we need a include a way to update data/arrays, and buffers and texture have different approaches to doing that (buffer is 1D and for a texture you can update patches). I'm not sure if we can abstract that away.

Beyond canvas and viewport, we should add visuals, props, arrays, and transforms to the model.

Yeah, each object has properties, and transforms are just a property on a visual, I presume ... but we can discuss that in more detail.