Add compute shaders

[RandomGamingDev]

Increasing access

Although WebGL doesn't have official compute shaders, they can be emulated using a few vector shaders, a FBO, and fragment shaders for the actual calculations.

While p5.js's focus isn't computation, this would be perfect for many types of rendering (e.g. raytracing, raymarching, and certain types of culling). It wouldn't provide a speed benefit compared to doing it yourself, but it would mean less boilerplate being required, allow for computations for computation visualizations which are also popular in p5.js, make it easier to create more advanced graphics in p5.js, and also introduce a lot of beginners to the topic of compute shaders (part of p5.js's key principles is to help beginners, which is also part of why shaders, and attempts to make shaders easier, which is why I think this would work well).

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

Feature request details

Create computer shader equivalents to createShader() and loadShader() (e.g. createComputeShader() and loadComputeShader()). p5.js would handle the boilerplate in terms of setting up the vertex shaders, part of the fragment shader, and FBO, meaning that the user would only get specific variable inputs and outputs, with the output getting written to a TypedArary buffer.

[Vaivaswat2244]

Hey, @RandomGamingDev,
maybe for making the computeShader feature, the following approaching might help,
webgl/p5.Shader.js

initComputeFBO(width, height) {
  const gl = this._renderer.GL;

  this._computeFramebuffer = gl.createFramebuffer();
  gl.bindFramebuffer(gl.FRAMEBUFFER, this._computeFramebuffer);

  // Creating Texture to store compute shader results
  this._computeTexture = gl.createTexture();
  gl.bindTexture(gl.TEXTURE_2D, this._computeTexture);
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, width, height, 0, gl.RGBA, gl.UNSIGNED_BYTE, null);

  // FBO implementation
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);

  
  gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, this._computeTexture, 0);

  // Check for FBO completeness
  if (gl.checkFramebufferStatus(gl.FRAMEBUFFER) !== gl.FRAMEBUFFER_COMPLETE) {
    console.error("Failed to initialize compute framebuffer");
  }

  gl.bindFramebuffer(gl.FRAMEBUFFER, null);
}

computeShader(width, height, callback) {
  const gl = this._renderer.GL;

 
  gl.bindFramebuffer(gl.FRAMEBUFFER, this._computeFramebuffer);
  gl.viewport(0, 0, width, height);
  if (callback) callback(this._computeTexture);
  gl.bindFramebuffer(gl.FRAMEBUFFER, null);
}

[Vaivaswat2244]

@davepagurek any advice on this? or something I should change?

[Rishab87]

@davepagurek , if no one's worknig on this can I try solving it?

[davepagurek]

hi @Rishab87 and everyone! I think before we jump right into implementation, there are a few details to iron out, which I'd love all of your help with if you're interested!

• The general behind the scenes setup: something like what @Vaivaswat2244 suggested is a good start if we're using a fragment shader to do computation. it's possible we even want to use regular p5.Framebuffers initialized with FLOAT data to piggyback off of existing rendering code to start with, but a raw WebGL approach could be ok too later on (see also @perminder-17's recent work on a minimal filter shader runner for use in 2D mode.) One other alternative, if the main goal is to get numbers readable by JavaScript, is to use WebGL 2 transform feedback and a vertex shader, but this is less good if you'd like to read data in another shader to keep everything on the GPU, so I lean towards a fragment shader approach too.
• The interface exposed to users: how do we expect users to specify inputs and outputs? For example, if a user is doing something like a particle simulation where each particle has a position and velocity, how do we pass in the existing state, what should the output of the shader be onto the data texture, and how are we letting the user retrieve that data?
  • Can we make a default shader and use shader hooks so that users are able to write only the body of their computer shader with as little boilerplate as possible?
  • An additional technical challenge: how, in the shader, do we want to handle outputting more values than we can fit in one pixel (assuming a fragment shader is doing the computation)?
• The object model: Is there just a global function to run a compute shader, and if users alternate running different compute jobs, do we resize/reuse a single texture under the hood? Would we instead make each compute task a separate object with its own internal texture that users create upfront and then reuse? (I think I lean towards the latter, especially if we have to end up storing info about how to interpret texture output data.)

Let me know what your thoughts are!

[Rishab87]

@davepagurek , I'm more inclined towards fragment shader approach as we can use some existing p5.js code and overall its a great balance between being powerful being easy to understand

Talking about interface exposed to users:
To make it beginner friendly and reduce boiler plate we can provide a high level api like this:

computeTask = createComputeTask(1000, `
    // User-defined compute shader code
    vec2 updatePosition(vec2 position, vec2 velocity) {
      return position + velocity;
    }
    
    vec2 updateVelocity(vec2 velocity) {
      return velocity + vec2(0.01, 0.0);
    }
    
    void main() {
      vec4 particle = texture2D(u_particleData, gl_FragCoord.xy / u_resolution);
      vec2 position = particle.xy;
      vec2 velocity = particle.zw;
      
      position = updatePosition(position, velocity);
      velocity = updateVelocity(velocity);
      
      gl_FragColor = vec4(position, velocity);
    }
  `);
}

and then get the output like this:

let particles = computeTask.getData();

Internally we'll be setting up FBO, initialize particle data and after computation swapping input and output fbos

To handle more outputting values than we can fit in one pixel, maybe we can use multiple pixels to represent a single particle though I'm not sure

Object Model:
I recommend creating separate objects for each compute task. This approach provides better encapsulation and flexibility

I'm completely new to shaders, webgl etc so please correct me if I said anything wrong!

[davepagurek]

That sounds good so far! Right now it works well because position and velocity in 2D pack perfectly into a floating point vec4. Would this setup work easily if the simulation was in 3D? Or even if you just had position?

From a technical standpoint, it gets a lot harder to output data that takes more than one pixel to store. This may not be the best way, but one idea could be, if we make the user just specify a function that returns a struct with the state data. Possibly even just all floats for simplicity, e.g.:

struct State {
  float px;
  float py;
  float pz;
  float vx;
  float vy;
  float vz;
};

State compute() {
  State result;
  result.px = /* something */;

  // ...

  return result;
}

...then, under the hood, we could interleave the outputs into adjacent pixels. So we'd automatically generate a main function that looks something like:

void main() {
  State result = compute();
  if (mod(gl_FragCoord.x, 2.0) == 0.0) {
    gl_FragColor = vec4(result.px, result.py, result.pz, result.vx);
  } else {
    gl_FragColor = vec4(result.vy, result.vz, 0.0, 0.0);
  }
}

That could let us output more data than one pixel can hold, but also would then require us to automatically generate something similar to decode texture data (e.g. State getState(int index)) rather than getting users to manually extract it via texture(mySampler, myCoord). That's also potentially feasible for us to generate in a similar fashion to the encoder.

Anyway, thats just one idea to consider! I'm open to others too.

[Rishab87]

yes, you're right my solution was kind of rigid, a user defined struct is a great approach, user api will look like something like this then?:

const instance = new computeShader({
  properties: {
    position: { type: 'vec3' },
    velocity: { type: 'vec3' },
    mass: { type: 'float' },
    color: { type: 'vec4' }
  },
  computeFunction: `
    void updateParticle(inout vec3 position, inout vec3 velocity, inout float mass, inout vec4 color) {
      position += velocity;
      // ... other update logic
    }
  `
});

based on this we'll generate some glsl code and generate a function to get the state of a particle and then user can access particels something like this:

let particles = particleSystem.getParticles();
point(particle.position[0], particle.position[1], particle.position[2]);

Overall this approach sounds good to me!