Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add CubeTexture blur #30165

Open
wants to merge 12 commits into
base: dev
Choose a base branch
from
9 changes: 7 additions & 2 deletions examples/webgpu_cubemap_adjustments.html
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,12 @@
cube2Texture.generateMipmaps = true;
cube2Texture.minFilter = THREE.LinearMipmapLinearFilter;

renderer = new THREE.WebGPURenderer( { antialias: true } );
await renderer.init();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a quick fix to deal with an "uninitialized renderer" warning, which should maybe be an error since it just skips my shader entirely and makes a black texture? It also says I should use renderAsync, but i'm using camera.update, which doesn't seem to have an async version? Not sure why this hasn't been a problem for fromEquirectangularTexture...


const cube2Blur = new THREE.CubeRenderTarget(cube2Texture.source.data[0].width);
cube2Blur.fromCubeTexture(renderer, cube2Texture, 0.1);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm throwing in a fixed-size blur for now to test - to have it hooked up to the backgroundBlur UI I believe we'll need some kind of cubeBlurNode that works a bit like pmremNode, automatically regenerating the texture as needed.


// nodes and environment

const adjustments = {
Expand All @@ -95,7 +101,7 @@

const custom1UV = reflectNode.xyz.mul( uniform( rotateY1Matrix ) );
const custom2UV = reflectNode.xyz.mul( uniform( rotateY2Matrix ) );
const mixCubeMaps = mix( pmremTexture( cube1Texture, custom1UV ), pmremTexture( cube2Texture, custom2UV ), positionNode.y.add( mixNode ).clamp() );
const mixCubeMaps = mix( pmremTexture( cube1Texture, custom1UV ), pmremTexture( cube2Blur.texture, custom2UV ), positionNode.y.add( mixNode ).clamp() );

const proceduralEnv = mix( mixCubeMaps, normalWorld, proceduralNode );

Expand Down Expand Up @@ -133,7 +139,6 @@

// renderer and controls

renderer = new THREE.WebGPURenderer( { antialias: true } );
renderer.setPixelRatio( window.devicePixelRatio );
renderer.setSize( window.innerWidth, window.innerHeight );
renderer.toneMapping = THREE.LinearToneMapping;
Expand Down
1 change: 1 addition & 0 deletions src/Three.WebGPU.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ export { default as Lighting } from './renderers/common/Lighting.js';
export { default as BundleGroup } from './renderers/common/BundleGroup.js';
export { default as QuadMesh } from './renderers/common/QuadMesh.js';
export { default as PMREMGenerator } from './renderers/common/extras/PMREMGenerator.js';
export { default as CubeRenderTarget } from './renderers/common/CubeRenderTarget.js';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why this wasn't exported before for WebGPU - hopefully this is okay?

export { default as PostProcessing } from './renderers/common/PostProcessing.js';
import * as PostProcessingUtils from './renderers/common/PostProcessingUtils.js';
export { PostProcessingUtils };
Expand Down
62 changes: 55 additions & 7 deletions src/nodes/pmrem/PMREMUtils.js
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ export const textureCubeUV = /*@__PURE__*/ Fn( ( [ envMap, sampleDir_immutable,

} );

const bilinearCubeUV = /*@__PURE__*/ Fn( ( [ envMap, direction_immutable, mipInt_immutable, CUBEUV_TEXEL_WIDTH, CUBEUV_TEXEL_HEIGHT, CUBEUV_MAX_MIP ] ) => {
export const bilinearCubeUV = /*@__PURE__*/ Fn( ( [ envMap, direction_immutable, mipInt_immutable, CUBEUV_TEXEL_WIDTH, CUBEUV_TEXEL_HEIGHT, CUBEUV_MAX_MIP ] ) => {
elalish marked this conversation as resolved.
Show resolved Hide resolved

const mipInt = float( mipInt_immutable ).toVar();
const direction = vec3( direction_immutable );
Expand All @@ -241,7 +241,7 @@ const bilinearCubeUV = /*@__PURE__*/ Fn( ( [ envMap, direction_immutable, mipInt

} );

const getSample = /*@__PURE__*/ Fn( ( { envMap, mipInt, outputDirection, theta, axis, CUBEUV_TEXEL_WIDTH, CUBEUV_TEXEL_HEIGHT, CUBEUV_MAX_MIP } ) => {
const getSample = /*@__PURE__*/ Fn( ( { outputDirection, theta, axis, sampler } ) => {

const cosTheta = cos( theta );

Expand All @@ -250,11 +250,11 @@ const getSample = /*@__PURE__*/ Fn( ( { envMap, mipInt, outputDirection, theta,
.add( axis.cross( outputDirection ).mul( sin( theta ) ) )
.add( axis.mul( axis.dot( outputDirection ).mul( cosTheta.oneMinus() ) ) );

return bilinearCubeUV( envMap, sampleDirection, mipInt, CUBEUV_TEXEL_WIDTH, CUBEUV_TEXEL_HEIGHT, CUBEUV_MAX_MIP );
return sampler( sampleDirection );

} );

export const blur = /*@__PURE__*/ Fn( ( { n, latitudinal, poleAxis, outputDirection, weights, samples, dTheta, mipInt, envMap, CUBEUV_TEXEL_WIDTH, CUBEUV_TEXEL_HEIGHT, CUBEUV_MAX_MIP } ) => {
export const blur = /*@__PURE__*/ Fn( ( { n, latitudinal, poleAxis, outputDirection, weights, samples, dTheta, sampler } ) => {

const axis = vec3( select( latitudinal, poleAxis, cross( poleAxis, outputDirection ) ) ).toVar();

Expand All @@ -267,7 +267,7 @@ export const blur = /*@__PURE__*/ Fn( ( { n, latitudinal, poleAxis, outputDirect
axis.assign( normalize( axis ) );

const gl_FragColor = vec3().toVar();
gl_FragColor.addAssign( weights.element( int( 0 ) ).mul( getSample( { theta: 0.0, axis, outputDirection, mipInt, envMap, CUBEUV_TEXEL_WIDTH, CUBEUV_TEXEL_HEIGHT, CUBEUV_MAX_MIP } ) ) );
gl_FragColor.addAssign( weights.element( int( 0 ) ).mul( getSample( { theta: 0.0, axis, outputDirection, sampler } ) ) );

Loop( { start: int( 1 ), end: n }, ( { i } ) => {

Expand All @@ -278,11 +278,59 @@ export const blur = /*@__PURE__*/ Fn( ( { n, latitudinal, poleAxis, outputDirect
} );

const theta = float( dTheta.mul( float( i ) ) ).toVar();
gl_FragColor.addAssign( weights.element( i ).mul( getSample( { theta: theta.mul( - 1.0 ), axis, outputDirection, mipInt, envMap, CUBEUV_TEXEL_WIDTH, CUBEUV_TEXEL_HEIGHT, CUBEUV_MAX_MIP } ) ) );
gl_FragColor.addAssign( weights.element( i ).mul( getSample( { theta, axis, outputDirection, mipInt, envMap, CUBEUV_TEXEL_WIDTH, CUBEUV_TEXEL_HEIGHT, CUBEUV_MAX_MIP } ) ) );
gl_FragColor.addAssign( weights.element( i ).mul( getSample( { theta: theta.mul( - 1.0 ), axis, outputDirection, sampler } ) ) );
gl_FragColor.addAssign( weights.element( i ).mul( getSample( { theta, axis, outputDirection, sampler } ) ) );

} );

return vec4( gl_FragColor, 1 );

} );

export const getBlurParams = ( sigmaRadians, cubeRes, maxSamples )=>{

// Number of standard deviations at which to cut off the discrete approximation.
const STANDARD_DEVIATIONS = 3;

const radiansPerPixel = Math.PI / ( 2 * cubeRes );
const sigmaPixels = sigmaRadians / radiansPerPixel;
const samples = 1 + Math.floor( STANDARD_DEVIATIONS * sigmaPixels );

if ( samples > maxSamples ) {

console.warn( `sigmaRadians, ${
sigmaRadians}, is too large and will clip, as it requested ${
samples} samples when the maximum is set to ${maxSamples}` );

}

const weights = new Array( maxSamples ).fill( 0 );
let sum = 0;

for ( let i = 0; i < samples; ++ i ) {

const x = i / sigmaPixels;
const weight = Math.exp( - x * x / 2 );
weights[ i ] = weight;

if ( i === 0 ) {

sum += weight;

} else {

sum += 2 * weight;

}

}

for ( let i = 0; i < weights.length; i ++ ) {

weights[ i ] = weights[ i ] / sum;

}

return { radiansPerPixel, samples, weights };

};
91 changes: 91 additions & 0 deletions src/renderers/common/CubeRenderTarget.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,19 @@
import { texture as TSL_Texture } from '../../nodes/accessors/TextureNode.js';
import { positionWorldDirection } from '../../nodes/accessors/Position.js';
import NodeMaterial from '../../materials/nodes/NodeMaterial.js';
import { blur, getBlurParams } from '../../nodes/pmrem/PMREMUtils.js';
import { uniform } from '../../nodes/core/UniformNode.js';
import { uniformArray } from '../../nodes/accessors/UniformArrayNode.js';
import { float, vec3, Fn } from '../../nodes/tsl/TSLBase.js';

import { WebGLCubeRenderTarget } from '../../renderers/WebGLCubeRenderTarget.js';
import { Scene } from '../../scenes/Scene.js';
import { CubeCamera } from '../../cameras/CubeCamera.js';
import { BoxGeometry } from '../../geometries/BoxGeometry.js';
import { Mesh } from '../../objects/Mesh.js';
import { BackSide, NoBlending, LinearFilter, LinearMipmapLinearFilter } from '../../constants.js';
import { cubeTexture as TSL_CubeTexture } from '../../nodes/accessors/CubeTextureNode.js';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is weird - when I tried to have a local variable named cubeTexture it didn't work - said it was a function instead (this one). Looks like it's getting properly renamed here, but apparently not? Is it somehow globally included somewhere?

import { Vector3 } from '../../math/Vector3.js';

// @TODO: Consider rename WebGLCubeRenderTarget to just CubeRenderTarget

Expand Down Expand Up @@ -72,6 +78,91 @@

}

fromCubeTexture( renderer, cubeTex, sigmaRadians = 0, poleAxis = new Vector3( 0, 1, 0 ) ) {

const currentGenerateMipmaps = cubeTex.generateMipmaps;

cubeTex.generateMipmaps = true;

this.texture.type = cubeTex.type;
this.texture.colorSpace = cubeTex.colorSpace;

this.texture.generateMipmaps = cubeTex.generateMipmaps;
this.texture.minFilter = cubeTex.minFilter;
this.texture.magFilter = cubeTex.magFilter;

// The maximum length of the blur for loop. Smaller sigmas will use fewer
// samples and exit early, but not recompile the shader.
const MAX_SAMPLES = 20;

const blurMaterial = new NodeMaterial();
blurMaterial.side = BackSide;
blurMaterial.depthTest = false;
blurMaterial.depthWrite = false;
blurMaterial.blending = NoBlending;

const weights = uniformArray( new Array( MAX_SAMPLES ).fill( 0 ) );
const dTheta = uniform( 0 );
const n = float( MAX_SAMPLES );
const latitudinal = uniform( 0 ); // false, bool
const samples = uniform( 1 ); // int
const envMap = TSL_CubeTexture( null );

const cubeSampler = Fn( ( [ sampleDirection ] )=>{

return envMap.sample( sampleDirection );

} );
blurMaterial.fragmentNode = blur( { n, latitudinal: latitudinal.equal( 1 ), poleAxis: vec3( poleAxis ), outputDirection: positionWorldDirection, weights, samples, dTheta, sampler: cubeSampler } );
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sunag thanks for getting the PMREM version of this blur working - now this one is giving me trouble. I'm getting a lot of exceptions like getOutputNode( builder ) where builder is undefined. Also this line device.queue.writeBuffer( bufferGPU, 0, buffer, 0 ); is giving me TypeError: Overload resolution failed. which sounds pretty bad. I'm going to try to make a simpler repro in a three.js example instead of MV.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I think this is just async issues with WebGPURenderer in MV.


const geometry = new BoxGeometry( 5, 5, 5 );
const mesh = new Mesh( geometry, blurMaterial );

const scene = new Scene();
scene.add( mesh );

const camera = new CubeCamera( 1, 10, this );

const width = cubeTex.source.data[0].width;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've run into this issue before in Three (and complained about it) - this feels super fragile, but how am I supposed to find the size of a cubeTexture? Some have image, some have source, and I'm pretty sure if you generate one with a render target it has none? Seems like we're missing an API for this. Can we please just have a .size?


envMap.value = cubeTex;
latitudinal.value = 1;
const blurParams1 = getBlurParams( sigmaRadians, width, MAX_SAMPLES );
weights.value = blurParams1.weights;
samples.value = blurParams1.samples;
dTheta.value = blurParams1.radiansPerPixel;

if ( sigmaRadians <= 0 ) {

camera.update( renderer, scene );

} else {

Check failure on line 139 in src/renderers/common/CubeRenderTarget.js

View workflow job for this annotation

GitHub Actions / Lint testing

A space is required after '['

Check failure on line 139 in src/renderers/common/CubeRenderTarget.js

View workflow job for this annotation

GitHub Actions / Lint testing

A space is required before ']'

const blurTarget = new CubeRenderTarget( Math.min( this.width, width ) );
camera.renderTarget = blurTarget;

camera.update( renderer, scene );

camera.renderTarget = this;
envMap.value = blurTarget.texture;
latitudinal.value = 0;
const blurParams2 = getBlurParams( sigmaRadians, blurTarget.width, MAX_SAMPLES );
weights.value = blurParams2.weights;
samples.value = blurParams2.samples;
dTheta.value = blurParams2.radiansPerPixel;

camera.update( renderer, scene );
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that this function is finally running I'm getting an exception that comes from here (oddly not the first camera.update call?): TypeError: Failed to execute 'writeBuffer' on 'GPUQueue': Overload resolution failed. No idea what to do with that.


blurTarget.dispose();

}

cubeTex.currentGenerateMipmaps = currentGenerateMipmaps;
geometry.dispose();
blurMaterial.dispose();

}

}

export default CubeRenderTarget;
Loading
Loading