Render without background

[LewisBridgeman]

I am trying to figure out how to return an RGBA image without combining with the solid background colour inside the rasterizer.
I've adjusted the out_color variable to have an extra channel, and in the forward pass (forward.cu) it is simple enough to make the change:

if (inside)
{
  final_T[pix_id] = T;
  n_contrib[pix_id] = last_contributor;
  for (int ch = 0; ch < CHANNELS; ch++)
	  out_color[ch * H * W + pix_id] = C[ch]; // + T * bg_color[ch];
  out_color[CHANNELS * H * W + pix_id] = 1-T;
}

But I am unsure about how to make the change in the backward pass. Any ideas how this might be implemented?

[yuedajiong]

too may lines need modification.

1. load RGBA instead of RGB, modify the data preprocessing( norm + fore + back + ...)
2. change CHANNELS to 4, default is 3. (config.h)
   change obtain(chunk, geom.rgba, P * 4, 128); (rasterizer_impl.cu)
3. append code in preprocessCUDA() and renderCUDA() to compute 'A' in forward.cu: from GS official opacity? another parameter like 'feature'? (I am trying tooooooo)
4. modify backward (studying :-) )
5. modify loss

---

I have another idea:
the official implementation, there is a parameter named 'background', from pytorch to cuda.
then we can let it as 'mask', we do segmentation on 2d image, and pass the mask to render internal, and just render with mask. the method can focus on foreground object, and reduce computation.
maybe wrong. :-(

[LewisBridgeman]

I think I may have figured out a rather hacky way to do it, but here goes:

rasterize_points.cu
line 68:

torch::Tensor out_color = torch::full({NUM_CHANNELS + 1, H, W}, 0.0, float_opts);

forward.cu
line 303:

float C[CHANNELS + 1] = { 0 };

line 354:

for (int ch = 0; ch < CHANNELS; ch++)
    C[ch] += features[collected_id[j] * CHANNELS + ch] * alpha * T;
C[CHANNELS] += alpha * T;

line 372:

for (int ch = 0; ch < CHANNELS + 1; ch++)
    out_color[ch * H * W + pix_id] = C[ch];

backward.cu
line 449:

float accum_alpha = 0;
	float accum_rec[C] = { 0 };
	float dL_dpixel[C+1];
	if (inside)
		for (int i = 0; i < C+1 + 1; i++)
			dL_dpixel[i] = dL_dpixels[i * H * W + pix_id];

	float last_alpha = 0;
	float last_alpha_color = 0;
	float last_color[C] = { 0 };

line 527:

accum_alpha = last_alpha * last_alpha_color + (1.f - last_alpha) * accum_alpha;
last_alpha_color = 1;
dL_dalpha += (1 - accum_alpha) * dL_dpixel[C];
dL_dalpha *= T;
// Update last alpha (to be used in the next iteration)
last_alpha = alpha;

line 536:

//float bg_dot_dpixel = 0;
// for (int i = 0; i < C; i++)
// 	bg_dot_dpixel += bg_color[i] * dL_dpixel[i];
// dL_dalpha += (-T_final / (1.f - alpha)) * bg_dot_dpixel;
// dL_dalpha += alpha * dL_dpixel[C + 1];

In essence, I am treating the 4th channel like another colour channel. Except the "feature" colour from the gaussian is constant (1.f). Allows me to copy the code used to compute the existing colour channels.

[jkulhanek]

Can't you just use final_T?
https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/59f5f77e3ddbac3ed9db93ec2cfe99ed6c5d121d/cuda_rasterizer/forward.cu#L369

[jkulhanek]

It's exported inside imgBuffer, I will do some testing to figure out how to read it.
https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/59f5f77e3ddbac3ed9db93ec2cfe99ed6c5d121d/diff_gaussian_rasterization/__init__.py#L97

[jkulhanek]

Could be possibly extracted as:

alignment = 128
offset = (alignment - imgBuffer.data_ptr()) % alignment
total_size = raster_settings.image_height * raster_settings.image_width * 4
accumulation = imgBuffer[offset: offset + total_size].view(torch.float32).clone().view((raster_settings.image_height, raster_settings.image_width))

In diff_gaussian_resterization/init.py at line 99. I still have to verify if this works.

[jkulhanek]

Ok, its working well. Here is the fork if you want it:
graphdeco-inria/diff-gaussian-rasterization#35

[ladzin]

I tried this fork and it worked great. But I made a mistake: I assumed that gradients with respect to "accumulation" would flow correctly, but that was not the case since backward wasn't implemented -- so I got, silently, the wrong gradients. Would it be helpful to add an error/warning or implement the backward pass?

[pknmax]

how it is rendering without background? please explain.

[jkulhanek]

If you set the background colour to black then you get rgb image and accumulation - basically rgba image

[HochCC]

Ok, its working well. Here is the fork if you want it: graphdeco-inria/diff-gaussian-rasterization#35

@jkulhanek @pknmax
hi, I still have doubts about how to use it in the training process, could you kindly provide a example?

[yoyo-000]

Is there a better solution to this problem now? I'm working on something similar....