Shader overlay

[Tom-TBT]

Hi Thomas,
I'm facing an issue. Basically, I would like to print a Box object on top of an object with a shader (in my case SquareWaveGrating).

When I put a normal box object on top, it is always covered by the shader (In the list of object to present, the obj with the shader is the first).
So I tried the overlay with an object having a shader too, but the result is strange. After I compiled the stimulus, two consecutive play won't give always the same output. Sometimes the shader is overlayed, sometimes no.

Do you have an idea where it could come from? Like taking the shader objects from a dictionary and not a list.

Thanks,
Tom

[teuler]

Hi Tom,
I have to look into that - maybe I can manage tomorrow. Your "dict" idea sounds plausible but I seem to remember that I had some object z-odering scheme in place; pyglet supports that. Let me check.
I the meantime, could you please send my the stimulus in question plus any changed shader files for testing?
Best
Thomas

[teuler]

Btw. Are you using lightcrafters (DLP machines) for stimulus display?

[Tom-TBT]

I sent you the files directly.

And yes, we are using DLPs. All the same DLP3010.
http://www.ti.com/tool/DLP3010EVM-LC

I believe it is not compatible with the QDSpy interface. Or at least when I looked briefly at it more than a year ago.

Bests,
Tom

[teuler]

We'll have a open source stimulator / lightcrafter meeting  on Tuesday May 28, 2019 in Innsbruck - if you are interested, I can send you details. Best Thomas Am 06.12.2018 um 17:02 schrieb Tom Boissonnet:

I sent you the files directly. And yes, we are using DLPs. All the same DLP3010. http://www.ti.com/tool/DLP3010EVM-LC I believe it is not compatible with the QDSpy interface. Or at least when I looked briefly at it more than a year ago. Bests, Tom — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#28 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AK9GeqcHJxUx--xeUe_Azrh-fzuvhEnBks5u2T-agaJpZM4ZGhaX>.

-- Dr. Thomas Euler Professor for Ophthalmic Research CIN - Centre for Integrative Neuroscience Institute for Ophthalmic Research University of Tübingen Otfried-Müller-Str. 25 D-72076 Tübingen +49 7071 29-85028 (office) +49 7071 29-25011 (fax) thomas.euler@cin.uni-tuebingen.de www.eulerlab.org --

[teuler]

Hi Tom,

I looked at the rendering code and indeed there was a dict used where a list should have been - at least I did not understand, why I used a dict there. However, this did not solve your problem. Another think I found was that I used a normal batch object for drawing, where an ordered batch would also work (one that keeps the sequence of the objects to draw w/o optimizing). I think there is the key to solving the problem. At least the red box now reliably disappears behind the grating.

For time reasons, I could not test it with an own example now and I don't quite understand the structure of your stimuli. Therefore, please check if by changing the order of adding the grating and the red box object, if the problem can be solved. If not, I'll do some more digging.

The changes are in the "experimental" branch.

Best
Thomas

[Tom-TBT]

Thank you Thomas,
I will check this soon. And also create a simpler stimulus with gratings for testing purpose.

And could you send me the details about the meeting in Innsbruck, I'm interested!

Bests,
Tom

[teuler]

Hi Tom, Can you please send me your real-world email address? Then I put you on the mailing list for the meeting. The date is already fixed; it’ll be Tuesday May 28. Best Thomas

…

Am 10.12.2018 um 09:07 schrieb Tom Boissonnet ***@***.***>: Thank you Thomas, I will check this soon. And also create a simpler stimulus with gratings for testing purpose. And could you send me the details about the meeting in Innsbruck, I'm interested! Bests, Tom — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[Tom-TBT]

Hi Thomas,
tom.boissonnet@embl.it

Tom

[Tom-TBT]

Hi Thomas,

I tried with simplier stim, and I started to look at renderer_opengl. I saw different things:

First IVShObjGr is used as a dict in some part of the code, like in the functions
add_shader_handle() , set_shader_time() and set_shader_parameters()

Then we would also have to change it in delete_shader_handles() line 592, but I don't think that's the problem anyway.

I had not time to look in detail, also because I don't know piglet much. But it seems to me that it has to do with the order set at the creation of ShaderBindGroup objects. Order is always set to 0, and the order of the parent is set to _iObj. Is it the same as the id of the object we create?

Again I don't know how it works but this seems strange like this. I will continue exploring later this week.

And about the meeting, thanks again, however I didn't saw at first but I have a retreat with my unit these exact days.... I really wanted to come, that's unfortunate.

Tom

[teuler]

Hi Tom,
thanks for your message. If you send me a simpler stimulus that reproduces your problem (one that does not use your general library), I can have a closer look. I agree that it has to do something with the ordering in groups and I believe that it can be solved but for this I need to "follow" the creation of the objects and their sequence through the code, hence a simpler stimulus would help.
Best
Thomas

P.S.: Too bad that you cannot come to the LCr meeting.

[Tom-TBT]

Hi Thomas,
here is a stimulus for testing purpose. It displays different gratings objects that are supposed to overlay.

import QDS
p = {"_sName": "Test shader overlap",
     "_sDescr": "A test for shader overlapping"}

QDS.Initialize(p["_sName"],p["_sDescr"])

QDS.DefObj_EllipseEx(1, 200,  200, _enShader=1)
QDS.DefObj_BoxEx(    2, 1280, 720, _enShader=1)
QDS.DefObj_EllipseEx(3, 200,  200, _enShader=1)

QDS.DefObj_EllipseEx(11, 200, 200)
QDS.DefObj_BoxEx(    12, 1280, 720)
QDS.DefObj_EllipseEx(13, 200, 200)

QDS.DefShader(4, "SQUARE_WAVE_GRATING")
QDS.DefShader(5, "SQUARE_WAVE_GRATING")
QDS.DefShader(6, "SQUARE_WAVE_GRATING")
QDS.SetShaderParams(4, [100, 1, (0,0,0,255),(255,0,0,255)])
QDS.SetShaderParams(5, [300, 1, (0,0,0,255),(0,255,0,255)])
QDS.SetShaderParams(6, [150, 1, (0,0,0,255),(0,0,255,255)])

QDS.SetObjShader([1,2,3], [4,5,6])
QDS.SetObjColorEx([11,12,13], [(255,0,0),(0,128,0),(0,0,255)])

QDS.LogUserParameters(p)
QDS.StartScript()

#############################################
time_intro = 1
#Let's first render all individual object
#QDS.Scene_RenderEx(time_intro, [11], [(-400,100)],[(1,1)], [0]) #Scene 0.0

QDS.Scene_RenderEx(time_intro, [1] , [(-400,-100)], [(1,1)], [0]) #Scene 0.1
QDS.Scene_RenderEx(time_intro, [2] , [(0,0)]      , [(1,1)], [0]) #Scene 0.2
QDS.Scene_RenderEx(time_intro, [3] , [(400,  100)], [(1,1)], [0]) #Scene 0.3

QDS.Scene_RenderEx(time_intro, [11,3], [(-400,100),(400,  100)],[(1,1),(1,1)], [0,0]) #Scene 0.4
## We need scene 4, without presenting a shaded object with a non-shaded object before presenting only non-shaded objects, 
#  we get an error:  KeyError: CommonShaderParentGroup(3)
#TRY it by commenting the scene 0.4
#TRY Uncommenting scene 0.0 also solve the problem, but see how the object remains on the screen. until other non-shaded object are presented

QDS.Scene_RenderEx(time_intro, [13], [(400,-100)],[(1,1)], [0]) #Scene 0.5
QDS.Scene_RenderEx(time_intro, [12], [(0,0)],[(1,1)], [0])      #Scene 0.6
QDS.Scene_RenderEx(time_intro, [11], [(-400,100)],[(1,1)], [0]) #Scene 0.7

#############################################

#Then let's try different combinaison.
time_sce_1 = 1
time_sce_2 = 1
time_sce_3 = 1
#(I'm on the master branch, not experimental)
#Scene 1.x render the object in random order. Bug: The order the index given doesn't affect the overlap
# Also the index of an object doesn't seem to play a role in the order.
if True:
    QDS.Scene_RenderEx(time_sce_1, [1,2] , [(-400,-100),(0,0)], [(1,1),(1,1)], [0,0],1) #Scene 1.1   Should not show the small obj
    QDS.Scene_RenderEx(time_sce_1, [2,1] , [(0,0),(400,100)]  , [(1,1),(1,1)], [0,0],0) #Scene 1.2
    QDS.Scene_RenderEx(time_sce_1, [3,2] , [(-400,-100),(0,0)], [(1,1),(1,1)], [0,0],1) #Scene 1.3   Should not show the small obj
    QDS.Scene_RenderEx(time_sce_1, [2,3] , [(0,0),(400,100)]  , [(1,1),(1,1)], [0,0],0) #Scene 1.4

#Scene 2.x   non-shaded background with shader. Bug: shader is always rendered on top of the non-shaded object
if True:
    QDS.Scene_RenderEx(time_sce_2, [1,12] , [(-400,-100),(0,0)], [(1,1),(1,1)], [0,0],1) #Scene 2.1   Should not show the small obj
    QDS.Scene_RenderEx(time_sce_2, [12,1] , [(0,0),(400,100)]  , [(1,1),(1,1)], [0,0],0) #Scene 2.2
#Scene 3.x   non-shaded background with two shader. Bug: shader is always rendered on top of the non-shaded object
if True:
    QDS.Scene_RenderEx(time_sce_3, [12,1,3] , [(0,0),(-400,-100),(400,100)]  , [(1,1),(1,1),(1,1)], [0,0,0],1) #Scene 3.1
    QDS.Scene_RenderEx(time_sce_3, [1,12,3] , [(-400,-100),(0,0),(400,100)]  , [(1,1),(1,1),(1,1)], [0,0,0],0) #Scene 3.2 Should not show the first small obj
#############################################    
QDS.EndScript()

I'm also working on it. So far here I am. In QDSpy_core_presenter.py line 348, I'm using the enumerate function index and pass it to the self.Batch.add_shader_handle that would then create the ShaderBindGroup with the order of this enumerate.

for iObj, ObjID in enumerate(ObjIDs):
            if ObjID < 0:
              continue
            iObjList  = self.Stim.ObjDict[ObjID]
            iSh       = self.Stim.ObjList[iObjList][stm.SO_field_shProgIndex]
            if iSh >= 0:
              # Create Group object referencing to requested shader and set
              # shader parameters (uniforms)
              #
              shPar   = self.Stim.ShList[iSh][stm.SH_field_Params]
              shType  = self.Stim.ShList[iSh][stm.SH_field_shaderType]
              #self.Batch.add_shader_handle(ObjID, self.ShProgList[iSh], shType)
              self.Batch.add_shader_handle(ObjID, self.ShProgList[iSh], shType, iObj)
              x       = ObjPosXY[iObj][0] +self.Stage.dxScr
              y       = ObjPosXY[iObj][1] +self.Stage.dyScr
              a_rad   = (ObjRot[iObj]+90.0)*np.pi/180.0
              self.Batch.set_shader_time(ObjID, self.tFrRel_s)
              self.Batch.set_shader_parameters(ObjID, [x,y], a_rad, shPar)
              print(self.Batch.IVShObjGr)
              
            else:
              # No shader
              #
              self.Batch.add_shader_handle(ObjID, iObj)

It seems like a good direction, but then I have other problems with the order of the parent objects, and the shaded objects that have the wrong shader.

Let me know if you progress with it. In the meantime, I wish you good holidays.

Tom

[teuler]

Hi Tom,
I have your demo running and looking at it more closely.
Did you make any progress in the meantime?
Best
Thomas

[Tom-TBT]

Hi Thomas,
not really, I don't understand pyglet very well. I'd start by changing ShaderBingGroup to an OrderedGroup, but then I have to give an order to the object created, and the display gets messed up whatever order I give.

What do you know about the order of the display? I see that for a normal object, all object have the order set to 0. How come that they are displayed in the order we create them?

So I tried to change values around to see what happens, but I haven't progressed much more than that, sorry.

Best,
Tom

[teuler]

Hi Tom,
o.k. - thanks for the update. I just wanted to know if there is anything new that I can work with.
I think your scripts reveal at least two issues:

1. that there is some error when adding a normal object after only showing shader objects. This is a general problem that I probably should try solve first because it may prevent finding a clear order solution for the shaders. I guess this problem is related to having the normal and shader objects in different lists.
2. the order problem you noticed.
   I hope I'll find some time this week to look at problem 1) first.
   Best
   Thomas