How to load the dataset as event stream input

[wtyytw1]

Hi,

I am wondering whether I can just load the EVS dataset as the input of "edopt", rather than connect to a camera?
Further, I do not have ATIS and I only have DAVIS 346, Prophesee EVK4. How can I use these camera and their data as input of EDOPT?

Thanks a lot

[arrenglover]

Hi @wtyytw1 - thanks for reaching out and for your patience in the reply.

The current version of code works with any 640x480 resolution event camera provided it can be interfaced with our YARP middleware. We have a bridge between Prophesee cameras and YARP to run our cameras live, or using a dataset saved/converted to the YARP format data.logwhich you can playback in the YARP environment using yarpdataplayer. The live camera or playback method is transparent to the algorithm code.

If you need some instructions on how to convert your dataset to a data.log and using the yarpdataplayer let me know.

If you want to use a different resolution camera - it hopefully shouldn't crash, but there may be some parameter adjustment required. We also now have Prophesee EVK4 cameras that we could test the algorithm with.

If you want to use a inivation camera interfaced to the YARP environment we would need to write a bridge between the inivation drivers and the YARP environment. This is something you could possibly contribute to as we don't have any of the cameras to test. I would give you support in the development.

Let me know what direction you want to take and I can give more details.

[wtyytw1]

Thank you for your help! I have two additional questions.

I am trying to use the Prophesee EVK4 for pose tracking. Is there any open-source code available that can convert the ".raw" files recorded by the Prophesee software into the ".log" format?

Furthermore, I successfully used the Prophesee EVK4 with "atis-bridge" and "EDOPT." However, I am finding it challenging to track my 3D-printed car model. Due to the absence of a spray painting process, I'm unable to apply texture to the model as defined in "model_tex.png," as shown in the attached picture. Does EDOPT require the model to be printed with textures, or is there a way to modify the texture file so that EDOPT can work with my untextured, "bare" model?

Thank you very much for your assistance. I look forward to your response.

[arrenglover]

Hi @wtyytw1 !

The 3D printed car is great and I'm surprised to see it! However, it's probably a hard way to get EDOPT working for you. Yes the algorithm will use the textures on the car to help tracking and I can imagine it doesn't work if the algorithm expects texture and the object doesn't have it. A quick solution could be to use a black permanent marker to highlight contrast changes on the object model so the camera can see similar gradients?

you should be able to use meshlab https://www.meshlab.net/ to load the object model and remove the texture and save the object model. I think EDOPT won't work very well as there isn't as much information available and would suggest an alternative method

Instead, if you have any other textured object that you also have the model for (YCM dataset?) you can easily replace the loaded object file and use a custom object. If you need help with this option let me know.

Finally - yes we have code to convert datasets to our .log format. The code is available here:
https://github.com/event-driven-robotics/bimvee

and an example usage is here:
https://github.com/robotology/event-driven/blob/main/python_tools/ev2converter.py
see the last two lines which loads a particular format and then the export is IITyarp format. you would need to replace the import function with the correct one for your dataset. We should already have a converter for .raw

[wtyytw1]

Thank you for your support!
I purchased a physical car model that appears to be nearly identical to the one used in your dataset, although the color of my car model is more yellow. The pictures of it is shown below.

However, I am experiencing difficulties with accurately estimating and tracking the pose. Initially, I position the car to closely match the predefined object pose specified in the configuration file and then begin tracking by pressing "G". However, the tracking consistently fails, as demonstrated in the videos available in the Google Drive link: https://drive.google.com/drive/folders/1xt7Ozz6aZ576jpVopNoc8YG5461kR_7p?usp=drive_link

Here is the modified config file:

#config file for using the car with the live camera

object_path = "/EDOPT/objects/car/model.obj"
object_pose = ( 0, 10, 300, 0, 0, 0, 1 )
camera_pose = ( 0, 0, 0, 1, 0, 0, 0 )

eros_k 9
eros_d 0.5
proc_size 100
render_scaler 1.0

[CAMERA_CALIBRATION]

w 1280
h 720
fx 1331.4040420582908
fy 1331.4040420582908
cx 638.59857302891817
cy 371.33539144951339
k1 0
k2 0
p1 0
p2 0

[arrenglover]

Hi,

I don't have permissions to see the video. Please open them to public access.

The colour of the car shouldn't matter as long as there is contrast in the same places that match the object model you loaded.

My guess (I haven't seen videos) is that is the problem. For example, there are several differences that would produce different output on the cameras - the window border is very thick, the wheel hubs are much closer colour to the tires, and the car is missing the logo on top. unfortunately these differences could be enough to cause the expected observation (from the loaded model file) and the actual observation (from the camera) to not match. Can you scan in your car model with any tools? Mobile phone have some apps to do it, some better than others. Or any other way you can get a precise match between model and real object.

Alternatively it could be the change in resolution. I'll see if I can't check EDOPT working with the EVK4 cameras this week on my side to rule that out.

[wtyytw1]

Thank you for your support.

I open the access of the video:https://drive.google.com/drive/folders/1xt7Ozz6aZ576jpVopNoc8YG5461kR_7p?usp=sharing

n the video demonstration, the pose appears to 'fly away' right after tracking starts. Could the slight discrepancy between the features of the model file and the actual car be responsible for such a significant tracking error? I will conduct a scan of the model later when I return to the office.

[arrenglover]

edopt-ekv4.mp4

seems like the higher resolution works simply by adjusting the camera parameters in the config file as you have done. Therefore I think the biggest issue is the difference in the model.

[arrenglover]

Hi @wtyytw1 ,

I managed to see the videos.

Yes I think it's possible that some differences in the model and object, even as small as yours could cause problems. Especially if there are edges in a "negative region" where the tracker expects there not to be any edges.

I currently re-ran the code using the dockerfile from scratch and can confirm it is working for me with my car and the 640x480 resolution cameras. I'll see if I can update to use the 1280x720 cameras you are using the rule that out as a possibility.

[wtyytw1]

Thank you for your help! May I ask where you bought the model?

[arrenglover]

sorry @wtyytw1 this object was just one I found in a pile of random objects in the lab. I generated the model by using a 3D object scanner (e.g. https://all3dp.com/1/best-3d-scanner-diy-handheld-app-software/). I processed the model a little bit to make sure there where no errors when loading in EDOPT and reducing the number of vertices for fast projection - using meshlab. I think, instead of trying to get the precise object that matches my model, it is easier to get the object + model together (ie.. YCB dataset objects) or try scanning. I think mobile phone scanning applications might have improved in the last 5 years since I last tried - let me know if you have any success using it. Otherwise buying a dedicated hardware for scanning gives a quite precise model with texture. Then just replace the model file for you object.

[wtyytw1]

Hi arrenglover,
I tried to scan the model and get the texture and 3D .obj files, the tracking is more stable now (which means the result would not just fly away). However, the program still encounters difficulties in accurately tracking the objects.

I upload the videos, config file and 3D object files in google drive: https://drive.google.com/drive/folders/1asMsSIWpueILBYCNdmPrHMmbH8m7O5l9?usp=sharing

What might be the cause of this issue?

Thank you in advance.

[arrenglover]

Hi @wtyytw1 - I would first check your camera set-up. It looks very noisy in the video - i.e. the camera is constantly triggering events from the low textured background, and from the non-texture regions of your object.

The camera has several options to change - The physical lens focus, the physical lens aperture, and the sensitivity of the camera biases. These three will all affect the resulting events with some inter-dependency - which makes it hard to give a precise method of tuning.

The sensitivity of the camera bias should be able to be set when running the atis-bridge-sdk --s 50. Default values by prophesee is around 70 and is quite sensitive. I typically end up using something like 50. You should notice a change with this value.

I normally do something like - 1. set a high sensitivity (--s 70) 2. reduce the aperture such that is it somewhat near closed 3. make sure the object is in focus (it should be easier with the closed aperture), 4. tune down the sensitivity (--s X) until you no longer get events all the time all over the image but still can pick up the major edges of the object.

you can just visualise the events using the vFramer app built alongside EDOPT. Try vFramer --help to get options for visualisation.