Using EDOPT with Inivation event camera

[jorou125]

If you want to use a inivation camera interfaced to the YARP environment we would need to write a bridge between the inivation drivers and the YARP environment. This is something you could possibly contribute to as we don't have any of the cameras to test. I would give you support in the development.

Let me know what direction you want to take and I can give more details.

Originally posted by @arrenglover in #13

Hi @arrenglover ,

In this previous issue you suggested providing support to interface Inivation cameras to EDOPT. I am interested in working on this. Is the support still available and how should I approach this. I am new to YARP but I am quite interested in learning how to create a bridge between the Inivation sensor and YARP. Thank you in advance for your time.

[arrenglover]

Hi @jorou125 ,

So if you look at the code here: https://github.com/robotology/event-driven/blob/main/cpp_tools/atis3-bridge/atis-bridge-sdk.cpp you can see that this piece of code includes both the prophesee driver libraries, so we can access the data through their API, and YARP so we can push data onto the network.

If you create a similar interface, you will need to find the inivation drivers and API that allow you to grab data from your camera from a custom written application. You will need to be able to compile the API and YARP together (follow the CMake files I have for the ATIS bridge). Then copy the way that I send data to YARP after reading the data from the camera.

Once you have the API for your camera, and begin to compile them I could help you with the YARP side of things. Good luck!

[jorou125]

Hi @arrenglover ,

I believe I'm on the right track, but I have encountered an error that I believe comes from difficulties I have with YARP. I am able to compile both the API and YARP together, and I am able to read events (X packets and X k events sent per second) when I start the custom bridge. By following the documentation for EDOPT, I am able to open the visualization window with the appropriate model. While I am able to read events from the camera, the moment I connect the ports of the camera to the EDOPT the following error occurs:

OpenCV (4.2.0) ../modules/core/src/matrix.cpp:465: error: (-215: Assertion failed) 0 <= roi.x && 0 <= roi.width && roi.x + roi.width <= m.cols && 0 <= roi.y && 0 <= roi.height && roi.y + roi.height <= m.rows

I searched for the cause of this error and have yet to find it. I made sure that all the events had valid coordinates. I am using all the default options for EDOPT. During the development of atis-bridge-sdk, have you encountered this error? If you have any suggestions, please share. Thank you in advance!

EDIT: I noticed that while all the events sent to YARP from my custom bridge are valid, the ones read by 'erosUpdate' from the class 'EROSfromYARP' in 'erosdirect.h' are not. The events sent to YARP from my custom bridge are valid. The events read from YARP are alternatively not. They are the same input events separated by "virtual" invalid events from unknown origin. For a set of events, every real event is separated by a constant abnormal event. The error seems to originate from communication with YARP.

[arrenglover]

Hi @jorou125 - sorry for the long delay in response this time.

This is an opencv issue that indicating you are accessing the pixels outside the defined image size. It could be:

1. the camera is producing strange pixels values e.g. negative or large values. You said you checked this so we assume this isn't it.
2. the image is not set to be the correct size -> make sure you have the correct camera parameters in the intrinsic parameters in the config file. you'll need to change them to your camera resolution and measured parameters for it to work.

fyi we have a camera calibration for event cameras here: https://github.com/robotology/event-driven/tree/main/cpp_tools/calibration

you can also visualise the output of your camera using "vFramer" which is here: https://github.com/robotology/event-driven/tree/main/cpp_tools/vFramer

both should have "help functions" to understand the parameters i.e. vFramer --help

[jorou125]

Hi @arrenglover, thank you for the feedback.

I was able to determine the source of the invalid events and found a "workaround" but have yet fixed the problem. I noticed that these invalid events seemed tied to the timestamp of the recorded events. When converting the Inivation event packets to the event-driven packets, I commented out the part where I transfer the timestamp of each individual events. For some reason, this "fixed" my problem and no longer throws the OpenCV error. While looking at the events, I noticed something strange. Every invalid event was still there, but were all (x=0,y=0,p=0,ts=0). While this no longer triggers an error, it does double the amount of events processed by EDOPT as every two event is not a real one. The work-around I found was to simply skip the event if it was (x=0,y=0,p=0,ts=0).

The code now works: I am able to capture events with the Inivation camera and use them with EDOPT, although I have yet found a proper solution for the issue described above. While the Inivation SDK and YARP seem to be working rather well as I have no lag in event output, the tracking from EDOPT is particularly difficult and fails. Could it be linked to the difference in parameters we've discussed in a previous issue?

[arrenglover]

Hi @jorou125 - nice work finding the problem.

The event-driven library has the option to turn-off the use of individual event timestamps. Each packet still has a timestamp making the temporal resolution still sub millisecond, but not microsecond (each event has temporal jitter on the order of milliseconds in any case). For EDOPT the order of events is sufficient for the processing. To turn on and off individual event timestamps you can do so in the cmake. e.g. try cd build then ccmake .. and you should be able to find the option USE_TIMESTAMPS. For EDOPT is should be off (but shouldn't break if it is on).

However, it is strange that you are getting "fake" events on EDOPT side. I would check your iniviation-bridge code to see if somehow you are still copying data into the data packet when you "skip" the timestamp. In theory you should only copy the position/polarity/camera #.

Let me know if you have the USE_TIMESTAMPS cmake flag on - or if you have somehow otherwise tried to modify how timestamps are handled by event-driven.

Could you contribute your bridge to the event-driven repository and open a pull request?

[jorou125]

Hey @arrenglover ,

Was able to correct the fake event problem. The USE_TIMESTAMPS cmake flag was off, but in early testing I had defined ENABLE_TS and for some reason that was the cause. As soon as I commented out that part of the code, the fake events disappeared.

The bridge only works for the Inivation DVXplorer Micro camera as it's the one I have, but could easily be adapted to other Inivation sensors if needed. I'll sort things out, clean out the code and add a few worthwhile control options and will contribute my bridge to the event-driven repository and open a pull request, it will be my pleasure! I should be able to finish all of this in the following week.

If you could take the time to verify the parameters you used for the article, I would love to try to reproduce your results!

Thank you for your support.

[arrenglover]

Hi @jorou125 ,

Great that you found the problem. ENABLE_TS can be turned on and off globally in the project as a cmake variable. Note it is always possible to read the packet timestamp (sub millisecond accuracy).

Thanks so much for your contribution! I've seen the pull requests and will check them - it's just been super busy in this period.

About the re-production of the results in the article. We have found that the hardware is greatly affecting if we achieve the same results. The code is set up to run always "real-time" so a faster update rate is going to produce better results with less failure. We checked again on our laptop and found that we could reproduce the results of the paper using the parameters in the repository main branch - however sometimes we find, especially different graphics cards, will greatly change the rate at which the renders can be produced and the algorithm update rate.

Maybe we did experiments with dp = 2 at this line of code:

EDOPT/code/main.cpp

Line 120 in 803c142

double dp = 1;//+rescale_size / 100;

I'm not sure why this is hard-coded. sorry :|

our computer specs:

Dell Precision 3591
Intel Core 7 165Hx22
NVIDIA RTX 2000