enhancements

[arrenglover]

Some ToDo's for EDOPT @lunagava :

• update to SCARF
• update to EVK-4 as standard and optimise any code due to this change (memory copy from GPU?)
• streamline code for readability - removing depreciated functions etc.
• investigate cause of tracking failures
• flexible DP enhancement
• pre-compute projections to remove GPU reliance
• puck/circular (uts space) object (e.g. possible no GPU and mesh)
• template tollerance

[lunagava]

We listed three main sources of failure @arrenglover:

1. $\Delta X$ > $\Delta P$ when the object moves fast between updates.
   This could be due to the computation taking long or the object high-speed.
2. A blip in compute time (an operation suddenly takes longer than expected) or in observation (noise which is more common with real-camera but also shades are visible in the synthetic datasets)
3. Multi-axes update is currently done. The alternative is change one axis at update, but moving more axes takes more update. We want to investigate which axis update makes more easily to fail (if there is a trend).

To-do:

• plot time of failure for different update times with a multi-dof object motion dataset
• plot time of failure (for update times) for different $\Delta P$ (e.g. $\Delta P=1$, $\Delta P=2$, $\Delta P=3$)
• in both plot put as a reference dashed line the typical update time for the algorithm running in real-time
• plot tracker output $X$ vs. ground truth $X_{gt}$ for all update times

In the case faster updates do not fail, the estimated optical flow (by zicci) would help in choosing the right $\Delta P$

[lunagava]

The sequence shows a mustard bottle translating horizontally at slow speed (10 cm/s). The EDOPT paper set up this speed to limit the failure cases in real-time.

I run the sequence for different update times to investigate the tracker behaviour. I used the same parameters and algorithm implementation of the EDOPT paper replacing EROS with SCARF.

I plot the tracker 6-dof state vs. the ground truth. I had to convert the quaternions into Euler angles. The vertical lines indicate the failure time, computed as the error (computed as absolute difference between each state component and the ground truth) greater than a threshold for a number of successive iterations. Specifically failure is given if x,y,z are greater than 4 cm for 20 iterations and if roll,pitch and yaw are greater than 20 degrees for 20 iterations.

1 ms

2 ms

3ms

4ms

5ms

6ms

10ms

I grouped the plots in an histogram showing time of failure for the different update times.

The failures seemed to correspond to a decrease in z which scaled up the template.
The object template is warped after applying the Mexican hat. The scale up leads to a bigger positive and negative regions which are more likely to capture/overlay with the observation, giving a bigger score.

This study required a deeper analysis on observation and template superposition.
It opened some questions:

• Is it worth to do the Mexican hat on the warped templates despite the greater computational cost?
• Is the dp not precise enough, which determines the template to diverge?
• Is SCARF too noisy and/or has weak edges?

[lunagava]

Afterwards, I realised that dp2 boolean variable was set to true. When is true, it means that both the templates with dp=1 and dp=2x1 are compared, which means 13*2 comparisons. This does not help to verify which dp works better.

Moreover, we noticed in the template image some holes. We found a bug in Sobel filtering of the template. We were selecting only the diagonal components, so we corrected for both x and y.

cv::Sobel(canny_img, f1, CV_32F, 1, 0);
cv::Sobel(canny_img, f2, CV_32F, 0, 1);
f = cv::abs(f1)+cv::abs(f2);

The correct Sobel resulted in no failure for 3 ms, 4 ms and 5 ms, which before was failing.
3ms

4ms

5ms

[lunagava]

dp2 = false allows to set dp = 1 or dp = 2.
I selected the failure cases for dp = 1, namely 5 ms, 6 ms and 10 ms and I rerun for dp = 2.

[lunagava]

We generated other two sequences for the mustard bottle with high-speed motion:

4-dof motion: x,y,z,roll

6-dof motion: x,y,z,roll,pitch,yaw

I report the tests on 6-dof motion dataset because it is more challenging and realistic.

1ms

2ms

Changing the Mexican blur parameter, 2ms and 3ms work.

\\int blur = proc_size / 20;
int blur = proc_size / 8;

2ms

3ms

4ms

[lunagava]

We discussed other points:

• Look that dp and pixel shift match
• Play with different blurs on template generation, looking at pixel distance for example
• Check if there is a difference between affine warps and real projection
• Tuning dp for different axes
• Observability difference: pitch and zoom can look the same for different dps

[lunagava]

• understand theoretically if update time = 2 ms skips the search space, while 1 ms falls inside
• estimate object speed to move the search space but still keeps dp = 1/2 pixels
• plot dp + mexican hat/gaussian curve + edge's width after sobel. before vs. after mexican template.