block solver fails for BaseFixedSizeEdge with 3 vertices

[atuldivekar]

I implemented a BaseTernaryEdge derived from BaseFixedSizeEdge with 3 vertices instead of 2.
(Similar to BaseBinaryEdge implementation, just 3 vertices)

I tried to solve it with block_solver.hpp, get an error in buildStructure():

schurMatrixLookup->addBlock(i1, i2);
assert(c < (int)_blockCols.size() &&
"accessing column which is not available");

The error is because the 2nd and 3rd kinds of vertices are all treated as landmarks.
but the loop that calls schurMatrixLookup->addBlock(i1, i2) does not recognize this.

For each marginalized v
consider attached vertex pairs v1,v2 (could be on same or different edges attached to v)
with binary edge, v1,v2 will be non-marginalized i.e. hessianIndex will be < numPoses
with ternary edge, v1,v2 could be marginalized i.e. hessianIndex could be >= numPoses
-- this causes the error..

Or is the block_solver not the correct way to use the Ternary Edge?

I see the example sclam_odom_laser.cpp which uses BaseFixedSizeEdge but I am not sure how that optimizer works.

[RainerKuemmerle]

Did you try with a variable sized solver instead?
Which vertices of your edge are landmarks?

[RainerKuemmerle]

Duplicates #597