Structural tangents are cool too, and SArrays can probably be primitives

[willtebbutt]

Natural tangents are helpful because they're sometimes more intelligible to humans than structural tangents in a number of situations, and can play nicely with generic linear operations written in rrules in a number of situations, making it more straightforward to write generic rrules. However, this is not always the case, and they can cause complications.

The purpose of this issue is to document situations in which structural tangents are either preferable or a necessity, and the complications that can occur when structural and natural tangents interact. Some of it is obvious, and I'm intentionally not proposing to do anything, I just needed to get this stuff out of my head so that I can focus on other work for the next couple of weeks without this bothering me.

Maybe we should link this in the docs that @oxinabox recently merged?

In the following, please assume that the primal is an AbstractArray, the structural tangent a Tangent, and the natural tangent another AbstractArray.

Interactions between structurals and naturals

Suppose we have code in which a hand-written pullback produces a natural tangent for a primal, and an automatically generated pullback produces a structural tangent for the same primal (say, because it hit getfield). At some point, these will need to be added together. This means that code must exist to convert one to the other, and it must be hit. To my knowledge, we don't really deal with this at the minute.

Sometimes structurals are more natural

Clarity is in the eye of the beholder, but to my eye it's not uncommon for the structural tangent to be much more straightforwardly interpretable than the natural tangent. Moreover, it's always obvious what the structural tangent is, whereas one has to think (sometimes for an extended period of time, and consult with others) to determine an appropriate natural tangent. This problem is compounded by our current lack of a rigorous definition for the natural tangent.

One example of this is Fill arrays. If we think about a Fill as a struct in the context of AD, it's incredibly clear what's going on and what its structural tangent represents. Conversely, if we think about it as an array, its natural tangent requires a good deal of thought. Again, the lack of a proper definition of a natural tangent is possibly the culprit.

Another good example is the WoodburyPDMat. I haven't the foggiest idea what an appropriate natural tangent would be, nor do I particularly want to. The structural doesn't suffer this problem. In this sense, it's much more natural to think about the structural tangent. Later in this issue, I'm going to use this as an example of a situation in which using the structural tangent is necessary, and the natural is undefined. If anyone wants to argue that I must derive a natural tangent for this matrix, then we will need to have a separate conversation.

Sometimes the natural is simply redundant

On the WoodburyPDMat example above, we intentionally only implement a few high-level linear algebra operations. There's simply never any need for a natural tangent because all of the code has been written in an AD-friendly manner (non-mutating, small number of differentiable high-level operations).

Other examples are ColVecs and RowVecs in the JuliaGPs ecosystem. They're thin wrappers around an AbstractMatrix which are really only designed to make its interpretation in a particular context clear. While getindex is defined, it's considered a bug if it's hit inside AD. Instead, the use of getfield is central to ColVecs and RowVecs usage. Consequently, AD ought always to be able to derive pullbacks automatically in practice -- certainly we don't want to have to write any rules, or define ProjectTo.

Structural tangents are sometimes a necessity

Symmetric matrices are a good example of something that can use a natural differential a decent chunk of the time, but sometimes simply cannot.

If a Symmetric wraps is a primitive, then we'll often want to use the natural differential. For example, this rule for svd. This is fine.

In other situations, it's easier to think about rules producing structural tangents. For example, Iain Murray's derivation of the rrule for the Cholesky factorisation uses only the upper triangle of a Symmetric, so it's easier to think in terms of the structural when writing that rrule.

A more extreme example is if one were to put a WoodburyPDMat inside a Symmetric and do AD. The tangent of a WoodburyPDMat will generally be a Tangent, which cannot itself be stored inside of a Symmetric. Therefore, the tangent of a Symmetric{<:Number, <:WoodburyPDMat} must be a Tangent.

A condition under which a given AbstractArray can be treated as a primitive

By calling a type T primitive, I mean that the tangents of T are always of type T themselves.

Assuming that we define the pullback of getfield on a given struct to return a Tangent, the above yields the following condition for the possibility of treating a particular AbstractArray (which is a struct) as a primitive:

Each field must be able to take the type of any of its tangents.

Interestingly, this seems to preclude treating a number of very common AbstractArrays as primitives:

• Symmetric (wraps an AbstractMatrix, whose tangent may be a Tangent)
• Diagonal (for the same reason)
• Transpose / Adjoint (for the same reason)
• SArray (wraps a Tuple, whose tangent is always a Tangent)

This is, of course, not to say that natural tangents ought not to be used for these types, it is simply to say that it's not possible to preclude the need for the use of a structural tangent some of the time.

Wait, we can't treat SArray as a primitive?

The above claim was made under the assumption that we make the pullback for the rrule for getfield return a Tangent. This is crucial. As @oxinabox pointed out the other day, we don't always have to define getfield this way.

If for an SArray we make said pullback return another SArray, the problem disappears. Indeed, in this case, we can safely treat SArrays as primitives and never have to worry about the structural derivative. For example, the rrule for getfield might be something like

function rrule(::typeof(getfield), X::SArray, ::Any) # there's only one field, so nothing else to access
    getfield_pullback(TY::Tangent) = SArray(TY...)
    return S.data, getfield_pullback # S.data is a tuple, so its tangent must be a Tangent
end

We could also ensure that all other operations on SArrays return SArrays.

It's not obvious that the same can be done for Symmetric / Diagonal / Transpose / Adjoint etc. The thing that made it work for SArray was the ability to meaningfully convert the structural tangent of its field (a Tangent) into the type of its field (a Tuple).

Ultimately, this seems to come down to how we define getfield's rrule. If we are willing to put the work in to ensure that any tangent for a field can be converted into a thing that can be put back into the primitive (and still have sensible semantics, like + and Real * working) then we might be able to treat a given type as primitive. This simply isn't possible for lots of arrays because, as discussed above, we can't generally rule out the need for a structural tangent, which rules out Symmetric etc.

This doesn't seem to have tied down exactly what we can / can't treat as a primitive, but I thnk it gets us closer, and it at least suggests that we can ensure the non-existance of structural tangents for SArrays if someone is willing to put the work in (and figure out where the code should live!)

[mcabbott]

It's pretty easy to make Zygote give you errors from natural + structural, for instance:

julia> gradient(x -> (real(x) + x.im), 1+im)
ERROR: MethodError: no method matching +(::NamedTuple{(:re, :im), Tuple{Nothing, Int64}}, ::Int64)

julia> gradient(x -> (sum(x) + last(x)), 1:3)
ERROR: MethodError: no method matching +(::NamedTuple{(:start, :stop), Tuple{Nothing, Int64}}, ::FillArrays.Fill{Int64, 1, Tuple{Base.OneTo{Int64}}})

These don't seem to be very common in the wild? In these two cases, I think the rule for getproperty should call a ProjectTo which standardises on the natural. But in general, it seems tricky.

[willtebbutt]

These don't seem to be very common in the wild?

Not super common, but I don't think I've worked on a project where I've not encountered them (for example, if you do enough things with a Diagonal, you will probably encounter at some point), and I usually wound up type-pirating to get around them.

In these two cases, I think the rule for getproperty should call a ProjectTo which standardises on the natural.

Again, not completely sure that I agree re. Fills and naturals for the reasons laid out above (the natural for a Fill is deeply unintuitive to me, whereas the structural is simple). For example, could / should @no_opt most things in FillArrays, including sum, and just get the structural.

edit: I meant @opt_out, not @no_opt.

[oxinabox]

It's pretty easy to make Zygote give you errors from natural + structural

For these cases one option is overloading +(::Tangent{FooArray}, ::AbstractArray})
which could then do either a projection onto the natural, or a structuralization onto the Tangent.
Or potentially even a form of thinking the just delays til we go to accumulate against a primal (Keno was talking about that a whole ago)

Possibly this should be a seperate issue

[willtebbutt]

For these cases one option is overloading +(::Tangent{FooArray}, ::AbstractArray})

Certainly -- this kind of code has to be written if we're ever allowing naturals and structurals to interact. I think there's a whole separate issue to be had about the points in the code at which we want the structural and at which points we want the natural.

For example, I think it's reasonable to assert that we often want the natural inside rules, as evidenced by people's experience over the last few months (not always though). Conversely, constructors like structurals, so it seems possible that we want to ensure that naturals don't escape rules, or something like that.

Possibly this should be a seperate issue

Yes.

[mcabbott]

In these two cases,

not completely sure that I agree re. Fills and naturals for the reasons laid out above (the natural for a Fill

Then you're looking at different examples, or talking about generalities. My specific two examples do not contain any tangent for an input x::Fill, natural or otherwise.

But if you don't find the natural for x::Fill intuitive, the natural gradient for a range will be worse.

However, the natural gradient for x::Complex is not so bad. So there are some structs for which we always want to normalise to normal. Notice also that in my example, it does not hit +(::Tangent{Foo}, ::Foo} -- the natural tangent from the other path lies in a subspace which isn't a subtype.

I think that's clearly what we want for SArrays too. Handling a Tangent{SVector}(; data=Tuple(...)) seems strictly inferior. In fact accessing its fields directly (in the forward code) seems extremely odd here, these are arrays. (Although obviously people do odd things in the wild & we should try to build guardrails, where possible.)

[willtebbutt]

My specific two examples do not contain any tangent for an input x::Fill, natural or otherwise.

Wow, sorry, I completely misread your example. You're right that it's painful working with these types. This is why I prefer the structural tangent for them, although I also agree that it's not always what you need.

I think that's clearly what we want for SArrays too. Handling a Tangent{SVector}(; data=Tuple(...)) seems strictly inferior.

I agree in the specific case of SArrays that we probably want them to be primitive.

In fact accessing its fields directly (in the forward code) seems extremely odd here, these are arrays. (Although obviously people do odd things in the wild & we should try to build guardrails, where possible.)

This I disagree with, in the general case. Array authors access fields all the time when writing methods optimised for their arrays, Fill arrays being a good example (getindex is essentially just getfield here). It's simply not odd to want to access fields of a structured array.

[mcabbott]

Again, that paragraph you are quoting from is specifically about SArrays, not generalities. I'm not sure I have ever seen anyone access the field by name; I had no idea what its name was. They are used as arrays in user code, and their interface with the outside world is through the usual AbstractArray interface. What the package authors do behind closed doors is, of course, nobody else's business.

I agree that what to do with other types is more tricky. Types that straddle the border between being containers and being arrays, I guess. Although I suppose I'd still like better examples of how these get used, to ground discussion. For instance field access is not really part of the API of Diagonal, and even less so for Symmetric (since it gets you meaningless data) but perhaps it is for some other weird type.

[willtebbutt]

Again, that paragraph you are quoting from is specifically about SArrays, not generalities. I'm not sure I have ever seen anyone access the field by name; I had no idea what its name was. They are used as arrays in user code, and their interface with the outside world is through the usual AbstractArray interface. What the package authors do behind closed doors is, of course, nobody else's business.

Cool. Apologies for generalising! Yeah, we're definitely in agreement that users probably shouldn't be accessing SArrays via getfield, that just seems weird. That being said, if we implement getfield properly, it should just be fine 🤷 (if ill-advised).

Although I suppose I'd still like better examples of how these get used, to ground discussion.

Well the ColVecs / RowVecs / Fill / WoodburyPDMat are all examples intended for this purpose.

ColVecs / RowVecs / WoodburyPDMat are of the "don't ever give me a natural tangent" flavour -- they're entirely written with a small set of AD-friendly operations in mind, so we have no need to write rrules for them, except perhaps to @non_differentiable some things here and there, and @opt_out of any generic rules that we don't like (although I don't think we've encountered any yet fortunately -- might change as people start writing more generically-typed rules though).

Fill arrays are a bit different -- pretty much every function they provide specialised implementations for should produce a structural tangent just fine without writing any rrules (e.g. sum, broadcast, map etc), but there are probably others that don't have specialised implementations and hit generic fallbacks, and therefore need the fallback rules + projection mechanism. So Fill is a good example where we should expect a mixture of different tangent types to appear during the reverse-pass of AD.

Are these examples missing something that you're interested in seeing?

For instance field access is not really part of the API of Diagonal, and even less so for Symmetric (since it gets you meaningless data) but perhaps it is for some other weird type.

I agree that field access is typically not part of the user-facing API of an array (although I also agree that it might be in some weird instances). However, AD ought to just work on functions written by library authors if they write code with AD in mind (library authors shouldn't have to write rrules if they write their code in an AD-friendly manner).

That being said, if they also want their types to work with the generic fallbacks defined for AbstractArrays, they'll need to write ProjectTo, and functionality to convert natural tangents into structural tangents so that one can AD through the constructors (unless they make ProjectTo return a structural, in which case the natural doesn't exist. Not sure what kinds of problems that can cause though -- I imagine it preceludes the use of some generic rrules that they might be interested in. As you say though, examples of this would help the discussion).

As regards people accessing internal fields of types when they probably shouldn't be, I agree that they probably shouldn't be. However, we do know how to AD through ill-advised field access, and we know how to represent the tangent. To my mind, from an AD's perspective, there's not really a distinction between approved field access by type authors, and ill-advised field access by misguided users.

Taking the Diagonal example, if someone writes

foo(D::Diagonal) = sum(D.diag)

we can AD it with no problem provided that the rule system doesn't somehow get in the way. If the standard library changes the internals of Diagonal and breaks a user's code, then of course the user is entirely to blame.

[mcabbott]

examples of how these get used

Are these examples missing something

Yes, thanks for the types, but what I meant was whether there are closer to end-to-end examples which give errors / wrong results / are inefficient. If ColVecs / RowVecs are glorified NamedTuples, which are always treated as structs, always Tangent, then there is no problem right?

The tricky cases seem to be (1) where some generic rule passes a special type along, and hence its generic pullback probably expects an AbstractArray, and (2) where the same object is used twice, and one path treats it as an array, the other as a struct, hence + fails. My explicit examples were of (2).

The simplest case I can picture for (1) is this, but I don't know why it isn't an error:

julia> gradient((x,y) -> (x*y).re, 1+im, 2)
((re = 2, im = nothing), (re = 1 - 1im, im = nothing))

Of course, this one should be avoided in multiple ways: You should write real() not .re to be safe. The getfield of Complex should not construct a Tangent/NamedTuple, since Complex is clearly simple enough to always be its own gradient; Maybe that needs a special overload as you suggest for SArray, or maybe the rule for getproperty just calls ProjectTo which handles it.

But there might be others which are less easy to avoid -- what are the simplest such? For Fill I can invent things if I use field access. But, the entire point of Fill is that it's a cheap thing to pass into generic array code which does not know that it's this type... if you knew this all the way through, you would use a number... so this seems like cheating:

julia> gradient(x -> sum(x .+ 10), Fill(3,3))
(3-element Fill{Int64}: entries equal to 1,)

julia> gradient(x -> (x .+ 10).value, Fill(3,3))  # generic unbroadcast does not handle Tangent/NamedTuple
ERROR: MethodError: no method matching size(::NamedTuple{(:value, :axes), Tuple{Int64, Nothing}})
Stacktrace:
  [1] unbroadcast(x::Fill{Int64, 1, Tuple{Base.OneTo{Int64}}}, x̄::NamedTuple{(:value, :axes), Tuple{Int64, Nothing}})
    @ Zygote ~/.julia/packages/Zygote/TaBlo/src/lib/broadcast.jl:51

[willtebbutt]

If ColVecs / RowVecs are glorified NamedTuples, which are always treated as structs, always Tangent, then there is no problem right?

You're correct. Again, lack of clarity on my part -- with these kinds of things I was just trying to assemble some concrete examples of things that do indeed work just fine, so that I can point to them if anyone is ever confused about the need for structural tangents for AbstractArrays.

The tricky cases seem to be (1) where some generic rule passes a special type along, and hence its generic pullback probably expects an AbstractArray, and (2) where the same object is used twice, and one path treats it as an array, the other as a struct, hence + fails. My explicit examples were of (2).

Agreed. The options for resolving case 2 are clearer to me than 1. Either, you must (I think)

• define a promotion rule such that if structural + natural ever occurs, something sensible happens, and you must define a constructor to accept the natural, or
• we insist that natural never escape pullbacks, thus ensuring that this situation never arises. Constructors can deal with structural tangents, so no need to worry about this in this situation.

I quite like the second approach because, unless I'm mistaken, it would make explicit what I think we're currently doing implicitly in a bunch of rrules for constructors (conversion from natural to structural), for example Diagonal.

I'm struggling to come up with good examples for (1). Any thoughts?

But, the entire point of Fill is that it's a cheap thing to pass into generic array code which does not know that it's this type... if you knew this all the way through, you would use a number...

This not always the case. My favourite example of this is in AbstractGPs. The third field of our FiniteGP type is always a covariance matrix (I'm not sure why we don't make it subtype AbstractMatrix now that I read the code again). Very often, it will be either

Diagonal(some_Vector_of_Reals)
Diagonal(Fill(s, N))

for some real number s and integer N, and some_Vector_of_Reals<:AbstractVector{<:Real}. A thing we rely on is specialising based on whether or not this matrix is Diagonal or a Diagonal{<:Real, <:Fill}, beause there are often entirely different algorithms that we can use in those cases. TemporalGPs relies on this entirely (because efficient inference isn't possible this is matrix isn't diagonal), and OILMMs.jl (which is currently being reworked into a more general package) only works when this is a Diagonal{<:Real, <:Fill}.

We need it to be an AbstractMatrix{<:Real} so that we can be sure it can be added to another matrix of the same size (wouldn't work for scalars or vectors). Generally speaking, if we hit this code path we couldn't care less what kind of matrix it is.

So I think the way in which this differs from what you're presuming is that some of the time we couldn't care less whether we have a Fill or not, but other bits of the time we really do.

But there might be others which are less easy to avoid -- what are the simplest such?

I think the Fill example is a good one. It can be made to feel a bit more realistic by opting out of sum by defining my_sum (if I'm not mistaken Zygote defines a rule for sum). Something like

julia> using FillArrays

julia> using Zygote

julia> using FillArrays: AbstractFill

julia> # Work around a Zygote performance bug by using Zygote.literal_getfield manually.
       my_sum(x::AbstractFill) = Zygote.literal_getfield(x, Val(:value)) * length(x)
my_sum (generic function with 1 method)

julia> Zygote.gradient(my_sum, Fill(5.0, 10))
((value = 10.0, axes = nothing),)

julia> Zygote.gradient(x -> my_sum(x .+ 10), Fill(5.0, 10))
ERROR: MethodError: no method matching size(::NamedTuple{(:value, :axes), Tuple{Float64, Nothing}})

[mcabbott]

we insist that natural never escape pullbacks

I'm not sure what this means exactly. By "natural tangent" we mean something returned by a pullback which isa AbstractArray, no? (Or isa Number) What happens inside the function we don't care.

What I'm hoping to elucidate with examples is whether any types actually have to straddle the border between array-like and struct-like. I don't see why we shouldn't treat Fill as array-like, 100% of the time. Like Complex should be Numer-like, 100% of the time. Conversion is free. Whereas it sounds like ColVecs / RowVecs are like NamedTuples, structural 100% of the time --- maybe they are what you mean by "never escapes"? Can we partition all types into one category or the other?

[willtebbutt]

I'm not sure what this means exactly. By "natural tangent" we mean something returned by a pullback which isa AbstractArray, no? (Or isa Number)

Yeah, more or less. My working definition is (excluding primals like Array and Float64 etc) that a natural tangent (/ differential, I've stopped calling them differentials) is anything that's not a Tangent, or one of the other tangent types in CRC (ZeroTangent, NoTangent, etc).

What happens inside the function we don't care.

Definitely -- modulo it returning acceptable types.

What I'm hoping to elucidate with examples is whether any types actually have to straddle the border between array-like and struct-like.

Symmetric / Diagonal have to straddle, because they can wrap arbitrary AbstractMatrixs / AbstractVectors respectively, and said AbstractMatrix / AbstractVector might be a container-like AbstractArray, meaning that it only has a structural tangent, which can't be put inside a Symmetric / Diagonal.

e.g. the tangent of a

Symmetric{<:Real, <:WoodburyPDMat}

would have to be a Tangent, because the tangent of a WoodburyPDMat must be a Tangent.

I don't see why we shouldn't treat Fill as array-like, 100% of the time.

This is a fair point -- I also can't see any reason, beyond my lingering doubts about the properties of using it as its own tangent (I'm concerned that sometimes it needs to behave like a struct and other times like an array, and maybe that leads to inconsistencies. I'm planning on trying to convince myself one way or the other over the next couple of days).

maybe they are what you mean by "never escapes"?

By "never escapes", I mean that a pullback always returns a Tangent (or ZeroTangent etc).

My understanding is that natural tangents are primarily a way to

1. make generic rrules possible to write (which is great, this is a massive win), and
2. produce an interpretable tangent.

This suggests to me that they're important in exactly two places:

1. inside rrules, and
2. outside of AD (e.g. to make the result of AD more interpretable if a structural tangent is returned and for some reason it's helpful to have in natural form. I'm sure examples exist here, but I've actually yet to see one)

Outside of these two situations, it's potentially a different matter. For example, inside the AD pipeline, we could require all tangents to be structural (or ZeroTangent etc), which would then be simple to deal with (+ always well defined, default constructor rrules happy). We could make this easy for users to do by requiring project_to to instead return the structural tangent (presumably a breaking change, not possible in 1.x, I assume?), and providing another function to do the structural -> natural mapping (Tangent to Symmetric, for example, when it makes sense to do, and erroring if the particular Tangent doesn't have a corresponding natural tangent). This could be done at the start of a pullback definition, if the user would rather have e.g. an AbstractArray than a Tangent (could also unthunk automatically etc).

My suspicion is that this kind of change would be minimally invasive, and would have the benfit of localising errors to the inside of rrules (instead of not being able to +, conversion to / from natural tangent might fail. The former case tends to happen in the middle of an AD pipeline, while the latter should happen inside an rrule that a human wrote. Consequently, the latter is probably easier to understand and debug). It would also make the rule-writer's expectations about the types they're going to get be really explicit. By asking for the natural tangent, you're saying "I want something array-like, on which I can perform the usual array operations" (in the context of tangents for AbstractArrays).

For example, the rule for *(::AbstractVecOrMat, ::AbstractVecOrMat) would become something like

function rrule(
    ::typeof(*),
    A::AbstractVecOrMat{<:CommutativeMulNumber},
    B::AbstractVecOrMat{<:CommutativeMulNumber},
)
    Y = A * B
    project_Y = ProjectToNatural(Y)
    project_A = ProjectTo(A)
    project_B = ProjectTo(B)
    function times_pullback(ȳ)
        Ȳ = project_Y(ȳ)
        dA = @thunk(project_A(Ȳ * B'))
        dB = @thunk(project_B(A' * Ȳ))
        return NoTangent(), dA, dB
    end
    return Y, times_pullback
end

edit: re FillArrays, if you have an operation which AD successfully derives, it'll presumably expect a Tangent as the tangent for its output on the reverse-pass. Trying to find a good example now.

edit2: no, with getfield implemented appropriately, this should be fine.

edit3: break up long sentence.

[mcabbott]

By "never escapes", I mean that a pullback always returns a Tangent

You mean every pullback always returns a Tangent, I think. (Or maybe not for Complex.) I'm going to dub this the nuclear option. I think it means that arrays unknown to the AD system can never propagate through AD, as arrays. They all need special reconstructor functions before they can be used within generic rules. It can't "just work" with SArrays & OffsetArrays.

The "not straddling" idea is I think to tie the behaviour to type of x. Perhaps, for some types, all pullbacks always return a Tangent, while for others, no pullback ever returns such. The type of x::Symmetric contains the type of parent(x), maybe that counts. If every x is on one side of the fence or the other, then my problem (2) never occurs, + is never confused.

I still think we need better examples of problem (1). That is, sufficiently weird array types not to have easy natural tangents, which are nevertheless the result of the forward pass of some generic function, whose generic rrule we might hope to use. If the construction of a natural tangent is possible but expensive, then you can see some virtue to delaying it until certainly necessary. If the construction is impossible, then you cannot use the generic rrule under any scheme, so we're not in problem (1) anymore. (You will either need to write another, or opt-out.)

[willtebbutt]

I'm going to dub this the nuclear option.

Haha fair enough. I would prefer it to be characterised as the "clean" option, in that it cleanly separates the world of the users / rule-implementers from the world of the AD.

I think it means that arrays unknown to the AD system can never propagate through AD, as arrays.

Is this a problem? AD systems like structural tangents -- that's their "native" representation of any struct, and by extension their native representation of any AbstractArray that isn't a primitive.

They all need special reconstructor functions before they can be used within generic rules.

I'm not convinced we don't already have to do this somewhere, it's just unclear where. For example, we already have to write code to map from the natural to the structural inside the a hand-written rrule for the constructor.

How is it that an AbstractArray tangent comes into existence in the first place if not via conversion from a structural tangent? (structural tangents being the only thing that can be generated automatically).

The "not straddling" idea is I think to tie the behaviour to type of x. Perhaps, for some types, all pullbacks always return a Tangent, while for others, no pullback ever returns such. The type of x::Symmetric contains the type of parent(x), maybe that counts. If every x is on one side of the fence or the other, then my problem (2) never occurs, + is never confused.

Sorry, I'm really struggling to follow this para. Could you elaborate?

I still think we need better examples of problem (1).

Is the Fill example not sufficient? It has the key properties that we care about, does it not?

edit: ignore my last comment. Reading your para again.