Provider initiated negotiations are not implementable

[jimmarino]

Bug Report

The contract negotiation state machine allows providers to initiate a negotiation by sending an offer message. However, this sequence is not well-defined and has several implementation issues.

Flow description

An offer flow is only defined in the binding specification. This needs to be pulled out to the abstract negotiation spec document.

Consumers cannot decide which offer notifications to receive

There is no way for the consumer to decide which offer notifications it is interested in. In the current approach, the provider would have to indiscriminately send all new offers, which is not desirable. There is also a major scalability concern as all notifications must be done unicast by the provider to consumers.

Proposal

These issues and the complexity supporting provider-initiated negotiations indicate that the initial requirement prompting this feature may not be modeled correctly. That requirement is to provide a way for providers to notify consumers about an offer. This could be handled in two ways:

• Catalogs are designed to be crawled periodically. The consumer could perform a diff of the new and previous catalog to determine new offers
• An event stream dataset could be created by the provider that the consumer negotiates a contract for and opens a transfer process. The event stream provides notifications of new offers.

Either of these approaches addresses the initial requirement, significantly simplifies the DSP design, and allows the consumer to decide which offers it would like to be notified about.

The proposal is to remove provider-initiated negotiations and provide a recommendation to use one of the two approaches in a separate best practices document.

[juliapampus]

to be further discussed

[jimmarino]

I think we need to look at the concrete use cases prompting this feature. One potential use case is provider event notification to multiple consumers. This is best modeled using the existing DSP mechanisms outlined above. This approach is consistent with pub/sub messaging paradigms where consumers subscribe to provider notifications (not the other way around).

To model the event notification use case with the proposed mechanism, there would need to be some out-of-band step where the consumer indicates to the provider that it is interested in a topic (i.e., it subscribes). Otherwise, the provider must indiscriminately send all notifications to all consumers it somehow knows about, which is not scalable or performant. More importantly, in this case, having the "out-of-band" step requires non-specified behavior for this feature, which, by definition, renders it non-interoperable.

Let's consider the case where the subscription step is done by another process outside of DSP. This is still easier to model with the approach specified above. In that case, the consumer-initiated contract negotiation would be checked on the provider for the existence of the out-of-band process approval. This can be signaled in an interoperable way at the dataspace level through policy.

The bar to add a feature is that it cannot be easily achieved using existing mechanisms. The event notification use-case can already be handled in a completely interoperable way using existing DSP mechanisms. In contrast, the provider-initiated flow requires non-interoperable features (the out-of-band step).

Therefore, I think we should pose the question: which concrete use-case cannot be solved using existing mechanisms, and is it worth it to include a feature in the spec that requires non-interoperable steps? In other words, why does the subscription have to be performed out-of-band rather than through DSP?

[jimmarino]

Some additional issues regarding performance, scalability, and potential DDoS attacks that we did not have a chance to get into.

The provider-originated offer approach models notifications as contract negotiations in the control plane. This should not be done because a notification is different than a negotiation. Another way of stating this is to say that a notification is a data plane concept, not a control plane one. In the current state machine, the provider-originated offer creates a new contract negotiation. The sequence is as follows:

1. If the provider is running a connector that implements reliable messaging, the connector must persist the contract state machine and its identifier prior to sending the offer to to the client.

2. The client receives the offer. There currently is no mechanism for the client to say, "No thanks." If it sends an 4XX error back, the provider will likely retry or put the negotiation in an error state. The only clean option is to send a terminate message back to the provider callback address. If the client is reliable and is designed to issue an asynchronous callback, it must persist the contract negotiation.

That sequence raises several issues:

1. It's not a scalable way to handle notifications. Every event results in the creation and completion of a contract negotiation that must be done 1:1 (unicast) to every client. In the approach outlined above, notifications are sent via the data plane and don't incur this overhead. They could also be sent over a multicast protocol or eventing system (e.g. a cloud event grid) that supports high-volume pub/sub messaging. In the provider-originated approach, there is also no way for the client to filter events without creating a contract negotiation. In contrast, this is simple to do in the data plane.

2. A provider can inadvertently DDoS a client by sending thousands of offer requests that result in starting contract negotiations. Technically a consumer could do the same thing to the provider, but it is less likely to happen since negotiation requests are not prompted by events. Also, in the case where a request is sent by the consumer, the provider can immediately reject it if the consumer is not authorized since client credentials can be matched against the offer. There is no defined way to match provider credentials in the other approach.

3. The provider has no way to gracefully "cleanup" rejected offers. If it receives a 4XX by the consumer, it will likely retry (depending on the specific error code) or put the contract negotiation in an error state.

[dimitrov-anasoft]

This was my initial thought - a provider is generally not aware of its possible consumers. So why bother specifying something which is of questionable utility? Also, this makes the spec unnecessarily convoluted.