[repository schema] Scenario Relationships

[kleihan]

Background

Scenarios are one of the key features of Orchestra. There is a strong requirement to clearly define relationships between scenarios. Version 1.0 does not provide this. The issue was also raised in #212, together with a proposed solution. The proposal in this issue is to cover the basic types of relationships between scenarios and establish well-defined rules for them.

Elements supporting scenarios are messages, groups, components, fields, code sets, and datatypes. The focus of this issue is on those elements that typically consist of many other elements. This applies to messages, groups, components, and code sets.

The scenarios of such an element can have any number of nested elements. There are no rules that address a potential overlap of elements that are shared between scenarios. This leads to redundancies across scenarios that negatively impact the maintenance effort and may easily lead to inconsistencies.

Scenario Relationships

There are two basic relationships that should be defined in Orchestra and made optionally available for the design of scenarios. There will be use cases that benefit more from one type than the other and the choice of type is part of the design process.

• Subset: a target scenario (B) is a subset of a source scenario (A)
• Superset: a target scenario (B) contains a source scenario (A) and adds additional elements

Subsetting

Subsetting requires the ability to exclude certain elements from the target scenario. Excluding elements is already covered in Orchestra v1.0 with the attribute supported that only needs to be used when an element is not supported. It has the values forbidden and ignored. The choice depends on whether the message may contain this element even if it is not used. The target scenario needs to contain at least all elements that are to be excluded. It may repeat elements from the source scenario that are needed in the target scenario but with different attributes, e.g. a mandatory instead of an optional presence.

Supersetting

Supersetting requires the ability to add elements to the target scenario. This is also covered by Orchestra v1.0 as you can simply add elements to one scenario that are not in another. Similar to subsetting, the target scenario may repeat elements from the source scenario that need different attributes, e.g. a mandatory instead of an optional presence.

Role of base scenario

The issue that needs to be resolved is to establish a relationship between two scenarios and define the type to be either a subset or a superset. Relationships always exist between two scenarios at a time. The base scenario should only be allowed to be a source scenario and never a target scenario. When using the base scenario as source, subsetting means that the base is large, whereas supersetting means that it is small, compared to the target scenario.

New attributes

The ability to define relationships between two scenarios requires additional attributes for elements in Orchestra v1.1 as follows:

• Reference to a source scenario
• Relationship type

Rules for Relationships

With the definition of a relationship type comes a number of rules that allow tools to validate or even enforce this relationship. This is especially important for use cases (e.g. FIX tag=value encoding) where the ordering of elements across scenarios is important.

Rules for subsetting

1. Source scenario contains all elements of the target scenario
2. Target scenario can have less than or equal number of elements
3. Target scenario cannot be "base"
4. Elements from source scenario may be repeated in target scenario to modify their attributes
5. Absent attributes of repeated elements from source scenario remain unchanged (no defaults)
6. Ordering of elements must not change between source and target scenario
7. New elements may only be added to the source scenario, which does not have to be "base"
8. New elements in the source scenario will automatically be available in target scenarios unless excluded there

Rules for supersetting

1. Target scenario contains all elements of source scenario
2. Target scenario can have more than or equal number of elements
3. Additional elements are appended at the end of the elements from the source scenario
4. Target scenario cannot be "base"
5. Elements from source scenario may be repeated in target scenario to modify their attributes
6. Absent attributes of repeated elements from source scenario remain unchanged (no defaults)
7. Ordering of elements cannot change between source and target scenario
8. New elements may be added to the source and/or the target scenario

Summary

The proposal is to start with two basic relationship types with clear rules for them. There may be use cases that require more complex relationship types with a different set of rules that mix the two basic types. It is not intended to cover everything that may be technically possible in the first step but to keep it simple and ease the adoption of scenarios as a concept. Orchestra V1.0 has been insufficient in this respect and also the scenario enhancements of Orchestra V1.1 RC1 did not close that gap.

[kleihan]

Some additional comments. Scenario relationships can support the efficient design of scenarios and ease the maintenance task. The related requirements (e.g. consistency, ordering) are different from those of the recipient of an Orchestra XML file who wants it to be as easy to consume as possible. A separate topic is the question whether the coverage of Orchestra features beyond the static definition of messages and their elements (=QuickFIX) should be subject to bilateral agreement and/or defined inside the Orchestra XML file. That allows a recipient to reject a file, e.g. because of the inability to deal with scenarios.

The requirement to combine subsetting and supersetting can be covered by creating an intermediate scenario, e.g. creating scenario B by subsetting scenario A, followed by creating scenario C by supersetting scenario B. The Orchestra XML file provided to the counterparty may then only include scenario C. This requires a tool that fully declares scenario C using the information available from scenarios A and B. It also requires to drop the relationship information (scenario B is not provided), making it fully backward compatible with Orchestra v1.1.

The proposal also makes it possible to fully declare the scenarios A, B, and C whilst declaring the intended relationship. A tool can then check the compatibility of the scenarios with the relationship, e.g. the unintended change of the ordering across scenarios.

The requirement to reorder elements across related scenarios may be another use case and should be covered by a different relationship type.

[jimnup]

I think we have to look at the use cases and 90% use of FIX. The vast majority of FIX adoption uses a very small subset of available fields, components, messages and then extends that very small subset. This is what a scenario is. The idea of having to specify if a scenario is a subset or superset is a non-starter. Also, the notion that a scenario needs to list everything available in the element (be it a message, codeset, component) and disable it is a nonstarter - way too much work and too much parsing to be of value for automation in the community. A scenario should only require what is in use. A typical scenario for the New Order message should likely be less than 50 elements. If we have to include in that scenario the full entirety of the message elaborating what is not in use is unwieldy at best.

[donmendelson]

To help with this discussion, here's a bit of background on scenarios.

Original Proposal

The original model was additive only. In my original proposal for scenarios, there would be a "base" scenario (base class) for a message type that would contain the fields that are common to all scenarios of a given message type. These would be inherited by all other scenarios (subclasses). For example, the base scenario of Instrument component might contain Symbol, which would be suitable for equities while the scenario for futures would contain MaturityMonthYear, and the scenario for options would also contain PutOrCall and StrikePrice.

The rationale for this design was that each field of a message would only need to be entered and edited in one place. If a field needs to be added to all scenarios of NewOrderSingle for regulatory reasons (actual case), then it can be inserted in the base scenario and thus inherited in all scenarios.

Issues with Original Proposal

• Requires analysis to determine what a derived scenario actually contains.
• FIX tag-value encoding requires fields in a repeating group to be listed in a specific order. This introduces difficulties for any inheritance model.

For these reasons, the original proposal was rejected by the working group. The term "base" was retained for a default scenario, however.

Common practice

FIX Latest is the universe of all possible message scenarios across all markets and asset classes. I'm in agreement with @jimnup that actual message designs are much more focused in practice. He said, "A typical scenario for the New Order message should likely be less than 50 elements." I'll go further and say that in some protocols, an order message is less than 50 bytes.

Therefore, all the additive and subtractive operations are overkill. Just list the contents of each scenario explicitly. Keep it simple. This takes care of the ordering of fields. Yes, users who compose Orchestra files may want to keep additional relationship metadata, but this can be managed by tooling to produce the final Orchestra file that they share with counterparties.

[kleihan]

After further discussion, I suggest to reduce the proposal to the addition of the ability for messages, groups, components, and code sets to reference another scenario and to define the semantic of the relationship in the spec as follows.

Whenever an element references another scenario, it must be a restriction of the referenced scenario. This supports the "subsetting" as described above and can be validated by tools. Furthermore, it allows the definition of message variants for ISO 20022 (see https://www.iso20022.org/catalogue-messages/additional-content-messages/variants for details).