Deprecation of implicit scenario identification

[kleihan]

Orchestra v1.1 RC1 introduced a distinction between datatypes and code sets for a field with #170. This does not change the implicit link between field and code set scenarios as defined by Orchestra v1.0. The scenario of a field or a code set are always defined, absence simply means that it is "base". Orchestra v1.1 RC1 further introduced scenarios for datatypes with #151.

The proposal is to support a datatype scenario and a code set scenario as part of the field definition independent of a field scenario. See below for details.

For clarity, scenario names are used as reference (Orchestra v1.1 requires the scenario ID and names are optional). When defining a field with a datatype scenario, the Orchestra XML file needs to contain a datatype with the same scenario. There is an implicit link between the two element types. Otherwise, the field is not properly defined. The following example shows a correct definition:

<fixr:field id="34" name="MsgSeqNum" type="int" scenario="SeqNum"/>
<fixr:datatype name="int" scenario="SeqNum">
	<fixr:mappedDatatype standard="XML" base="xs:positiveInteger"/>
	<fixr:mappedDatatype standard="ISO11404" base="Ordinal"/>
</fixr:datatype>

The problem now comes with a requirement to create a scenario "Short" for the field that is independent of the datatype scenario, e.g. to limit the field length to 5. This cannot be expressed without adding a new attribute, e.g. typeScenario, so that the existing attribute scenario only applies to the field.

<fixr:field id="34" name="MsgSeqNum" type="int" typeScenario="SeqNum" implMaxLength="5" scenario="Short"/>

The same issue arises with code sets. When defining a field to have a code set, the Orchestra XML file needs to contain a code set with the same scenario. Otherwise, the field is not properly defined. The following examples show correct definitions:

<fixr:field id="59" name="TimeInForce" type="TimeInForceCodeSet"/>
<fixr:codeSet id="59" name="TimeInForce" type="char">
    <fixr:code name="Day" id="59001" value="0"/>
</fixr:codeSet>

<fixr:field id="59" name="TimeInForce" type="TimeInForceCodeSet" scenario="DayOrders"/>
<fixr:codeSet id="59" name="TimeInForce" type="char" scenario="DayOrders">
    <fixr:code name="Day" id="59001" value="0"/>
</fixr:codeSet>

It is hence not possible to combine a field scenario with a different code set scenario without adding an explicit attribute, e.g. codeSetScenario.

See also issue FIXTradingCommunity/fix-orchestra-spec#49.

[francescoloconte]

If I understood correctly, when defining a Field (like in the example above), its type would need to be identified by a tuple [type_name; type_scenario], instead of only the type name as it is now (since the Field's scenario and it's type's scenario may be different).

I think it follows naturally that every element that supports scenario, such as Datatype, now requires a tuple to be identified when used. So, I think this makes sense.

On a separate note, I am still not comfortable with the change in #170 , where the name of the attribute that indicates the Field's type depends on whether it is a dataType or a codeSet (type or codeSet). I think the attribute name should be unique, i.e. type,. The separation of namespaces for dataTypes and codeSet can also lead to a type and a codeSet having the same name. In general, this change is saying that dataTypes and codeSets are very different things. I think they are the same: they are datatypes. The initial requirement was to provide a way to tell them apart when we are using them in Fields' definitions, but we have ended up with a design that splits them apart as if they are very different things.

[donmendelson]

@francescoloconte I agree that a code set is logically a subclass of datatype, not a disjoint class. It has the added attribute that its data domain is a finite set of valid values.

If I understood correctly, when defining a Field (like in the example above), its type would need to be identified by a tuple [type_name; type_scenario], instead of only the type name as it is now (since the Field's scenario and it's type's scenario may be different).

"...[identified by] only the type name as it is now" is true if by "now" you mean Orchestra v1.0. In v1.1 RC1, it is identified by the tuple of name plus scenario id (where scenario id defaults to 1 for base scenario).

Although that scheme met most requirements in my view, some users reported that the fact that name could refer to either a datatype or a code set was confusing. Hence the attempt to make the distinction with issue #170.

[francescoloconte]

Premise: for the sake of the discussion, I will refer to codesets as Bounded types, and to others types (e.g., integers, float, strings, etc.) as Unbounded types, whereas the former types have a finite number of possible values, and the latter types have an infinite number of possible values. This is also called cardinality.

@donmendelson , would it be correct to say that the shortcoming seen by the users (as you indicated above) can be summarized as it follows: When looking at a Field's Type, one cannot tell whether it is Bounded or Unbounded (e.g., an Integer or a AllocStatusCodeSet). To find out, one has to search the codeSets element for a matching Bounded type, and if not found, one has to search the datatypes element for a matching Unbounded type. Their desire is to avoid this search by surfacing the cardinality of the Field's Type to the Field's definition. This is achieved by the addition of an attribute in the Field's definition, as indicated by #170."

Since both Bounded and Unbounded data types are types, perhaps it would have made sense to have them under the same XML element to require only 1 search, while they currently go under codeSets and datatypes respectively, which creates also two unique name spaces, where one can have a codeSet and a dataType with the same name.

I see issues with the type system surfacing more often lately; for example, in the conversation about support for binary protocols,, representation of sets, and Mapped Data Types, to name a few.

As you indicate in this comment, I agree that we should dedicate time to review the Orchestra type system, and "clean it up", at the risk of breaking backward compatibility, which is fine now since usage is not high, but it will provide a solid base for the standard to evolve. I don't feel confident building more complexity while the type system is not clarified.

[kleihan]

@francescoloconte as long as we are in the Release Candidate stage of a version of a Technical Standard, there is no guaranteed backward compatibility. The idea is that RCs should be implemented by early adopters and provide the possibility to take back a change that was previously made because of disadvantages identified by an implementation. That changes with the release of Beta for a given version. The final publication of a version has to be backward compatible with Beta. Only a new version (with its own series of Release Candidates) can then make breaking changes.

That currently means that Orchestra v1.1 RC2 can undo changes made with RC1.1. We should try to avoid it, but such changes are covered under the standard process.

[kleihan]

Just to summarize the status quo: Orchestra v1.0 has a single attribute type and the user has to look for it in the definition of <fixr:datatypes> and <fixr:codeSets>. With Orchestra V1.1 RC1, the attribute type can only be used for datatypes and the attribute codeSet can only be used for code sets. The attributes are mutually exclusive, using both or none of them for a field definition is an error. The user has to determine which of the attributes was used and then look for it in <fixr:datatypes> or <fixr:codeSets>.

You used the terms bounded and unbounded to distinguish the two. That is correct for most cases but not for all, for example a datatype char is bounded by the fact that it can only have 255 values (if one counts non-printable characters as well). Code sets, on the other hand can be unbounded if they have a union dataytype of ReservedXXXPlus, allowing any number of custom values. My view is that the two terms do not help to distinguish the two cases.

What are the options for reversing the change made with RC1? Following the argument from @donmendelson that a code set is logically a subclass of datatype, it may make sense to allow an optional attribute codeSet in addition to the the mandatory attribute type. The datatype of the field would then be available without having to lookup the code set (if it has one). Presence of the optional code set would then limit the values of the field accordingly. The code set definition itself should probably maintain its own attribute type for clarity and the schema validator can ensure that the datatype of field and code set fit together.

What would that mean for the topic of this discussion issue (Deprecation of implicit scenario identification)? The two additional attributes typeScenarioand codeSetScenario would still be applicable. A field definition could then have its own scenario V with a datatype scdenario B and a code set scenario C.

Example (all examples are bad but I hope I can make my point clear):
A field containing an explicit list of MICs under ESMA jurisdictions. MICs are strings with a length of 4 characters. The field needs its own scenario (in addition to base) to express a default value (e.g. only allowed in an ExecutionReport but not in a TradeCaptureReport).

<field
    id="30" name="LastMkt" scenario="DefaultMarket" value="NYSE" 
    type="String" typeScenario="Acronym4" 
    codeSet="LastMktCodeSet" codeSetScenario="ESMA">
</field>