[Legacy iMC Thread] Be able to specify if a field can accept extra-values

[francescoloconte]

[Poster Xavier B.]

We have a need which arises regularly and there is currently no satisfactory solution in Orchestra. We have two parties which are connected through an intermediary system (routing system). The two parties can agree to support custom values for some specific fields with a code set but the intermediary system does not know what are these values. We need the parties to be able to specify in their Orchestra file which fields can accept extra-values, so our system won't reject the messages if values are not in the code set.

We are thinking to use unionDataType by specifying "string"/"character" (unionDataType="string") or a specific data type based on a regexp.

In standard FIX, they are several fields which can accept extra-values but the only way to know it is by parsing the field description. For example, MsgType can accept values starting with "U", TradingSessionID can accept values starting with any character, TriggerOrderType can accept any character, etc.

Do you think this is something we could include in the Orchestra specification? Or maybe do you have other suggestions?

[francescoloconte]

Have you already checked these? The 'unionDataType' attribute has already been added to code sets.

• Union datatype fix-orchestra-spec#10
• [repository schema] unionDataType should be an attribute of a code set, not field #161

[francescoloconte]

[Poster Xavier B.]

The problem is the restriction. unionDataType only allows some predefined types.

<xs:restriction base="xs:string">

<xs:enumeration value="Qty"/>

<xs:enumeration value="Reserved100Plus"/>

<xs:enumeration value="Reserved1000Plus"/>

<xs:enumeration value="Reserved4000Plus"/>

<xs:enumeration value="Tenor"/>

</xs:restriction>

[francescoloconte]

[Poster Hanno K.]

Xavier B. please provide an example of a code set that you want to extend. If the code set has datatype "int" then why do you need a unionDataType of "string"? ReservedXXXPlus would allow you to add values. String values should not be supported for a code set with integers. It does exits the other way around in FIX Latest, e.g. TradingSessionID(336) started out as a string and we added a code set as well as Reserved100Plus to accomodate pre-defined numbers (1,2,3,..) as well as strings ("Day, Morning,...) for backward compatibility.

Have you looked into the Orchestra element and whether that could support regular expressions for a code set value?

[francescoloconte]

[Poster Xavier B.]

Hi Hanno, in our case we want to be able to extend any code set. However in my initial post I provided examples of tags which can be extended in standard FIX and which currently don't define any unionDataType:

• MsgType can accept values starting with "U",
• TradingSessionID can accept values starting with any character,
• TriggerOrderType can accept any character.

We want our application to accept these extended values without having to define them in the code set. Currently the only solution we found is to parse the description of the fields to find something like "Bilaterally agreed values can be used" but it is not satisfactory.

As a side note, I don"t see any reason of restricting types which can be used for "unionDataType". Especially because Orchestra is not supposed to be FIX-specific while the restricted types are FIX-specific.

[francescoloconte]

[Poster Hanno K.]

Hi Xavier, I think I understand now where you want to go with this and I think that the concept of "unionDataType" is not the right one here. You want to be able to add user-defined values to any code set without changing the data type. However, the "unionDataType" is also for a union of two different data types, e.g. string and integer. ReservedXXXPlus is for a union of int+int but there is no equivalent for String+String or char+char.

• MsgType already has data type "String" and you want a "union" with "String" where the first character is a "U"
• TradingSessionID has data type "String" and a union with Reserved100Plus which is an integer of 100 or above
• TriggerOrderType has data type "char" and the field definition allows to also use any other OrdType value which is also "char". It does not accept any character.

What I believe you want is the ability to express that a code set supports values of the same data type that are not explicitly listed in the Orchestra XML file. That could be achieved by adding an attribute to the definition of a code set, e.g. "customCodes=0/1". That can then be used for the close to 70 code sets where FIX Latest has text to allow bilaterally/mutually agreed values, e.g. SecurityType.

The alternative may be to add ReservedStringPlus and ReservedCharPlus to indicate that user-defined string or characters are permitted. However, that clashes for fields that are defined as string (e.g. TradingSessionID) and already have a union with integer values (e.g. Reserved100Plus). Those fields may support user-defined strings as well as integer values above 100 and we should not add a second union data type. Having something like "customCodes=0/1" would solve that problem and focus solely on the aspect of user-defined values with the same datatype.

To make a long story short, I would advise to keep the concept of union data types separate from the concept of user-defined values.

[francescoloconte]

[Poster Hanno K.]

I forgot your side note. It is true that the current unionDataType values are all FIX data types. However, I see the mechanism of supporting user-defined values of an integer field above a specific number as generic and not FIX-specific. I agree that "Qty" is rather fiX-specific and solved a FIX-only problem of IOIQty(27) that started as character values and was changes to also support decimal number. Probably not a good design choice back then.

Strictly speaking, unionDataType must be a union of two different data types, even if they have the same base data type, e.g. int+Reserved100Plus is ok but int+int is not.

[francescoloconte]

[Poster Xavier B.]

Thank you Hanno for your reply. I suggested the use of "unionDataType" because it is already supported by our application but any other solution is good to us. Having an attribute "customCodes=0/1" (I assume 0/1 means false/true) would fulfill our need. It does not allow to specify the format of the custom codes but for our use case, we don't need it.

[francescoloconte]

[Poster Xavier B.]

Can this request be considered for the next Orchestra release?

[francescoloconte]

Hi Xavier,

Currently we are finalizing RC1 of Orchestra v1.1.

Do you mean for RC2 of Orchestra v1.1, or for Orchestra v1.2?

[francescoloconte]

[Poster Xavier B.]

If possible for v1.1 since we need it for our application. I think we need to refine the solution in a meeting first anyway.

[francescoloconte]

[Poster Hanno K.]

This forum can be used to discuss ideas but is not a pre-requisite for the formal step of adding an issue to https://github.com/FIXTradingCommunity/fix-orchestra/issues in GitHub. Francesco and I have been adding issues there in the past and GitHub also allows a discussion.

It is too late for v1.1 RC1 but there is plenty of time left until v1.1 is final. I understand the proposal to be an attribute on the field level that indicates whether bilaterally agree values are supported in addition to the standard values defined by FIX. This automatically applies to all FIX fields with a unionDatatype of ReservedXXXPlus but also to others where the field description has text like "Additional values may be used with bilateral agreement." The term "bilateral" appears 56 times in FIX Latest. Presence of ReservedXXXPlus simply means that the user-defined values are limited to a specific range. Absence means that there is no such limitations (FIX Latest only uses this for string fields).

Xavier B. I think this is a useful enhancement and I suggest that you add an issue into GitHub for that.

[martinswanson]

>>> The two parties can agree to support custom values for some specific fields with a code set but the intermediary system does not know what are these values. We need the parties to be able to specify in their Orchestra file which fields can accept extra-values, so our system won't reject the messages if values are not in the code set.

This sounds like a special case of the more general requirement for Orchestra to handle external codesets.

In the special case of external codesets that are restricted to certain participants, I think this is actually a common requirement for many business flows. An example that comes to mind is the atsOrderType field for FINRA's Consolidated Audit Trail (CAT) reporting. In this scenario, industry members operating an Alternative Trading System have to register their custom order types with FINRA, so they can be validated as part of the report submission process (see Alternative Trading System Order Type Identifier Registration).

An example of a more general case would be validation MIC codes. These are not defined in Orchestra, and are updated frequently.

I am not convinced that Orchestra should be used to specify external codesets. It feels more like content validation logic that belongs a layer above Orchestra, at the application level. In this case, you can simply specify the field with a datatype rather than a codeset. This is the route we took for FINRA-CAT, where the atsOrderType field has type StringList.

It might be useful to have a mechanism in Orchestra to identify the external codeset and specify where it resides.

If we do go down the route of trying to internalise external codesets in Orchestra itself, then it we could add the ability to flag a codeset as "extensible". In this case, any (static) codes that are specified in Orchestra could still get validated, but validation would not fail if there is an unknown code that conforms to the type/pattern for that codeset. In this case, we would still need a mechanism to identify the external codeset and locate it.

These options are not mutually exclusive, i.e. you can use either a datatype or a codeset, with or without specifying an external codeset.

[ltaikit]

Martin, you bring up interesting point about external code sets. ISO 20022 is in the midst of a standard revision right now, which I'm involved in. They are looking to address identifying whether a code set is external or not by way of an additional Boolean attribute - this will be a change to the ISO 20022 meta model (Part 1 of the standard). Regarding codes such as MIC I'm sure you're aware that some fields with the FIX datatype of "exchange" already types that field to contain only MIC value. The intent is not for FIX to capture all MIC values, it's up to the implementation to find the code values from the MIC website which provides the list in various formats https://www.iso20022.org/market-identifier-codes From: Martin Swanson ***@***.*** Sent: Friday, May 3, 2024 06:38 To: FIXTradingCommunity/fix-orchestra Cc: Subscribed

Subject: Re: [FIXTradingCommunity/fix-orchestra] [Legacy iMC Thread] Be able to specify if a field can accept extra-values (Discussion #200) >> The two parties can agree to support custom values for some specific fields with a code set but the intermediary system does not know what are these values. We need the parties to be able to specify in their Orchestra file which fields can accept extra-values, so our system won't reject the messages if values are not in the code set.

This sounds like a special case of the more general requirement for Orchestra to handle external codesets. In the special case of external codesets that are restricted to certain participants, I think this is actually a common requirement for many business flows. An example that comes to mind is the atsOrderType field for FINRA's Consolidated Audit Trail (CAT) reporting. In this scenario, industry members operating an Alternative Trading System have to register their custom order types with FINRA, so they can be validated as part of the report submission process (see Alternative Trading System Order Type Identifier Registration) <https://www.catnmsplan.com/sites/default/files/2020-03/CAT%20Alert%202019-01%20-%20Alternative%20Trading%20System%20(%E2%80%9CATS%E2%80%9D)%20Order%20Type%20Identifier%20Registration.pdf> An example of a more general case would be validation MIC codes. These are not defined in Orchestra, and are updated frequently. I am not convinced that Orchestra should be used to specify external codesets. It feels more like content validation logic that belongs a layer above Orchestra, at the application level. In this case, you can simply specify the field with a datatype rather than a codeset. This is the route we took for FINRA-CAT, where the atsOrderType field has type StringList. It might be useful to have a mechanism in Orchestra to identify the external codeset and specify where it resides. If we do go down the route of trying to internalise external codesets in Orchestra itself, then it we could add the ability to flag a codeset as "extensible". In this case, any (static) codes that are specified in Orchestra could still get validated, but validation would not fail if there is an unknown code that conforms to the type/pattern for that codeset. In this case, we would still need a mechanism to identify the external codeset and locate it. — Reply to this email directly, view it on GitHub <#200 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD35JL3NQQV6H37ZCO5KJ4DZANSHFAVCNFSM6AAAAABGXZR7XWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGMBUGMZTA> . You are receiving this because you are subscribed to this thread. <https://github.com/notifications/beacon/AD35JLZVFLKFOEEOIKMJ6DLZANSHFA5CNFSM6AAAAABGXZR7XWWGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQARX4QU.gif> Message ID: ***@***.***>