Support for binary protocols in Orchestra - Technical Proposal Draft.

[francescoloconte]

We would like to propose an enhancement to Orchestra to support binary protocols. Specifically, to support encoding (i.e., serialization) in binary formats, as in Layer 6 of the OSI 7-Layers model (the Presentation layer):

• Layer 7 (Application)
• Layer 6 (Presentation)
• Layer 5 (Session)
• Layer 4 (Transport)

Orchestra addresses Layer 7, i.e. the Application layer, by representing the structure and semantic of the API (Messages, Components, Groups, Fields, etc). Because of its FIX heritage, it does implicitly supports encoding in FIX tag/value format (i.e., it provides enough information for such encoding).

However, when encoding in other formats, like binary formats implemented by exchanges, more information is needed. For example, which character to use to pad fields, fields' offsets, padding justification, support for field-maps, etc.

Below is a draft proposal to improve Orchestra's model elements with new attributes to support binary encoding.

Note: the proposal is not finished and we publish it here to receive feedback and contributions.

Binary Protocols Support Proposals

Byte Order

Issue

Binary protocols may choose either big-endian or little-endian byte order for specific reasons. It's important for message protocols to clearly state its byte order in an Orchestra file. In FIX SBE, for instance, the byte order is globally specified in a message schema.

Proposal

Since datatypes are closely related to the encoding protocol, we recommend adding optional byteOrder attribute to the mappedDatatype element. It's common for binary protocols to have the same byte order for all datatypes, so the attribute should be also added to the datatypes element as a global setting.

Valid values should be bigEndian and littleEndian. The absence of the attribute would mean byte order is unspecified (e.g. for character types).

Example

<datatypes byteOrder="littleEndian">
	<datatype name="int16">
		<mappedDatatype standard="ISO11404" base="integer" ... />
	</datatype>
	<datatype name="price">
		<mappedDatatype standard="ISO11404" base="scaled" byteOrder="bigEndian" ... />
	</datatype>
	...
</datatypes>

See also:

• https://github.com/real-logic/simple-binary-encoding/wiki/FIX-SBE-XML-Primer

Null Value Indication

Issue

Many binary protocols are fixed-length, meaning every field must be included in the transmission, even if it's optional and not used in that specific message. Those protocols define a special value for each datatype, usually called null value. This null value indicates that an optional field is not being used. When the encoder or decoder encounters this null value, it treats the field as if it is not set.

Orchestra standard doesn't provide a way to map a datatype with its null value.

Proposal

We recommend adding nullValue attribute to the <mappedDatatype> element.

Example

<datatype name="u8">
	<mappedDatatype standard="ISO11404" base="integer" minInclusive="0" maxInclusive="255" nullValue="0xFF"/>
	<annotation>
		<documentation>8 bit Unsigned Integer, 0xFF in case of empty field</fixr:documentation>
	</annotation>
</datatype>

String Termination and Padding Characters

Issue

Most binary encodings use fixed-length fields for alphanumeric data, i.e. for string values. If the actual value doesn't occupy the field length entirely, it
could either be padded with a particular character on the beginning or the end (padded string), or terminated with NUL character (null-terminated string). In fact, a null-terminated string can be considered as a special case of padding with NUL control characters.

Protocols have different policies on including the null-terminator in the string length. This means that in some cases (e.g. in SBE, NYSE Pillar) if the actual length of the value fits the entire length of the field, then it's acceptable that the value doesn't include a null-terminator character. In other cases (e.g. in HKEX OCG) a null-terminator is required to be present in the field's value, making one byte less space available for the actual string.

Hence, there is a requirement to define a padding side and character, and a null-termination requirement flag for fixed-length string datatypes.

Proposal

We recommend adding following optional attributes to the mappedDatatype element:

• The paddingSide attribute which could be set to left or right.
• The paddingChar attribute which could be set to a single character or to a special value NUL which represents null control character (since this character cannot be used in XML directly).
• The nullTerminated attribute which could be set to true or false. The following rules apply when the attribute is set to true:
  • The field's value must be terminated by NUL character, even if the value is empty.
  • The maximum length of a string which the field could accommodate would be one character less than its declared length.
  • In case padding attributes are also set, the value on the wire would be terminated by a null character, and then the remaining space would be filled with padding characters.
  • If paddingSide="left" then null-terminator character must be present at the left side of a string.
  • If no padding attributes specified, the string would be padded with null characters on the right after the terminator character.

Example

<datatype name="charArray" kind="array">
	<mappedDatatype standard="ISO11404" base="array" element="character" paddingSide="right" paddingChar=" "/>
	<annotation>
		<documentation>Fixed length string padded on the right with spaces.</documentation>
	</annotation>
</datatype>
<datatype name="zcharArray" kind="array">
	<mappedDatatype standard="ISO11404" base="array" element="character" paddingSide="right" paddingChar="NUL"/>
	<annotation>
		<documentation>Fixed length string padded on the right with NUL (ASCII 0) characters.</documentation>
	</annotation>
</datatype>
<datatype name="zcharArrayStrict" kind="array">
	<mappedDatatype standard="ISO11404" base="array" element="character" nullTerminated="true"/>
	<annotation>
		<documentation>Fixed length null-terminated string. NUL (ASCII 0) character is mandatory.</documentation>
	</annotation>
</datatype>

Field Presence Map

Issue

Some protocols like HKEX OCG have a feature called Field Presence Maps, which allows to skip transmission of unset optional fields.
It's a special type of field which contains a bitmap describing the presence of subsequent fields in the actual message instance. The presence map provides an efficient way to check which optional fields are present in a received message. Bits corresponding to required fields are always set to 1.

Those types may be represented as special datatypes mapped to bitsets. In HKEX OCG, for example, a field of such datatype is included in a message header and also it precedes every repeating group instance:

<datatype name="Bitmap">
	<mappedDatatype standard="ISO11404" base="bitstring" element="bit">
		<annotation>
			<documentation>Field length is in number of octets.</documentation>
		</annotation>
	</mappedDatatype>
</datatype>

<fields>
	<field id="42" name="BodyFieldsPresenceMap" type="Bitmap" implLength="32">
		<annotation>
			<documentation>Indicates the list of fields that would be present in a message body immediately after this field. Each bit denotes corresponding field's presence.</documentation>
		</annotation>
	</field>
	<field id="101" name="NoEntitlementsBodyFieldPresenceMap" type="Bitmap" implLength="2"/>
</fields>

<groups>
	<group id="100" name="Entitlements">
		<numInGroup id="100" name="NoEntitlements"/>
		<!-- Presence bitmap for this group is the field with id 101 -->
		<fieldRef id="101" name="NoEntitlementsBodyFieldPresenceMap"/>
		<fieldRef id="102" name="EntitlementType"/>
		<fieldRef id="103" name="EntitlementIndicator"/>
		<fieldRef id="104" name="EntitlementID"/>
	</group>
</groups>

<components>
	<component id="1" name="Header">
		<!-- ... -->
		<fieldRef id="42" name="HeaderFieldPresenceMap"/>
	</component>
	<!-- Presence bitmap for this component is the field with id 42 -->
	<component id="10" name="TestRequestBody">
		<fieldRef id="43" name="FirstField"/>
		<fieldRef id="44" name="SecondField"/>
		<groupRef id="100" name="NoEntitlements"/>
	</component>
</components>

<messages>
	<message name="TestRequest">
		<structure>
			<componentRef id="1" name="Header"/>
			<componentRef id="10" name="TestRequestBody"/>
			<componentRef id="2" name="Trailer"/>
		</structure>
	</message>
</messages>

There should be a standard approach to associate the bitmap field with the components and groups it relates to, in the same way a data field may be associated with its length field using the lengthId attribute.

Proposal

We recommend adding optional presenceMapId attribute to container elements like message/structure, component, and group. This attribute must reference a field which contains presence flag for every direct member of the container element. This approach is like the one with the lengthId and discriminatorId attributes of a field definition, which are already a part of the standard.

Example

<components>
	<component id="1" name="Header">
		<fieldRef id="42" name="HeaderFieldPresenceMap"/>
	</component>
	<component id="10" name="TestRequestBody" presenceMapId="42">
		<fieldRef id="43" name="FirstField"/>
		<fieldRef id="44" name="SecondField"/>
		<groupRef id="100" name="NoEntitlements"/>
	</component>
</components>

<groups>
	<group id="100" name="Entitlements" presenceMapId="101">
		<numInGroup id="100" name="NoEntitlements"/>

		<fieldRef id="101" name="NoEntitlementsBodyFieldPresenceMap"/>
		<fieldRef id="102" name="EntitlementType"/>
		<fieldRef id="103" name="EntitlementIndicator"/>
		<fieldRef id="104" name="EntitlementID"/>
	</group>
</groups>

[francescoloconte]

We removed from the proposal the introduction of the offset attribute. We understood that offsets for fields in binary protocols are calculated from the size of the preceding fields, and they are not specified as absolute by the protocol. Where it is necessary to put empty spaces between fields, we noted that existing exchange binary protocols define "empty" fields for this purpose. This approach has the advantages that the protocol specifications do not need to calculate offsets for fields (based on all preceding fields), which is prone to error. Although we appreciate that SBE v1.0 considers offsets to be optionally available in some cases, we pose that this approach may not be optimal and instead, it should be perhaps changed to use the "empty fields" approach to remove the need for offsets altogether. We equally rejected the idea of providing an offset attribute "just in case someone wants to use it", because we aim to keep the standard minimal and simple.

[francescoloconte]

I have updated the main proposal to a new version, with more refinements.