Compiler Extensions

We're currently building a compiler extension generator that will take schemas for vendor extensions and generate compiler extensions that convert extensions found by gnostic into compiled protocol buffers.

Some goals:

• it would be good to have one compiler extension for multiple extension types. This would allow a vendor to generate a single compiler extension for all of the extensions that it supports.
• it might be useful to have a streaming interface between gnostic and compiler extensions. This would reduce the number of invocations of compiler extensions.
• As an example, we could provide a sample set of schemas that correspond to our x-google extensions and a scripted flow for generating a compiler extension for them.

Glossary:

• extension: a custom field in an OpenAPI model
• compiler extension: a binary that is invoked by gnostic and given an extension name and text data (perhaps in YAML) and returns a compiled Protocol Buffer representation of the extension.
• compiler extension generator: a tool that reads schemas that describe extensions and generates .proto files for the Protocol Buffer representations and source code for a compiler extension.

How to generate a compiler extension using the compiler extension generator

[TODO]: Fix the links once the code is merged into master.

NOTE: Samples and below steps are based on the forked branch https://github.com/guptasu/openapi-compiler/tree/forkProtoExtensions.

1. Define a json schema for the extension. Example : x-google.json and x-ibm.json

2. Invoke compiler extension generator with a file containing list of json schemas corresponding to the extension names that the generated extension compiler plugin will handle. This step will generate compiler extension which includes

• The protobuf message representation of the json schema
• Transformation code that will take an extension name and text data (as YAML) and returns a compiled Protocol Buffer representation of the extension. The created Protocol buffer is of type google.protobuf.any. The generated google.protobuf.any type's 'type_url` field will reference to the protocol buffer message that which is created from the given json schemas for the vendor extensions.

Example to generate compiler extension for the sample x-google.json:

```
GOOGLE_EXTENSION_SCHEMA="sample/x-google.json"
EXTENSION_OUT_DIR="github.com/googleapis/openapi-compiler/openapivendorext/sample/generated"

openapivendorextc $GOOGLE_EXTENSION_SCHEMA --out_dir_relative_to_gopath_src=$EXTENSION_OUT_DIR
# The generated files will be dropped inside $EXTENSION_OUT_DIR/openapi_extensions_google. The
# directory name 'openapi_extensions_google' always starts with 'openapi_extensions_' and the 
# 'google' term is derived from schema file name 'x-google.json' by extracting string between 'x-'
# and first 'dot'.
```

3. Install the generated compiler extension. But first run protoc` on the generated proto files.

   pushd <Directory where the proto files are generated>`
   `protoc --go_out=Mgoogle/protobuf/any.proto=github.com/golang/protobuf/ptypes/any:. *.proto`
   `go install`
   `pushd <Directory where the transformation code is generated>`
   `go install`
   

4. Invoking gnostic with a list of compiler extensions (these should be installed on your machine). gnostic will invoke compiler extensions for every vendor extension it encounters when parsing in a given OpenAPI spec. Compiler extensions are invoked in the order they are are passed into gnostic, and the invoking will stop as soon as any compiler extension handles the vendor extension.

Example to invoke gnostic for openapi_extensions_google:

gnostic --text_out=openapi_out.text --extension=google $GOPATH/src/github.com/googleapis/openapi-compiler/test/shared_parameters.json
# Since `--extension=google` is passed, gnostic will invoke the compiler extension with
# the name 'openapi_extensions_google'

3. TODO : Document and write samples on how to use consume the OpenAPI model that has these google.protobuf.Any types for the vendor extension.

Interesting code references : Start by exploring sample extension plugin installation script, input OpenAPI spec with vendor extension lines 10~16 and Generated OpenAPI Model lines 131-176