-
Notifications
You must be signed in to change notification settings - Fork 1.2k
feat(api): oasdiff OpenAI openAPI spec against ours #3529
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -65,6 +65,15 @@ jobs: | |
ref: ${{ github.event.pull_request.base.ref }} | ||
path: 'base' | ||
|
||
|
||
# Cache openai spec | ||
- name: Cache openai openapi spec | ||
id: cache-openapi | ||
uses: actions/cache@0400d5f644dc74513175e3cd8d07132dd4860809 | ||
with: | ||
path: ~/openai-openapi | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I believe the cache action by default has access only to a runners home dir? I could be wrong but that is where I have been storing and writing the cached binaries+files to/from |
||
key: openai-openapi.yml | ||
|
||
# Cache oasdiff to avoid checksum failures and speed up builds | ||
- name: Cache oasdiff | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we should probably also support a force no-cache conformance check as well - we should probably run it at the time of release. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree, but should I make this a separate action so it can be triggered by maintainers manually? |
||
if: steps.skip-check.outputs.skip != 'true' | ||
|
@@ -81,6 +90,12 @@ jobs: | |
curl -fsSL https://raw.githubusercontent.com/oasdiff/oasdiff/main/install.sh | sh | ||
cp /usr/local/bin/oasdiff ~/oasdiff | ||
|
||
- name: Get openai openapi spec | ||
if: steps.cache-openapi.outputs.cache-hit != 'true' | ||
run: | | ||
mkdir -p ~/openai-openapi | ||
curl -L https://app.stainless.com/api/spec/documented/openai/openapi.documented.yml -o ~/openai-openapi/openai-openapi.yml | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. do we not want to pin the version of the openai api? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. sure, I can pin it. Do we have a version in mind we want to use? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think the pinning is important, what's the latest version so we can get started with it? |
||
|
||
# Setup cached oasdiff | ||
- name: Setup cached oasdiff | ||
if: steps.skip-check.outputs.skip != 'true' && steps.cache-oasdiff.outputs.cache-hit == 'true' | ||
|
@@ -137,6 +152,25 @@ jobs: | |
run: | | ||
oasdiff breaking --fail-on ERR $BASE_SPEC $CURRENT_SPEC --match-path '^/v1/' | ||
|
||
# Run oasdiff to detect breaking changes in the API specification when compared to the OpenAI openAPI spec | ||
- name: Run OpenAPI Breaking Change Diff Against OpenAI API | ||
if: steps.skip-check.outputs.skip != 'true' | ||
continue-on-error: true | ||
shell: bash | ||
run: | | ||
OPENAI_SPEC=~/openai-openapi/openai-openapi.yml | ||
LOCAL_SPEC=docs/static/llama-stack-spec.yaml | ||
|
||
# Compare OpenAI spec against Llama Stack spec | ||
# Strip /v1 prefix from our spec to match OpenAI's paths | ||
# Filter out api-path-removed errors for OpenAI routes we don't implement | ||
oasdiff breaking --fail-on ERR \ | ||
"$OPENAI_SPEC" \ | ||
"$LOCAL_SPEC" \ | ||
--strip-prefix-revision "/v1" 2>&1 | \ | ||
grep -v "api-path-removed-without-deprecation" | \ | ||
grep -v "api-removed-without-deprecation" || true | ||
|
||
# Report when test is skipped | ||
- name: Report skip reason | ||
if: steps.skip-check.outputs.skip == 'true' | ||
|
Uh oh!
There was an error while loading. Please reload this page.