Skip to content

Adds Benchmark module to evaluate the SDK's first run performance #1489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 45 commits into from
Dec 2, 2022

Conversation

vitorpamplona
Copy link
Collaborator

@vitorpamplona vitorpamplona commented Jul 6, 2022

Fixes #1474, #825 and #1378, potential merger with #1414

Description
Adds a simple benchmark module to run performance tests on the engine and workflow. The module is focused on the first-time runs, which drive most of the performance issues we are having. The tests must be run in order.

I am not convinced that yet another module needs to be added, but given our struggles with performance, it might be a good short-term investment.

Current results in nanoseconds (seconds,milli,micro,nano) on an older Moto Z Play XT1635-02: (Android8, 3GB RAM, Octa-core 2.0 GHz) with Geekbench: 147 single core and 544 multi-core. These are single runs, but first-time runs (no caching):

   49,577,870   ns    trace    AJacksonMapperBenchmark.loadJsonMapper
2,258,560,068   ns    trace    BFhirContextLoaderBenchmark.loadR4
2,642,449,743   ns    trace    BFhirContextLoaderBenchmark.loadR5
1,665,396,000   ns    trace    BFhirContextLoaderBenchmark.loadDstu3
6,920,873,294   ns    trace    CCqlEngineFhirContextLoaderBenchmark.loadDstu2FhirModelResolver
3,897,679,346   ns    trace    CCqlEngineFhirContextLoaderBenchmark.loadR4FhirModelResolver
2,907,996,227   ns    trace    CCqlEngineFhirContextLoaderBenchmark.loadDstu3FhirModelResolver
  630,341,938   ns    trace    DFhirJsonParserBenchmark.parseLightFhirLibrary
1,258,804,813   ns    trace    DFhirJsonParserBenchmark.parseLightFhirBundle
2,422,628,522   ns    trace    EElmJsonLibraryLoaderBenchmark.parseImmunicyCheckCqlFromFhirLibrary
2,481,479,986   ns    trace    EElmJsonLibraryLoaderBenchmark.parseFhirHelpersCqlFromFhirLibrary
  116,937,355   ns    trace    FEngineDatabaseBenchmark.createAndGet
8,807,674,783   ns    trace    GCqlEvaluatorBenchmark.evaluatesLibrary (evaluation only)

Pixel 6 GB7N6 (Android 12, 8GB RAM, Octa-core (2x2.80 GHz Cortex-X1 & 2x2.25 GHz Cortex-A76 & 4x1.80 GHz Cortex-A55) with Geekbench: 1036 single core and 2720 multi-core:

   21,242,350   ns    trace    AJacksonMapperBenchmark.loadJsonMapper
  335,387,573   ns    trace    BFhirContextLoaderBenchmark.loadR4
  356,097,657   ns    trace    BFhirContextLoaderBenchmark.loadR5
  271,354,411   ns    trace    BFhirContextLoaderBenchmark.loadDstu3
  693,789,348   ns    trace    CCqlEngineFhirContextLoaderBenchmark.loadDstu2FhirModelResolver
  500,146,485   ns    trace    CCqlEngineFhirContextLoaderBenchmark.loadR4FhirModelResolver
  366,984,538   ns    trace    CCqlEngineFhirContextLoaderBenchmark.loadDstu3FhirModelResolver
  152,651,122   ns    trace    DFhirJsonParserBenchmark.parseLightFhirLibrary
  208,492,962   ns    trace    DFhirJsonParserBenchmark.parseLightFhirBundle
  297,588,461   ns    trace    EElmJsonLibraryLoaderBenchmark.parseImmunicyCheckCqlFromFhirLibrary
  276,356,160   ns    trace    EElmJsonLibraryLoaderBenchmark.parseFhirHelpersCqlFromFhirLibrary
   32,813,435   ns    trace    FEngineDatabaseBenchmark.createAndGet
1,018,112,914   ns    trace    GCqlEvaluatorBenchmark.evaluatesLibrary (evaluation only)

Alternative(s) considered
I tried building an app and a macro module, but I think this micromodule is a better approach.

Type
Testing

Screenshots (if applicable)

Checklist

  • I have read and acknowledged the Code of conduct.
  • I have read the Contributing page.
  • I have signed the Google Individual CLA, or I am covered by my company's Corporate CLA.
  • I have discussed my proposed solution with code owners in the linked issue(s) and we have agreed upon the general approach.
  • I have run ./gradlew spotlessApply and ./gradlew spotlessCheck to check my code follows the style guide of this project.
  • I have run ./gradlew check and ./gradlew connectedCheck to test my changes locally.
  • I have built and run the demo app(s) to verify my change fixes the issue and/or does not break the demo app(s).

Copy link
Collaborator

@jingtang10 jingtang10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add to the description how to run the benchmarking?

@jingtang10 jingtang10 enabled auto-merge (squash) December 2, 2022 16:58
@jingtang10 jingtang10 merged commit 8acbeb3 into google:master Dec 2, 2022
This was referenced Dec 2, 2022
@jingtang10 jingtang10 linked an issue Dec 2, 2022 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 Medium priority issue
Projects
Status: Complete
Development

Successfully merging this pull request may close these issues.

Extraction performance - Initial testing and benchmarking of extraction work Setup benchmark module
4 participants