Add example conformance report tool (project-chip#36913)

* Add example conformance report tool

* Integrate report into app engine

* Restyled by isort

* fix lint errors

* Add more targets

* Restyled by isort

* Add python build to startup script

* Update readme for the compute_engine VM

* fix spelling

* spelling

* Restyled by prettier-markdown

* limit ninja_jobs to cpu_count() - 2

* Fix typo; group cmds in the script

* Improved comments on find_executables(dirs)

---------

Co-authored-by: Restyled.io <[email protected]>

integrations/appengine/webapp_config.yaml

@@ -3,6 +3,9 @@ handlers:
     - url: /
       static_files: html/index.html
       upload: html/index.html
+    - url: /conformance/
+      static_files: html/conformance_report.html
+      upload: html/conformance_report.html
     - url: /(.*)
       static_files: html/\1
       upload: html/(.*)

integrations/compute_engine/README.md

@@ -1,29 +1,33 @@
-## Startup Script of Compute Engine
-
-A startup script is a file that contains commands that run when a virtual
-machine instance boots. Compute Engine provides support for running startup
-scripts on Linux and Windows virtual machines.
-
-### Create a virtual machine instance using startup script
-
-The `startup-script.sh` could be used as the startup script of a virtual machine
-instance which run Matter coverage report and publish the result via an App
-Engine service.
-
-You can create a virtual machine instance by using the gcloud compute instances
-create command with the `--metadata-from-file` flag.
-
-```
-gcloud compute instances create VM_NAME \
-  --image-project=PROJECT_NAME \
-  --image-family=ubuntu-22.04 \
-  --metadata-from-file=startup-script=FILE_PATH
-```
-
-Replace the following:
-
-`PROJECT_NAME`: the name of the project host the virtual machine instance
-
-`VM_NAME`: the name of the virtual machine instance
-
-`FILE_PATH`: the relative path to the startup script file
+## Google Cloud Compute Engine
+
+We have setup a Virtual Machine on
+[Google Cloud](https://cloud.google.com/products/compute) to generate both the
+[Matter SDK coverage report](https://matter-build-automation.ue.r.appspot.com)
+and the
+[Matter SDK Conformance report](https://matter-build-automation.ue.r.appspot.com/conformance_report.html).
+
+### The Matter SDK Virtual Machine and the "startup-script.sh"
+
+We created a VM named `matter-build-coverage`. The machine configuration is
+located
+[here](https://pantheon.corp.google.com/compute/instancesDetail/zones/us-central1-a/instances/matter-build-coverage?inv=1&invt=AbnAfg&project=matter-build-automation).
+Reach out to Google team members if you need to make changes to this VM.
+
+This virtual machine is scheduled to run daily, starting at 11:45PM and stopping
+at 2am. During boot, the machine runs the `startup-script.sh`.
+
+The `startup-script.sh` script contains commands to checkout the SDK repository
+and create both the SDK coverage report and conformance report. The startup
+script uses `scripts/build_coverage.sh` to generate the coverage report and
+`scripts/examples/conformance_report.py` to generate the conformance report. The
+resulting HTML files are published via an App Engine service and available here
+([coverage report](https://matter-build-automation.ue.r.appspot.com/),
+[conformance report](https://matter-build-automation.ue.r.appspot.com/conformance_report.html)).
+
+### Making Changes to "startup-script.sh"
+
+If you make changes to `startup-script.sh`, make sure you go to the
+[VM configuration](https://pantheon.corp.google.com/compute/instancesDetail/zones/us-central1-a/instances/matter-build-coverage?inv=1&invt=AbnAfg&project=matter-build-automation),
+click `edit` and update the startup script in the `Automation` text box, to
+reflect your changes. The script in the Matter SDK repo is just a copy of the
+configuration in the VM.

integrations/compute_engine/startup-script.sh

@@ -16,11 +16,23 @@
 # limitations under the License.
 #
 
+set -x
+
 cd /tmp
 rm -rf connectedhomeip
 git clone --recurse-submodules https://github.com/project-chip/connectedhomeip.git
 cd connectedhomeip
+
+# Generate Coverage Report
 ./scripts/build_coverage.sh 2>&1 | tee /tmp/matter_build.log
+
+# Generate Conformance Report
+source scripts/activate.sh
+./scripts/build_python.sh -i out/python_env
+python3 -u scripts/examples/conformance_report.py
+cp /tmp/conformance_report/conformance_report.html out/coverage/coverage/html
+
+# Upload
 cd out/coverage/coverage
 gcloud app deploy webapp_config.yaml 2>&1 | tee /tmp/matter_publish.log
 versions=$(gcloud app versions list \