Release documentation (deephaven#3462)

Add release documentation to help improve the process for other release managers

.github/CODEOWNERS

@@ -16,6 +16,7 @@
 /LICENSE.md @chipkent @rcaudy
 /NOTICE.md @chipkent @rcaudy
 /README.md @chipkent @rcaudy @margaretkennedy
+/RELEASE.md @devinrsmith @jcferretti @stanbrub
 /TRIAGE.md @chipkent @rcaudy
 /licenses @chipkent @rcaudy
 /docker @devinrsmith @jcferretti @rcaudy

RELEASE.md

@@ -0,0 +1,275 @@
+# Release
+
+This document is primarily targeted towards Deephaven release managers.
+It is meant to contain the necessary instructions for executing a Deephaven release.
+
+Please be sure to read and understand this document in full before proceeding with a release.
+If you have any questions or concerns about any of the commands, please reach out.
+
+## Artifacts
+
+### deephaven-core jars
+The deephaven-core jars are the most important artifacts.
+They serve as the basis for the Deephaven server application and downstream consumers.
+They are released to [Maven Central](https://repo1.maven.org/maven2/io/deephaven/).
+
+### deephaven-core wheel
+The deephaven-core wheel is the most important artifact for Python integration purposes.
+It is released to [PyPi](https://pypi.org/project/deephaven-core/) and [GitHub releases](https://github.com/deephaven/deephaven-core/releases).
+
+### deephaven-server wheel
+The deephaven-server wheel is used to start the Deephaven server application from the context of python.
+It is released to [PyPi](https://pypi.org/project/deephaven-server/).
+Note: the deephaven-server wheel is large (roughly, the same size as the Deephaven server application), so it is _not_ re-attached to GitHub releases.
+
+### Deephaven server application
+The Deephaven server application is an important artifact for running the Deephaven server with the default classpath and configuration points.
+It also serves as the basis for Deephaven server images.
+It is released to [GitHub releases](https://github.com/deephaven/deephaven-core/releases) in both tar and zip formats.
+
+### Deephaven server images
+The Deephaven server images are important artifacts for easily running the Deephaven server in docker.
+The images are released to the GitHub container registry `ghcr.io/`.
+The image building logic is separated into the [deephaven/deephaven-server-docker](https://github.com/deephaven/deephaven-server-docker) repository.
+
+The regular images are the `server` (includes Python environment) and `server-slim` (does not include Python environment).
+* [ghcr.io/deephaven/server](https://github.com/deephaven/deephaven-core/pkgs/container/server)
+* [ghcr.io/deephaven/server-slim](https://github.com/deephaven/deephaven-core/pkgs/container/server-slim)
+
+The extended images are all extensions to the `server` image and include additional Python dependencies.
+* [ghcr.io/deephaven/server-all-ai](https://github.com/deephaven/deephaven-core/pkgs/container/server-all-ai)
+* [ghcr.io/deephaven/server-nltk](https://github.com/deephaven/deephaven-core/pkgs/container/server-nltk)
+* [ghcr.io/deephaven/server-pytorch](https://github.com/deephaven/deephaven-core/pkgs/container/server-pytorch)
+* [ghcr.io/deephaven/server-sklearn](https://github.com/deephaven/deephaven-core/pkgs/container/server-sklearn)
+* [ghcr.io/deephaven/server-tensorflow](https://github.com/deephaven/deephaven-core/pkgs/container/server-tensorflow)
+
+### Deephaven server SBOM
+The Deephaven server software bill-of-materials is an experimental artifact that some consumers may want.
+It may also be useful for release management purposes to ensure that no unexpected dependencies are change from release to release.
+It is released to [GitHub releases](https://github.com/deephaven/deephaven-core/releases) in the [syft](https://github.com/anchore/syft) json format.
+
+### Deephaven python client
+The Deephaven python client is released as the `pydeephaven` wheel at [PyPi](https://pypi.org/project/pydeephaven/).
+
+### Deephaven go client
+The Deephaven go client is released as a [Go package](https://pkg.go.dev/github.com/deephaven/deephaven-core/go).
+
+## Release process
+
+The majority of the release procedure is controlled through the [publish-ci.yml workflow](./.github/workflows/publish-ci.yml).
+It is kicked off by a push to a branch name that matches `release/v*`.
+Please familiarize yourself with the steps in that workflow.
+
+### 0. Poll Deephaven team
+
+Ensure you are proceeding with a known release, and there aren't any blockers.
+
+### 1. Repository prerequisites
+
+These release notes assume that the Deephaven repository `[email protected]:deephaven/deephaven-core.git` is referenced as the remote named `upstream`.
+Please ensure your local repository is setup as such, or that you replace any commands with the appropriately named remote:
+```shell
+$ git remote get-url upstream
+[email protected]:deephaven/deephaven-core.git
+```
+
+### 2. Create a local release branch
+
+Ensure you are up-to-date with `upstream/main`, or at the commit that you want to start a new release from.
+If you are unsure what commit to start from, please ask.
+Please double-check you are on the version you expect to be releasing.
+The releases have so far proceeded with `releave/vX.Y.Z`, where `X.Y.Z` is the version number (this isn't a technical requirement), please replace `X.Y.Z` with the appropriate version.
+We also separate out the release branch from `upstream/main` with an empty commit (this isn't a technical requirement).
+
+```shell
+$ git fetch upstream
+$ git checkout upstream/main
+$ ./gradlew printVersion -q
+$ git checkout -b release/vX.Y.Z
+$ git commit --allow-empty -m "Cut for X.Y.Z"
+```
+
+#### Procedure for patch releases
+
+For patch releases, typically the branch will be based off of the previous release tag, and not `upstream/main`, and the necessary patch fixes can be cherry-picked from the `upstream/main` branch.
+The patch release manager is also responsible for bumping the patch version numbers as appropriate.
+
+Here is an example going from `X.Y.0` to `X.Y.1`:
+
+```shell
+$ git fetch upstream
+$ git checkout vX.Y.0
+$ git checkout -b release/vX.Y.1
+$ git cherry-pick <...>
+# Edit files, updating from `X.Y.0` to `X.Y.1`, and git add them.
+# See https://github.com/deephaven/deephaven-core/issues/3466 for future improvements to this process.
+$ ...
+$ git commit -m "Bump to X.Y.1"
+```
+
+### 3. Push to upstream
+
+Triple-check things look correct, the release is a "GO", and then start the release process by pushing the release branch to upstream:
+
+```shell
+$ git show release/vX.Y.Z
+$ git push -u upstream release/vX.Y.Z
+```
+
+Note: release branches are _not_ typically merged back into `main`.
+
+### 4. Monitor release
+
+The release will proceed with [GitHub Actions](https://github.com/deephaven/deephaven-core/actions/workflows/publish-ci.yml).
+The specific action can be found based off of the name of the release branch: [?query=branch%3Arelease%2FvX.Y.Z](https://github.com/deephaven/deephaven-core/actions/workflows/publish-ci.yml?query=branch%3Arelease%2FvX.Y.Z).
+
+The "Publish" step creates the artifacts and publishes the jars to a [Maven Central staging repository](https://s01.oss.sonatype.org).
+
+The "Upload Applications" step uploads the Deephahven server application as a *temporary* GitHub action artifact.
+
+The "Publish deephaven-core to PyPi" uploads the deephaven-core wheel to [PyPi](https://pypi.org/project/deephaven-core/).
+
+The "Publish deephaven-server to PyPi" uploads the deephaven-server wheel to [PyPi](https://pypi.org/project/deephaven-server/).
+
+Once the workflow job is done, ensure all publication sources have the new artifacts.
+
+### 5. Download artifacts
+
+Once the full publish-ci.yml worflow is done, the Deephaven server application artifact can be downloaded from the GitHub Artifacts (located in the "Summary" tab of the action), and the deephaven-core wheel can be downloaded from PyPi.
+These are currently manual steps taken from the browser.
+
+There is potential in the future for QA-ing these artifacts above and beyond the integration testing that CI provides, as the release is not set in stone yet.
+
+### 6. Create SBOM
+
+This step is optional, but encouraged.
+If you are unable to use these tools, it is possible for somebody else to do this part.
+
+The following tools are used: [syft](https://github.com/anchore/syft), [cyclonedx](https://github.com/CycloneDX/cyclonedx-cli).
+
+```shell
+syft server-jetty-X.Y.Z.tar -o json > server-jetty-X.Y.Z.tar.syft.json
+```
+
+Compare differences:
+```shell
+syft convert server-jetty-A.B.C.tar.syft.json -o cyclonedx-json=/tmp/A.B.C.cyclonedx.json
+syft convert server-jetty-X.Y.Z.tar.syft.json -o cyclonedx-json=/tmp/X.Y.Z.cyclonedx.json
+cyclonedx diff /tmp/A.B.C.cyclonedx.json /tmp/X.Y.Z.cyclonedx.json --component-versions
+```
+
+Please post the difference to the Deephaven team to ensure there are no unexpected new dependencies, removed dependencies, or updated dependencies.
+
+### 7. Maven Central jars
+
+The jars are put into a [Maven Central Repository Manager](https://s01.oss.sonatype.org) staging repository.
+You'll need your own username and password to sign in (to ensure auditability).
+
+Arguably, the Maven Central jars are the most important artifacts - once they are officially released from the staging repository, they are released "forever".
+This is in contrast with PyPi where build numbers _could_ be incremented and docker where tags can always be re-written.
+
+If any late-breaking issues are found during the release process, but the Maven Central jars have not been released from staging, the release process can theoretically be re-done.
+
+When ready, the staging repository will need to be "Closed" and then "Released".
+Once the staging repository has been "Released", there is no going back.
+
+The jars will be visible after release at [https://repo1.maven.org/maven2/io/deephaven/](https://repo1.maven.org/maven2/io/deephaven/).
+Sometimes it takes a little bit of time for the jars to appear.
+
+### 8. Tag upstream
+
+The `vX.Y.Z` tag is primarily meant for an immutable reference point in the future.
+It does not kick off any additional jobs.
+The release should only be tagged _after_ the Maven Central staging repository has been "Released".
+
+```shell
+$ git tag -a -m "[Release] X.Y.Z" vX.Y.Z release/vX.Y.Z
+$ git show vX.Y.Z
+$ git push upstream vX.Y.Z
+```
+
+### 9. GitHub release
+
+Create a new [GitHub release](https://github.com/deephaven/deephaven-core/releases/new) and use the `vX.Y.Z` tag as reference.
+
+The convention is to have the Release title of the form `vX.Y.Z` and to autogenerate the release notes in comparison to the previous release tag. Question: should we always generate release notes based off of the previous minor release, instead of patch? Our git release workflow suggests we may want to do it always minor to minor.
+
+Upload the Deephaven server application, deephaven-core wheel, and SBOM artifacts.
+
+Hit the GitHub "Publish release" button.
+
+### 10. Deephaven go client
+
+The go client release consists of simply tagging and pushing to upstream:
+
+```shell
+$ git tag -a -m "[Release] Deephaven Go Client X.Y.Z" go/vX.Y.Z release/vX.Y.Z 
+$ git show go/vX.Y.Z
+$ git push upstream go/vX.Y.Z
+```
+
+### 11. Deephaven python client
+
+To be filled in by Jianfeng.
+
+### 12. Deephaven.io release
+
+The (non-public) [deephaven.io](https://github.com/deephaven/deephaven.io) `next` branch needs to be merged into `main`.
+Ping Margaret.
+
+### 13. Deephaven images
+
+Navigate to your checkout of [deephaven/deephaven-server-docker](https://github.com/deephaven/deephaven-server-docker).
+Like before, `upstream` is expected to be the proper remote repository - in this case, `[email protected]:deephaven/deephaven-server-docker.git`. 
+
+```shell
+$ git remote get-url upstream
+[email protected]:deephaven/deephaven-server-docker.git
+```
+
+Checkout the appropriate commit, create a release branch, update `DEEPHAVEN_VERSION` to the new version, and push.
+
+```shell
+$ git fetch upstream
+$ git checkout upstream/main
+# If doing a patch release, instead check out the appropriate <sha>
+# $ git checkout <sha>
+$ git checkout -b release/vX.Y.Z
+# edit files `server.hcl` and `server-slim.hcl` and update the `DEEPHAVEN_VERSION`
+$ git add server.hcl server-slim.hcl
+$ git commit -m "Bump DEEPHAVEN_VERSION to X.Y.Z"
+$ git push -u upstream release/vX.Y.Z
+```
+
+This will create the [Release CI](https://github.com/deephaven/deephaven-server-docker/actions/workflows/release-ci.yml) job.
+
+Monitor the release.
+If all is green, you should be able to test the new release:
+
+```shell
+$ docker run --rm --name deephaven -p 10000:10000 ghcr.io/deephaven/server:X.Y.Z
+```
+
+The docker image release process is more forgiving than releasing jar artifacts.
+If something goes wrong during this stage, it can easily be corrected.
+
+If applicable (typically, during a normal release), follow-up, and create a PR to merge `release/vX.Y.Z` into `main`.
+Patch releases will not typically be merged back into main.
+
+### 14. Let everybody know
+
+Ping team, ping community, ping friends - the latest Deephaven has been released!
+
+### 15. Clean-up
+
+The release branches serve a purpose for kicking off CI jobs, but aren't special in other ways.
+Sometime after a release, old release branches can be safely deleted.
+
+## External dependencies
+
+There are a few external dependencies that Deephaven manages, but they have separate release lifecycles that don't directly impact the release of the other artifacts mentioned earlier.
+* [deephaven/barrage](https://github.com/deephaven/barrage)
+* [deephaven/deephaven-csv](https://github.com/deephaven/deephaven-csv)
+* [deephaven/web-client-ui](https://github.com/deephaven/web-client-ui)
+* [jpy-consortium/jpy](https://github.com/jpy-consortium/jpy)
+

build.gradle

@@ -45,6 +45,12 @@ project.tasks.register('outputVersion') {task ->
     }
 }
 
+project.tasks.register('printVersion') {task ->
+    task.doLast {
+        println(project.version)
+    }
+}
+
 tasks.register('nightly') {
     it.group 'Deephaven lifecycle'
     it.description 'A lifecycle task that defines the nightly tasks to be run in CI, see .github/workflows/nighty-check-ci.yml'