all: ensure that Go toolchain meets Apple’s notarization requirements

[artyom]

What version of Go are you using (go version)?

$ go version
go version go1.13.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/artyom/Library/Caches/go-build"
GOENV="/Users/artyom/Library/Application Support/go/env"
GOEXE=""
GOFLAGS="-ldflags=-w -trimpath"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/artyom/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/artyom/Library/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/artyom/Library/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/artyom/Repositories/artyom/rex/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lb/3rk8rqs53czgb4v35w_342xc0000gn/T/go-build499526677=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Installed go from https://dl.google.com/go/go1.13.3.darwin-amd64.tar.gz and then tried to use it to build project. This is on macOS Catalina 10.15:

Darwin MacBook-Air.local 19.0.0 Darwin Kernel Version 19.0.0: Wed Sep 25 20:18:50 PDT 2019; root:xnu-6153.11.26~2/RELEASE_X86_64 x86_64

What did you expect to see?

Go works as usual.

What did you see instead?

On first go call I'm presented with OS pop-up that "macOS cannot verify that this app is free from malware", with options either to discard this window (the process is killed then) or move the binary to trash. After whitelisting this binary via Preferences → Security & Privacy → General → Allow Anyway, go can be called fine.

Next I tried to build project with go install and got the came pop-up for "compile" tool.

[artyom]

To be able to run updated Go 1.13.3 and compile my project with it, I had to whitelist so far these binaries:

• go
• compile
• asm
• cgo
• link

Whitelisting each takes the following:

1. presented with os pop-up about binary not being verified;
2. click cancel, at this moment binary is killed;
3. go to Preferences → Security & Privacy → General, click "Allow Anyway" button;
4. try running the same command again;
5. presented with os pop-up about binary not being verified, but now with another option to run binary anyway;
6. click "open" button on that pop-up.

[bradfitz]

Good times.

/cc @golang/osp-team

[bradfitz]

Related: #34748 (we need a Catalina builder)

[networkimprov]

Discussion on Hacker News: https://news.ycombinator.com/item?id=21179970

Also, Catalina is reported to be a disastrous upgrade.

[andybons]

I’ll take a look at this.

[dmitshur]

I can reproduce on my personal MBP with macOS 10.15 (19A602) when installing from the .pkg installer too, however it's a different (less intrusive) experience.

The "macOS cannot verify that this app is free from malware" prompt appears when first running the installer. After accepting and installing, Go binaries such as go, compile, do not produce the same prompt. Also, re-running the .pkg installer doesn't ask again.

[andybons]

In the process of submitting the installer to Apple's notary service, but the process is very slow due to a service issue on Apple's side and our internal tooling keeps timing out. Diagnosing this has been difficult.

It may be that we will need to enable the Hardened Runtime which likely requires a newer version of Xcode that we're using to build releases and an audit of whatever entitlements are required for our binaries to run.

@aclements can you take a look at the link above and let me know which you think we'll need (if any)?

[aclements]

@andybons, I don't think we need any of the runtime exceptions from that page. The only one that we may even potentially want is "Disable Library Validation Entitlement", which would let users use the the plugin package without signing their plugins, but it seems like developers probably should sign their plugins.

[vp2177]

The (.pkg) installer doesn't even start for me. Is that the same issue, or unrelated?

"go1.13.3.darwin-amd64.pkg can't be opened because Apple cannot check it for malicious software."

Meanwhile, the older Go version I had installed from before the OS X Catalina update, continues to work fine, no warning dialogs.

[andybons]

@vp2177 it’s the same issue.

[andybons]

Good news: according to Apple, we don’t have to enable the hardened runtime until January 2020. That said, we’ll likely have to start signing the binaries themselves (not just the installer as we do now), so this issue should stay open even if we fix the problem in the short term via notarization of the installer.