proposal: cmd/go/v2: prohibit in-package tests from extending a package's API

[rogpeppe]

Currently in-package tests are free to add exported identifiers to the package that they are testing.
Specifically an in-package test:

• can define new variables
• can define new types
• can implement methods on non-test types

This makes tooling harder, because any given package can have several possible variants,
depending on which package's tests are being run.

Consider the following example:

-- go.mod --
module example

-- a/a.go --
package a

import (
	"fmt"

	"example/b"
)

var A fmt.Stringer = b.B{}

-- b/b.go --
package b

type B struct{}

-- b/b1_test.go --
package b

func (B) String() string { return "b" }

-- b/b2_test.go --
package b_test

import (
	"testing"

	"example/a"
)

func Test(t *testing.T) {
	if got, want := a.A.String(), "b"; got != want {
		t.Fatalf("got %q want %q", got, want)
	}
}

Running go test example/b succeeds because when b is being tested, the internal tests extended the B type so it implements fmt.Stringer. But example/a won't actually compile on its own because B doesn't implement fmt.Stringer without the test code added by the internal tests in example/b!

It's as if tests exist in a closely related but different version of the universe in which every package might not be quite the same as its usual version. And that version is potentially different for every package.

This "parallel universe" property of testing makes Go types harder to reason about and the Go tooling less efficient, because it's not possible to type-check a package once and for all in the general case.

Where does this problem come from?

Currently in Go, test code can be internal to a package, extending that package for the duration of the test and allowing the test code access to unexported identifiers within the package, or it can be external, with access to exported identifiers only. A hybrid approach is also possible, where tests are largely external, but some package-internal test code, often in an export_test.go file, is used to provide access to selected parts of the internal API.

It's the final case that is problematic, because external test code is free to depend on other packages which themselves depend on the package being tested which has been extended with extra test code.

All three approaches are common in existing Go code.

I believe that any solution to this issue should continue to support almost all existing code (including the possibility of automatic code migration with gofix), otherwise substantial parts of the ecosystem will not be able to move forward with new Go features.

Proposed solution

I propose that it should not be possible to write tests in the same package as that being tested. That is, all tests must be in the external test package (e.g. foo_test to test package foo).

A test file can use a special form of import statement to request access to unexported identifiers from the package being tested. If they use this special form, code within that file (only) may access unexported identifiers within the package without compiler errors.

A possible (straw man) spelling for the syntax is:

import "."

That is, import the package in the current directory.

This form is currently invalid because relative import paths are not allowed, and "." is not a valid package name by itself, so wouldn't overlap with existing import path syntax. As there is only one package that can be imported in this way, there is no ambiguity problem. It also satisfies issue 29036, as the imported package name can be automatically derived from the current file's package identifier by stripping the _test suffix.

Another possibility might be to add an extra keyword, recognized only within an import block; for example:

import "mypackagepath" internal

Whatever the form for the special import syntax, this solution seems to solve the initial problem. It allows both all-internal tests (always use import "."), all-external tests (never use import ".") and hybrid tests ("internal" test code can define selected functionality to be used by external tests, similarly to how export_test.go files are used today). It also reduces the overall complexity of the way that tests work in Go.

I believe that it should also be possible to automatically convert almost all existing code to the new rules. Some care would need to be taken to rename symbols that appear in both internal and external test code, but that's not an hard issue to solve. Perhaps that issue is rare enough that manual intervention could be required. More investigation is needed to see how common it is.

As an example, some test code that exposes selected functionality to external tests might look like this:

-- b/b.go --
package b

type B struct {
	internalField string
}

func GetBValue() B {
	return B{
		internalField: "b",
	}
}
-- b/b1_test.go --
package b_test
import "."		// allow access to unexported identifiers within this file.

func BInternalField(x b.B) string {
	return x.internalField
}
-- b/b2_test.go --
package b_test

import (
	"testing"

	"example/b"
)

func Test(t *testing.T) {
	x := b.GetBValue()
	if got, want := BInternalField(x), "b"; got != want {
		t.Fatalf("got %q want %q", got, want)
	}
}

Other possible solutions

We could disallow all tests that rely on direct access to unexported identifiers i.e. allow external tests only. This is an attractive option for some, but the change would be very disruptive for much code. I do not believe that it would be possible to migrate existing internal tests automatically, so I think this possibility is excluded.

We could continue to allow both internal and external tests, but treat internal test code as being in its own package, with access allowed to the tested package's unexported identifiers (and all symbols from the package available in global scope), but otherwise separate from it. External tests could use some kind of special import syntax (for example import "thispackage@test") to access symbols defined in the internal tests. This was my original thought, but the model seems more complex than necessary - we have an extra pseudo-package for tests and a special import path syntax.

We could prohibit internal test code from defining methods on types defined by the testing package. This solves some of the more obvious problems with the current situation, but the "parallel universe" issue is still present, and tooling probably would not become significantly simpler.

[Merovius]

How are tests any different from build tags in general here? If this is a problem, shouldn't we also disallow having different APIs under different sets of build tags? How would syscall work under those assumptions? And how would you check for this (AIUI this would require solving SAT ;-) )?

i.e. I agree that you can solve the test-portion of this, but I don't think you ever can get to a point where you have any guarantees of this kind for tooling.

[rogpeppe]

How are tests any different from build tags in general here?

Tests are different from build tags in general because there are no build tags for tests (deliberately).

With a given set of build tags, anything might change, but that's inevitable, given the way that build tags work. The test thing just adds another dimension to that. Even if you do know the set of build tags, you can't precompile a given package.

Consider this: with a given build environment (including tags), it is currently not possible to have a simple map from each package to type-checked information on that package.

With this proposal, that would be possible, making it easier to reason about and write tools that use that information.

[Merovius]

Tests are different from build tags in general because there are no build tags for tests (deliberately).

I was referring to the lined out problems of the current situations - tooling doesn't know what is and isn't part of the API because it depends on how you call the Go tool. If your tool relies on that knowledge, and that knowledge is principally unavailable because of build tags, how does this proposal actually solve anything? Your tool still won't be able to provide any better correctness guarantees.

it is currently not possible to have a simple map from each package to type-checked information on that package. With this proposal, that would be possible, making it easier to reason about and write tools that use that information.

APIs defined in test-files can not be used by importers (as you point out, test is not a build tag). Why would you need such a map? And how is a map keyed by "import path + test(yes/no) + <build_tags>" significantly worse than a map keyed by "import path + <build_tags>" for that purpose?

I honestly don't understand the problem you are trying to solve here. Or rather: I understand the problem and I suffered from it before, but I don't see why tests specifically should be addressed, while I consider the build-tag problems to be far more severe.

(BTW: A way to address the build-tag problem would be to disallow exported declarations in files guarded by build tags. But I'm not totally sure you'd want to)

[natefinch]

I'd definitely be in favor of disallowing exported identifiers in test files.

For build tags, I wish that were possible, because it seems like a bad design to have different APIs for different platforms, but at the same time, then you have APIs that essentially lie about what they support for some platforms.

I wish there were a solution for godoc that could merge all APIs for a package and indicate which are platform-specific.

[natefinch]

I do like the idea of keeping a package sealed off from testing code, so that the package is the package, and there's none of this kind of wacky thing where when you test, now the package has different code in it. Adding code to the code that you're testing at test time seems like a fantastically bad idea in general. By definition, then you're not actually testing the production code.

At the same time, I find internal tests to be immensely useful in my own work, providing fast, reliable, focused assurances that the code does what I expect it to do without having to change the API to do that testing.

Like Axel, I'm not sure about the real benefit here. It sounds like it's intended to make tooling faster and simpler, which I am all for, but are tests really a problem?

I'd like to hear more about the problem this is solving and what the benefits are.

[rogpeppe]

At the same time, I find internal tests to be immensely useful in my own work, providing fast, reliable, focused assurances that the code does what I expect it to do without having to change the API to do that testing.

This proposal still allows you to write tests that access internal package details. I'm not sure what you're driving at here.

[Merovius]

Additions:

APIs defined in test-files can not be used by importers

There is one exception to this: foo_test can import foo and then use exported APIs created in _test.go files of foo - which is the currently recommended way to access unexported APIs from test-packages.

Adding code to the code that you're testing at test time seems like a fantastically bad idea in general. By definition, then you're not actually testing the production code.

FWIW, a case where I actually added exported methods in tests to existing types is convenience helpers for constructing nested data structures. So, for example, I have a binary tree implementation and implement fmt.Stringer and fmt.Scanner to work with ((a b) c) or the like for readability in tests. The methods have to be exported to be callable for fmt, but I don't want to commit to the format and make it part of my actual API.

Lastly: If we don't want to reduce the utility of testing (i.e. still allow accessing internal identifiers), ISTM that this would require a language change - the prohibition of accessing unexported identifiers of foreign packages is part of the language and the compiler would need to both generate the necessary symbols when compiling the package-under-test and know to allow accessing unexported identifiers in certain situations. This would also mean, that we'd have to think about how this affect third-party build tools (e.g. bazel, gb…). It definitely makes this not much of a simplification (quite the opposite). Note, that the current (tooling-only) mechanism to access unexported identifiers from a _test-package is exactly what this proposal wants to forbid.

[natefinch]

This proposal still allows you to write tests that access internal package details. I'm not sure what you're driving at here.

I wanted to forestall any confusion in case someone might think that I wanted to remove internal testing entirely.

[rogpeppe]

APIs defined in test-files can not be used by importers (as you point out, test is not a build tag). Why would you need such a map? And how is a map keyed by "import path + test(yes/no) + <build_tags>" significantly worse than a map keyed by "import path + <build_tags>" for that purpose?

Such a map can't be keyed just by (importPath, isTest, buildTags) tuple, because test isn't a bool - it's a vector. It's more like (importPath, packageUnderTest, buildTags) - i.e. each package can have an indefinite number of versions, not just two.

To demonstrate, consider the following module:

-- go.mod --
module example
-- a/a.go --
package a

import (
	"fmt"
	"reflect"

	"example/b"
	"example/c"
)

var (
	stringerType = reflect.TypeOf(new(fmt.Stringer)).Elem()
	BIsStringer  = reflect.TypeOf(b.B{}).Implements(stringerType)
	CIsStringer  = reflect.TypeOf(c.C{}).Implements(stringerType)
)
-- b/b.go --
package b

type B struct{}

-- b/b1_test.go --
package b

func (B) String() string { return "b" }

-- b/b2_test.go --
package b_test

import (
	"testing"

	"example/a"
)

func Test(t *testing.T) {
	if got, want := a.BIsStringer, true; got != want {
		t.Fatalf("B: got %v want %v", got, want)
	}
	if got, want := a.CIsStringer, false; got != want {
		t.Fatalf("C: got %v want %v", got, want)
	}
}
-- c/c.go --
package c

type C struct{}
-- c/c1_test.go --
package c

func (C) String() string { return "c" }
-- c/c2_test.go --
package c_test

import (
	"testing"

	"example/a"
)

func Test(t *testing.T) {
	if got, want := a.BIsStringer, false; got != want {
		t.Fatalf("B: got %v want %v", got, want)
	}
	if got, want := a.CIsStringer, true; got != want {
		t.Fatalf("C: got %v want %v", got, want)
	}
}

Note that Test in c2_test.go and Test in b2_test.go both expect different results.
Now let's see how this looks under the go/packages API. Here's some code to print some info about all the packages within the example module:

package main

import (
	"fmt"
	"log"
	"strings"

	"golang.org/x/tools/go/packages"
)

func main() {
	pkgs, err := packages.Load(&packages.Config{
		Mode:  packages.LoadTypes,
		Dir:   ".",
		Tests: true,
	}, "./...")
	if err != nil {
		log.Fatalf("cannot load packages: %v", err)
	}
	packages.Visit(pkgs, func(pkg *packages.Package) bool {
		if strings.HasPrefix(pkg.PkgPath, "example/") {
			fmt.Printf("%s\n", pkg.ID)
		}
		return true
	}, nil)
}

When run in the example module directory, this prints:

example/b
example/c
example/a
example/b [example/b.test]
example/b_test [example/b.test]
example/a [example/b.test]
example/b.test
example/c [example/c.test]
example/c_test [example/c.test]
example/a [example/c.test]
example/c.test

Note that there are three versions of example/a, one for each package that depends on an internal module that's under test and has internal test code. In a larger code base where we're testing a lower level package with some much higher level testing infrastructure that depends on it, the number of extra packages could be considerable.

If I was writing some vet-like tool that relied on type information, this complicates matters; for example, which versions of example/a do I report errors in? When compiling Go packages, you may have seen duplicate error message being printed. This is the reason that happens.

[rogpeppe]

AFAICS if this proposal was implemented, there would be no need for any of the example/a [example/b.test]-style package identifiers - that's a complexity introduced entirely by the current system.

[Merovius]

That's an interesting example. Effectively a circular import after all. Weird, but makes sense :)

If I was writing some vet-like tool that relied on type information, this complicates matters; for example, which versions of example/a do I report errors in?

I would argue: The one that is the root of the analysis. I.e. running go test in example/b will only encounter one version of example/b and thus only one version of example/a. It's fair though, that you can have multiple roots and that causes ambiguity.

Alright, I see how this would simplify one use case for tools :) Still am of the opinion it's not worth the downside of having to either limit testing to exported APIs or having to change the language.

[rogpeppe]

Lastly: If we don't want to reduce the utility of testing (i.e. still allow accessing internal identifiers), ISTM that this would require a language change - the prohibition of accessing unexported identifiers of foreign packages is part of the language and the compiler would need to both generate the necessary symbols when compiling the package-under-test and know to allow accessing unexported identifiers in certain situations.

Yes, the compiler would need to generate unexported symbols when the package has tests that use the import "." syntax, but gc at least already generates those symbols; I don't know if gccgo elides unused unexported identifiers before link time or not.

And, yes, the compiler would need to know that unexported symbols could be accessed in this particular case, and that is indeed a language change. Personally I think that's better than the current somewhat hacky (and problematic in the larger view of things) situation, but yours is definitely a valid perspective - there is a trade-off here.

This would also mean, that we'd have to think about how this affect third-party build tools (e.g. bazel, gb…). It definitely makes this not much of a simplification (quite the opposite).

I'm not sure. It might well end up simplifying things in those tools too. I wouldn't rush to conclusions here.