Skip to content

Commit 2edeafa

Browse files
bassosimonebassosimone
authored
refactor: go test -short does not use the host network (#1352)
This diff refactors the codebase so that we avoid using the host network when running `go test -short`. This change is good in general, because now coverage tells us the amount of code we're covering without depending on interactions with an existing network, which means these tests behave in the same way in ~any place. I expect a coverage drop from this PR, because there's some coverage made with integration testing (if we consider integration tests the tests that require the host network interface with uncensored internet access). To make this happen, I needed to modify the `quictesting` package (now moved to toplevel and renamed `testingquic`) such that it attempts to get a known-to-work-well endpoint for QUIC _only_ when the developer using the package really needs it, rather than on import. Before doing this, there were several tests that panicked because `quictesting` could not figure out which IP address to use when you disable the WiFi or run inside another netns. Now we only figure this IP address out the first time a test requires us to give it either the domain or the endpoint that we should use. To be sure we continue to honour the promise that `go test -short` does not use the host network, I needed to refactor the CI such that we measure coverage inside a new network namespace with only localhost support. I think this compromise is acceptable, since the original ask was to avoid flaky network tests (see ooni/probe#2426). Because of this change in how we run the coverage checks, I am tentatively enabling also running all tests for pull requests, otherwise we don't know if a contribution breaks tests using the network. Hopefully, we should be fine because we are caching previous runs, so a bunch of tests should already be cached. Closes ooni/probe#2426. While there, enable again some backend integration tests, and close ooni/probe#2539.
1 parent 6b6271a commit 2edeafa

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+331
-66
lines changed

‎.github/workflows/alltests.yml

+1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# Runs the whole test suite
22
name: alltests
33
on:
4+
pull_request:
45
push:
56
branches:
67
- "release/**"

‎.github/workflows/coverage.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ jobs:
2424
go-version: "${{ steps.goversion.outputs.version }}"
2525
cache-key-suffix: "-coverage-${{ steps.goversion.outputs.version }}"
2626

27-
- run: go test -short -race -tags shaping -coverprofile=probe-cli.cov ./...
27+
- run: ./script/linuxcoverage.bash
2828

2929
- uses: shogo82148/actions-goveralls@v1
3030
with:

‎CONTRIBUTING.md

+20-7
Original file line numberDiff line numberDiff line change
@@ -36,13 +36,26 @@ functionality should pass existing tests. What's more, any new pull
3636
request that modifies existing functionality should not decrease the
3737
existing code coverage.
3838

39-
Long-running tests should be skipped when running tests in short mode
40-
using `go test -short`. We prefer internal testing to external
41-
testing. We generally have a file called `foo_test.go` with tests
42-
for every `foo.go` file. Sometimes we separate long running
43-
integration tests in a `foo_integration_test.go` file. We also
44-
sometimes have `foo_internal_test.go` when the main body of tests
45-
for `foo`, i.e., `foo_test.go` uses external testing.
39+
New code should have full coverage using either localhost or the
40+
[internal/netemx](./internal/netemx/) package. Try to cover all the
41+
error paths as well as the important properties of the code you've written
42+
that you would like to be sure about.
43+
44+
Additional integration tests using the host network are good,
45+
but they MUST use this pattern:
46+
47+
```Go
48+
func TestUsingHostNetwork(t *testing.T) {
49+
if testing.Short() {
50+
t.Skip("skip test in short mode")
51+
}
52+
}
53+
```
54+
55+
The overall objective here is for `go test -short` to only use localhost
56+
and [internal/netemx](./internal/netemx/) such that tests are always
57+
reproducible. Tests using the host network are there to give us extra
58+
confidence that everything is working as intended.
4659

4760
If there is a top-level DESIGN.md document, make sure such document is
4861
kept in sync with code changes you have applied.

‎internal/cmd/oohelperd/main_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@ type (
2121
)
2222

2323
func TestMainRunServerWorkingAsIntended(t *testing.T) {
24+
if testing.Short() {
25+
t.Skip("skip test in short mode")
26+
}
27+
2428
// let the kernel pick a random free port
2529
*apiEndpoint = "127.0.0.1:0"
2630

‎internal/database/actions_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -367,6 +367,10 @@ func TestPerformanceTestKeys(t *testing.T) {
367367
}
368368

369369
func TestGetMeasurementJSON(t *testing.T) {
370+
if testing.Short() {
371+
t.Skip("skip test in short mode")
372+
}
373+
370374
tmpfile, err := ioutil.TempFile("", "dbtest")
371375
if err != nil {
372376
t.Fatal(err)

‎internal/engine/experiment_integration_test.go

-1
Original file line numberDiff line numberDiff line change
@@ -130,7 +130,6 @@ func TestRunTelegram(t *testing.T) {
130130
}
131131

132132
func TestRunTor(t *testing.T) {
133-
t.Skip("https://github.com/ooni/probe/issues/2539")
134133
if testing.Short() {
135134
t.Skip("skip test in short mode")
136135
}

‎internal/engine/session_integration_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -492,6 +492,10 @@ func TestNewOrchestraClientProbeServicesNewClientFailure(t *testing.T) {
492492
}
493493

494494
func TestSessionNewSubmitterReturnsNonNilSubmitter(t *testing.T) {
495+
if testing.Short() {
496+
t.Skip("skip test in short mode")
497+
}
498+
495499
sess := newSessionForTesting(t)
496500
subm, err := sess.NewSubmitter(context.Background())
497501
if err != nil {

‎internal/engine/session_internal_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -225,6 +225,10 @@ func TestNewProbeServicesClientForCheckIn(t *testing.T) {
225225
}
226226

227227
func TestSessionNewSubmitterWithCancelledContext(t *testing.T) {
228+
if testing.Short() {
229+
t.Skip("skip test in short mode")
230+
}
231+
228232
sess := newSessionForTesting(t)
229233
ctx, cancel := context.WithCancel(context.Background())
230234
cancel() // fail immediately

‎internal/enginelocate/cloudflare_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,10 @@ import (
1212
)
1313

1414
func TestIPLookupWorksUsingcloudlflare(t *testing.T) {
15+
if testing.Short() {
16+
t.Skip("skip test in short mode")
17+
}
18+
1519
ip, err := cloudflareIPLookup(
1620
context.Background(),
1721
http.DefaultClient,

‎internal/enginelocate/geolocate_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -266,6 +266,10 @@ func TestLocationLookupSuccessWithResolverLookup(t *testing.T) {
266266
}
267267

268268
func TestSmoke(t *testing.T) {
269+
if testing.Short() {
270+
t.Skip("skip test in short mode")
271+
}
272+
269273
config := Config{}
270274
task := NewTask(config)
271275
result, err := task.Run(context.Background())

‎internal/enginelocate/iplookup_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,10 @@ import (
1313
)
1414

1515
func TestIPLookupGood(t *testing.T) {
16+
if testing.Short() {
17+
t.Skip("skip test in short mode")
18+
}
19+
1620
ip, err := (ipLookupClient{
1721
Logger: log.Log,
1822
Resolver: netxlite.NewStdlibResolver(model.DiscardLogger),

‎internal/enginelocate/resolverlookup_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ import (
99
)
1010

1111
func TestLookupResolverIPSuccess(t *testing.T) {
12+
if testing.Short() {
13+
t.Skip("skip test in short mode")
14+
}
15+
1216
rlc := resolverLookupClient{
1317
Logger: model.DiscardLogger,
1418
}

‎internal/enginelocate/stun_test.go

+8
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,10 @@ func TestSTUNIPLookupCannotDecodeMessage(t *testing.T) {
147147
}
148148

149149
func TestIPLookupWorksUsingSTUNEkiga(t *testing.T) {
150+
if testing.Short() {
151+
t.Skip("skip test in short mode")
152+
}
153+
150154
ip, err := stunEkigaIPLookup(
151155
context.Background(),
152156
http.DefaultClient,
@@ -163,6 +167,10 @@ func TestIPLookupWorksUsingSTUNEkiga(t *testing.T) {
163167
}
164168

165169
func TestIPLookupWorksUsingSTUNGoogle(t *testing.T) {
170+
if testing.Short() {
171+
t.Skip("skip test in short mode")
172+
}
173+
166174
ip, err := stunGoogleIPLookup(
167175
context.Background(),
168176
http.DefaultClient,

‎internal/enginelocate/ubuntu_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@ func TestUbuntuParseError(t *testing.T) {
3535
}
3636

3737
func TestIPLookupWorksUsingUbuntu(t *testing.T) {
38+
if testing.Short() {
39+
t.Skip("skip test in short mode")
40+
}
41+
3842
ip, err := ubuntuIPLookup(
3943
context.Background(),
4044
http.DefaultClient,

‎internal/experiment/dnscheck/dnscheck_test.go

+8
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,10 @@ func TestMakeResolverURL(t *testing.T) {
143143
}
144144

145145
func TestDNSCheckValid(t *testing.T) {
146+
if testing.Short() {
147+
t.Skip("skip test in short mode")
148+
}
149+
146150
measurer := NewExperimentMeasurer(Config{
147151
DefaultAddrs: "1.1.1.1 1.0.0.1",
148152
})
@@ -189,6 +193,10 @@ func TestSummaryKeysGeneric(t *testing.T) {
189193
}
190194

191195
func TestDNSCheckWait(t *testing.T) {
196+
if testing.Short() {
197+
t.Skip("skip test in short mode")
198+
}
199+
192200
endpoints := &Endpoints{
193201
WaitTime: 1 * time.Second,
194202
}

‎internal/experiment/echcheck/measure_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,10 @@ func TestMeasurerMeasureWithInvalidInput2(t *testing.T) {
6767
}
6868

6969
func TestMeasurementSuccess(t *testing.T) {
70+
if testing.Short() {
71+
t.Skip("skip test in short mode")
72+
}
73+
7074
sess := &mockable.Session{MockableLogger: log.Log}
7175
callbacks := model.NewPrinterCallbacks(sess.Logger())
7276
measurer := NewExperimentMeasurer(Config{})

‎internal/experiment/hhfm/hhfm_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,10 @@ func TestNewExperimentMeasurer(t *testing.T) {
3232
}
3333

3434
func TestSuccess(t *testing.T) {
35+
if testing.Short() {
36+
t.Skip("skip test in short mode")
37+
}
38+
3539
measurer := hhfm.NewExperimentMeasurer(hhfm.Config{})
3640
ctx := context.Background()
3741
sess := &mockable.Session{

‎internal/experiment/ndt7/dial_test.go

+8
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@ import (
1414
)
1515

1616
func TestDialDownloadWithCancelledContext(t *testing.T) {
17+
if testing.Short() {
18+
t.Skip("skip test in short mode")
19+
}
20+
1721
ctx, cancel := context.WithCancel(context.Background())
1822
cancel() // immediately halt
1923
mgr := newDialManager("wss://hostname.fake", log.Log, "miniooni/0.1.0-dev")
@@ -27,6 +31,10 @@ func TestDialDownloadWithCancelledContext(t *testing.T) {
2731
}
2832

2933
func TestDialUploadWithCancelledContext(t *testing.T) {
34+
if testing.Short() {
35+
t.Skip("skip test in short mode")
36+
}
37+
3038
ctx, cancel := context.WithCancel(context.Background())
3139
cancel() // immediately halt
3240
mgr := newDialManager("wss://hostname.fake", log.Log, "miniooni/0.1.0-dev")

‎internal/experiment/ndt7/ndt7_test.go

+8
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,10 @@ func TestGood(t *testing.T) {
131131
}
132132

133133
func TestFailDownload(t *testing.T) {
134+
if testing.Short() {
135+
t.Skip("skip test in short mode")
136+
}
137+
134138
ctx, cancel := context.WithCancel(context.Background())
135139
defer cancel()
136140
measurer := NewExperimentMeasurer(Config{}).(*Measurer)
@@ -162,6 +166,10 @@ func TestFailDownload(t *testing.T) {
162166
}
163167

164168
func TestFailUpload(t *testing.T) {
169+
if testing.Short() {
170+
t.Skip("skip test in short mode")
171+
}
172+
165173
ctx, cancel := context.WithCancel(context.Background())
166174
defer cancel()
167175
measurer := NewExperimentMeasurer(Config{noDownload: true}).(*Measurer)

‎internal/experiment/quicping/quicping_test.go

+8
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,10 @@ func TestSuccess(t *testing.T) {
123123
}
124124

125125
func TestWithCancelledContext(t *testing.T) {
126+
if testing.Short() {
127+
t.Skip("skip test in short mode")
128+
}
129+
126130
measurer := NewExperimentMeasurer(Config{})
127131
measurement := new(model.Measurement)
128132
measurement.Input = model.MeasurementTarget("google.com")
@@ -145,6 +149,10 @@ func TestWithCancelledContext(t *testing.T) {
145149
}
146150

147151
func TestListenFails(t *testing.T) {
152+
if testing.Short() {
153+
t.Skip("skip test in short mode")
154+
}
155+
148156
expected := errors.New("expected")
149157
measurer := NewExperimentMeasurer(Config{
150158
netListenUDP: func(network string, laddr *net.UDPAddr) (model.UDPLikeConn, error) {

‎internal/experiment/signal/signal_test.go

+4
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,10 @@ func TestNewExperimentMeasurer(t *testing.T) {
2323
}
2424

2525
func TestGood(t *testing.T) {
26+
if testing.Short() {
27+
t.Skip("skip test in short mode")
28+
}
29+
2630
measurer := signal.NewExperimentMeasurer(signal.Config{})
2731
measurement := new(model.Measurement)
2832
args := &model.ExperimentArgs{

‎internal/experiment/stunreachability/stunreachability_test.go

+8
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,10 @@ func TestRunWithUnsupportedURLScheme(t *testing.T) {
8989
}
9090

9191
func TestRunWithInput(t *testing.T) {
92+
if testing.Short() {
93+
t.Skip("skip test in short mode")
94+
}
95+
9296
measurer := NewExperimentMeasurer(Config{})
9397
measurement := new(model.Measurement)
9498
measurement.Input = model.MeasurementTarget(defaultInput)
@@ -158,6 +162,10 @@ func TestCancelledContext(t *testing.T) {
158162
}
159163

160164
func TestNewClientFailure(t *testing.T) {
165+
if testing.Short() {
166+
t.Skip("skip test in short mode")
167+
}
168+
161169
config := &Config{}
162170
expected := errors.New("mocked error")
163171
config.newClient = func(conn stun.Connection, options ...stun.ClientOption) (*stun.Client, error) {

‎internal/experiment/tlstool/internal/internal_test.go

+16
Original file line numberDiff line numberDiff line change
@@ -25,17 +25,33 @@ func dial(t *testing.T, d model.Dialer) {
2525
}
2626

2727
func TestNewSNISplitterDialer(t *testing.T) {
28+
if testing.Short() {
29+
t.Skip("skip test in short mode")
30+
}
31+
2832
dial(t, internal.NewSNISplitterDialer(config))
2933
}
3034

3135
func TestNewThriceSplitterDialer(t *testing.T) {
36+
if testing.Short() {
37+
t.Skip("skip test in short mode")
38+
}
39+
3240
dial(t, internal.NewThriceSplitterDialer(config))
3341
}
3442

3543
func TestNewRandomSplitterDialer(t *testing.T) {
44+
if testing.Short() {
45+
t.Skip("skip test in short mode")
46+
}
47+
3648
dial(t, internal.NewRandomSplitterDialer(config))
3749
}
3850

3951
func TestNewVanillaDialer(t *testing.T) {
52+
if testing.Short() {
53+
t.Skip("skip test in short mode")
54+
}
55+
4056
dial(t, internal.NewVanillaDialer(config))
4157
}

‎internal/experiment/urlgetter/getter_integration_test.go

+12
Original file line numberDiff line numberDiff line change
@@ -386,6 +386,10 @@ func TestGetterWithCancelledContextUnknownResolverURL(t *testing.T) {
386386
}
387387

388388
func TestGetterIntegrationHTTPS(t *testing.T) {
389+
if testing.Short() {
390+
t.Skip("skip test in short mode")
391+
}
392+
389393
ctx := context.Background()
390394
g := urlgetter.Getter{
391395
Config: urlgetter.Config{
@@ -510,6 +514,10 @@ func TestGetterIntegrationRedirect(t *testing.T) {
510514
}
511515

512516
func TestGetterIntegrationTLSHandshake(t *testing.T) {
517+
if testing.Short() {
518+
t.Skip("skip test in short mode")
519+
}
520+
513521
ctx := context.Background()
514522
g := urlgetter.Getter{
515523
Config: urlgetter.Config{
@@ -611,6 +619,10 @@ func TestGetterIntegrationTLSHandshake(t *testing.T) {
611619
}
612620

613621
func TestGetterHTTPSWithTunnel(t *testing.T) {
622+
if testing.Short() {
623+
t.Skip("skip test in short mode")
624+
}
625+
614626
// quick enough (0.4s) to run with every run
615627
ctx := context.Background()
616628
g := urlgetter.Getter{

0 commit comments

Comments
 (0)