Only link delayed transport AFTER real transport has called transportReady()

[zhangkun83]

Resolves #1477

If TransportSet fails to connect a transport (i.e., transportShutdown()
called without transportReady()), TransportSet will automatically
schedule reconnection for the next address, unless it has reached the end
of the address list, in which case it will fail the delayed transport.

This will avoid stream errors caused by bad addresses appearing before
good addresses in the resolved address list.

@ejona86 Please review this PR as a whole. I will squash all commits upon check-in.

[ejona86]

I think it's okay for now to mostly lose the statuses for earlier failed transports. However, it would probably be a good idea to add s to this log output in case someone needs to debug why earlier addresses failed.

[zhangkun83]

Done.

[ejona86]

@zhangkun83 LGTM.

Travis on Linux got stuck on :grpc-interop-testing:test for 10 minutes without making progress. I restarted the job and it seems to have gotten hung up on it again. I don't expect it is due to a bug in this PR, but this PR may be triggering it to break. I've been having trouble with timeouts exceeding in interop-testing with some of the PRs I've done. I think it is memory-related.

[zhangkun83]

I logged the test methods. It's stuck in InProcessTest.pingPong.

io.grpc.testing.integration.InProcessTest > veryLargeRequest STARTED
io.grpc.testing.integration.InProcessTest > veryLargeRequest PASSED
io.grpc.testing.integration.InProcessTest > deadlineExceeded STARTED
io.grpc.testing.integration.InProcessTest > deadlineExceeded PASSED
io.grpc.testing.integration.InProcessTest > deadlineNotExceeded STARTED
io.grpc.testing.integration.InProcessTest > deadlineNotExceeded PASSED
io.grpc.testing.integration.InProcessTest > exchangeMetadataUnaryCall STARTED
io.grpc.testing.integration.InProcessTest > exchangeMetadataUnaryCall PASSED
io.grpc.testing.integration.InProcessTest > unimplementedMethod STARTED
io.grpc.testing.integration.InProcessTest > unimplementedMethod PASSED
io.grpc.testing.integration.InProcessTest > veryLargeResponse STARTED
io.grpc.testing.integration.InProcessTest > veryLargeResponse PASSED
io.grpc.testing.integration.InProcessTest > emptyUnary STARTED
io.grpc.testing.integration.InProcessTest > emptyUnary PASSED
io.grpc.testing.integration.InProcessTest > exchangeMetadataStreamingCall STARTED
io.grpc.testing.integration.InProcessTest > exchangeMetadataStreamingCall PASSED
io.grpc.testing.integration.InProcessTest > pingPong STARTED
No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.

[zhangkun83]

Never mind. The test seems to be stuck in various method each time.
I also tried setting memory to 2G:

    test {
        testLogging {
            exceptionFormat = 'full'
            showExceptions true
            showCauses true
            showStackTraces true
            maxGranularity 3
            minGranularity 3
            events = ["started", "failed", "passed"]
        }
        maxHeapSize = '2000m'
    }

Still stuck.

[zhangkun83]

I am sure this is the kind of deadlock that I worried in #1408.
See zhangkun83#1 . The last commit, which moved the meat of transportReady() and transportShutdown() to the executor, fixed the test.

[zhangkun83]

This change will make obtainActiveTransport() always return a DelayedClientTransport on the first call (right now it would return the real transport). When DelayedClientTransport used with InProcessTransport, there is a risk of deadlock (#1510). I have to put this on hold until we fix the deadlock.

[zhangkun83]

Rebased onto master so it includes the fix from #1526.
Have run grpc-interop-testing:test for 1000 times on my desktop. Had no failures other than 55 times of:

io.grpc.testing.integration.Http2NettyTest > classMethod FAILED
    java.lang.RuntimeException at Http2NettyTest.java:60
        Caused by: java.io.IOException at Http2NettyTest.java:60
            Caused by: java.net.BindException

which I believe is unrelated to this change.

@ejona86 PTAL

[ejona86]

@zhangkun83, the new commit LGTM. And yeah, BindException seems unrelated.