For connection failures, optionally log previous payload in http-server-stabilizer

[varungandhi-src]

Currently, we get logs like this: example (see access docs)

{
  "jsonPayload": {
    "Timestamp": 1711370806641021700,
    "Body": "error encountered",
    "Function": "main.main.func4",
    "InstrumentationScope": "worker",
    "Caller": "build/main.go:376",
    "Resource": {
      "service.name": "syntax_highlighter",
      "service.instance.id": "syntect-server-7897fccdff-7l5r9"
    },
    "Attributes": {
      "pid": 62,
      "error": "read tcp 127.0.0.1:42772->127.0.0.1:44649: read: connection reset by peer",
      "port": 44649
    },
    "SeverityText": "ERROR"
  },
  "timestamp": "2024-03-25T12:46:46.641173544Z",
  "severity": "ERROR",
  ...
 }

The above example show ~200 error messages within a minute. However, there isn't much information to connect the various failures (e.g. did we send requests for the same file to different workers? did we send them to different workers?)

In http-server-stabilizer, we should try to keep a copy of the previous request body sent to a worker and then log specific JSON fields of interest (e.g. the filename).

[varungandhi-src]

Another potential idea: maybe we can eagerly check for process death in addition to directly just connecting to the process?