thread 'tokio-runtime-worker' panicked at nativelink-service/src/bytestream_server.rs:249:41

[rriskarimov]

I'm getting many warnings in my cas service logs:

thread 'tokio-runtime-worker' panicked at nativelink-service/src/bytestream_server.rs:249:41:
`async fn` resumed after completion
  2025-08-07T15:13:49.779565Z  WARN nativelink_util::buf_channel: Stream already closed when eof already was sent. This is often ok for retry was triggered, but should not happen on happy path.
    at nativelink-util/src/buf_channel.rs:158
    in nativelink_service::bytestream_server::inner_write with digest: DigestInfo("e09c2d36f013dd50a2edffed43bbaa175680e6ff23ada05dd6a1bdab6c7b8222-21875"), stream: WriteRequestStreamWrapper { resource_info: ResourceInfo { instance_name: "", uuid: Some("a0fef723-55a2-49d3-a032-70cba80d4512"), compressor: None, digest_function: Some("sha256"), hash: "e09c2d36f013dd50a2edffed43bbaa175680e6ff23ada05dd6a1bdab6c7b8222", size: "21875", expected_size: 21875, optional_metadata: None }, bytes_received: 0, first_msg: Some(WriteRequest { resource_name: "/uploads/a0fef723-55a2-49d3-a032-70cba80d4512/blobs/sha256/e09c2d36f013dd50a2edffed43bbaa175680e6ff23ada05dd6a1bdab6c7b8222/21875", write_offset: 16384, finish_write: false, data: b"<redacted>" }), write_finished: false }, stream.first_msg: "<redacted>"
    in nativelink_service::bytestream_server::bytestream_write
    in nativelink_service::bytestream_server::write with request: Streaming
    in nativelink_util::task::http_executor
    in nativelink::services::http_connection with remote_addr: 10.55.138.219:40498, socket_addr: 0.0.0.0:50051

Nativelink version: 0.6.0

CAS config:

{
  "stores": {
    "cas_FAST_SLOW_STORE": {
      "verify": {
        "backend": {
          "fast_slow": {
            "fast": {
              "size_partitioning": {
                "size": 64000,
                "upper_store": {
                  "noop": {}
                },
                "lower_store": {
                  "memory": {
                    "eviction_policy": {
                      "max_bytes": "32Gb",
                      "max_count": 100000000
                    }
                  }
                }
              }
            },
            "slow": {
              "filesystem": {
                "content_path": "/opt/nativelink/data/cas",
                "temp_path": "/opt/nativelink/data/tmp",
                "eviction_policy": {
                  "max_bytes": "400Gb"
                }
              }
            }
          }
        }
      },
      "verify_size": true,
      "hash_verification_function": "sha256"
    },
    "cas_STORE": {
      "existence_cache": {
        "backend": {
          "ref_store": {
            "name": "cas_FAST_SLOW_STORE"
          }
        },
        "eviction_policy": {
          "max_count": 1000000000,
          "max_seconds": 180000
        }
      }
    }
  },
  "servers": [
    {
      "listener": {
        "http": {
          "socket_address": "0.0.0.0:50051",
          "compression": {
            "send_compression_algorithm": "gzip",
            "accepted_compression_algorithms": [
              "gzip"
            ]
          },
          "advanced_http": {
            "experimental_http2_keep_alive_timeout": 1200
          },
          "tls": {
            "cert_file": "/opt/nativelink/cas.crt",
            "key_file": "/opt/nativelink/cas.key"
          }
        }
      },
      "services": {
        "cas": {
          "": {
            "cas_store": "cas_STORE"
        }
        },
       "capabilities": {},
        "bytestream": {
          "cas_stores": {
            "": "cas_STORE"
          }
        }
      }
    },
    {
      "listener": {
        "http": {
          "socket_address": "0.0.0.0:50061"
        }
      },
      "services": {
        "health": {}
      }
    }
  ]
}

[MarcusSorealheis]

We have a number of pending improvements in bytestream_server but we will need to add this issue to a list of challenges we need to seek to reproduce.

[MarcusSorealheis]

@rriskarimov Have you seen this issue again on main?

[MarcusSorealheis]

@rriskarimov Was there any error or only warning?