Potential Memory Leak With build_sensor_for_freshness_checks

[bolinzzz]

After upgrading to 1.8.10, I created a sensor to execute freshness checks based on the instructions here.

After the sensor is deployed, I began observing memory leaks. Screenshot below is showing memory usage of the container running one of our code locations.

This has led me to believe that build_sensor_for_freshness_checks is somehow causing the memory leak.

I haven't done any memory profiling, arrived at this conclusion solely because when I switched off the sensor, the memory leak stopped.

Some folks were saying the memory leak is caused by grpcio, when the grpc client connection fails, see this thread: grpc/grpc#36117

Can someone with more experience help to review the underlying code of build_sensor_for_freshness_checks, and identify any potential circumstances that might lead to memory leak. Thank you very much!

[garethbrickman]

This may be related to #18997, and #18997 (comment). What version of grpcio are you using? The solution in there was to pin the grpcio python package to 1.57.0

[bolinzzz]

I am using 1.66.2. Let me pin it to 1.57.0 and will revert back again. Thank you!

[bolinzzz]

I tried pinning grpcio to 1.57.0 but it did not work. After reducing the evaluation frequency of the sensor from 30s to 20mins helped to stop the leaking.