[codex] Fix Redis long-run memory retention by hengyoush · Pull Request #331 · hengyoush/kyanos

hengyoush · 2026-03-05T13:55:32Z

What changed

This PR fixes long-run memory retention in Redis parsing and adds a Linux long-run validation path for manual and GitHub-triggered verification.

Root cause

For Redis traffic, ReqQueue / RespQueue could retain parsed messages indefinitely when requests and responses were not matched for a long time, especially in one-sided or long-lived connection scenarios. The old Redis Match path returned without any queue bounding or eviction, so parsed RedisMessage payloads accumulated in memory.

Fix

bound pending parsed Redis messages by count and total bytes
trim stale unmatched Redis queues during Match
remove empty stream queues from the Redis stream maps
add Redis regression tests for unmatched request/response retention
fix pprof startup ordering so long-run profiling can run reliably
add a Linux long-run workflow and script for Redis validation

Validation

Functional validation

go test ./agent/protocol -run Redis

Remote Linux validation

Validated on 124.223.66.251.

Fixed build, one-sided Redis long run, 10 minutes

requests sent: 8,890,526
baseline heap: 39.99 MB
final heap: 37.49 MB

Pre-fix build, same one-sided Redis long run, 10 minutes

requests sent: 8,508,902
baseline heap: 70.28 MB
final heap: 1603.39 MB

The pre-fix heap profile was dominated by Redis parsing paths:

kyanos/agent/protocol.NewBinaryDecoder
kyanos/agent/protocol.ParseArray
kyanos/agent/protocol.getCmdAndArgs
kyanos/agent/conn.(*Connection4).parseStreamBuffer

The fixed build no longer showed Redis parsing as the dominant in-use heap consumer under the same workload.

Notes

This PR intentionally does not include local experimental validation scripts that are still uncommitted in the working tree.

vercel · 2026-03-05T13:55:38Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
kyanos	Ready	Preview, Comment	Apr 22, 2026 5:41pm

Issues-translate-bot · 2026-03-05T13:55:57Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
kyanos	Ready	Preview, Comment	Mar 5, 2026 1:55pm

Copilot

Pull request overview

Adds an opt-in pprof HTTP endpoint to the agent to support runtime profiling/debugging via new CLI flags.

Changes:

Add EnablePprof / PprofAddr to AgentOptions plus a defaulting helper (GetPprofAddr()).
Add --pprof and --pprof-addr persistent CLI flags.
Start a pprof HTTP server during agent setup when enabled.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
cmd/root.go	Wires new `--pprof` / `--pprof-addr` flags into the CLI to control profiling endpoint exposure.
agent/common/options.go	Extends `AgentOptions` with pprof settings and a helper for default address resolution.
agent/agent.go	Starts the pprof server during agent initialization when the option is enabled.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-05T14:02:15Z

+		addr := opts.GetPprofAddr()
+		common.AgentLog.Infof("Starting pprof server on http://%s/debug/pprof/", addr)
+		common.AgentLog.Info("pprof endpoints: /debug/pprof/heap, /debug/pprof/profile, /debug/pprof/goroutine")
+		if err := http.ListenAndServe(addr, nil); err != nil {


startPprofServer uses http.ListenAndServe with the default server (no timeouts) and no shutdown path. Please consider using an http.Server with at least ReadHeaderTimeout/IdleTimeout and tie shutdown to options.Ctx to avoid leaving an unbounded listener running and to reduce exposure to slowloris-style resource exhaustion if users bind to non-loopback addresses.

Suggested change

if err := http.ListenAndServe(addr, nil); err != nil {

srv := &http.Server{

Addr: addr,

Handler: nil,

ReadHeaderTimeout: 5 * time.Second,

IdleTimeout: 60 * time.Second,

}

// Tie server shutdown to the agent's context, if provided.

if opts.Ctx != nil {

go func() {

<-opts.Ctx.Done()

shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)

defer cancel()

if err := srv.Shutdown(shutdownCtx); err != nil && !errors.Is(err, http.ErrServerClosed) {

common.AgentLog.Errorf("Error shutting down pprof server: %v", err)

}

}()

}

if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {

cloudflare-workers-and-pages · 2026-03-05T14:15:31Z

Deploying kyanos with Cloudflare Pages

Latest commit:	`7e06a51`
Status:	✅ Deploy successful!
Preview URL:	https://82ca715d.kyanos.pages.dev
Branch Preview URL:	https://feat-pprof-endpoint.kyanos.pages.dev

View logs

Issues-translate-bot · 2026-03-05T14:15:49Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Deploying kyanos with Cloudflare Pages

Latest commit:	`a90b53e`
Status:	⚡️ Build in progress...

View logs

Add --pprof and --pprof-addr flags to enable pprof HTTP endpoint for runtime profiling and debugging. Security improvements: - Use http.Server with ReadHeaderTimeout (10s) and IdleTimeout (120s) - Graceful shutdown tied to options.Ctx with 5s timeout - Explicit pprof handler registration instead of default serve mux Changes: - Add EnablePprof and PprofAddr to AgentOptions - Add --pprof flag to enable pprof server (default: false) - Add --pprof-addr flag to configure listen address (default: localhost:6060) - Implement startPprofServer() with proper timeouts and shutdown handling - Update docs/debug-tips.md and docs/cn/debug-tips.md with usage examples Usage: kyanos watch --pprof kyanos watch --pprof --pprof-addr=:9090 The pprof endpoint provides access to: - /debug/pprof/heap - memory profiling - /debug/pprof/profile - CPU profiling - /debug/pprof/goroutine - goroutine dumps

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-05T15:29:00Z

+func (o AgentOptions) GetPprofAddr() string {
+	if o.PprofAddr == "" {
+		return "localhost:6060"
+	}
+	return o.PprofAddr
+}


GetPprofAddr() falls back to "localhost:6060", while the CLI flag default is 127.0.0.1:6060 and docs mention localhost:6060. Using localhost can also resolve to IPv6-only on some systems. Consider using a single constant/default (preferably 127.0.0.1:6060) across flags, docs, and this fallback to avoid surprising bind behavior.

Copilot · 2026-03-05T15:29:00Z

+| Endpoint | Description |
+|----------|-------------|
+| `/debug/pprof/` | Index page with all available profiles |
+| `/debug/pprof/heap` | Memory heap profile |
+| `/debug/pprof/profile` | CPU profile (30 seconds by default) |
+| `/debug/pprof/goroutine` | Goroutine dump |


The markdown table is malformed (each row starts with || instead of |), which won’t render correctly on GitHub. Remove the extra leading | so the table uses standard | ... | syntax.

Copilot · 2026-03-05T15:29:01Z

+| 端点 | 说明 |
+|------|------|
+| `/debug/pprof/` | 索引页面，显示所有可用的 profile |
+| `/debug/pprof/heap` | 内存堆 profile |
+| `/debug/pprof/profile` | CPU profile（默认 30 秒） |
+| `/debug/pprof/goroutine` | Goroutine 堆栈信息 |
+| `/debug/pprof/allocs` | 内存分配 profile |
+| `/debug/pprof/block` | 阻塞 profile |
+| `/debug/pprof/mutex` | 锁竞争 profile |


该 markdown 表格格式不正确（每行以 || 开头而不是 |），在 GitHub 上可能无法正常渲染。建议去掉多余的前导 |，使用标准的 | ... | 表格语法。

Copilot · 2026-03-05T15:29:01Z

 func SetupAgent(options ac.AgentOptions) {
 	startGopsServer(options)
+	startPprofServer(options)



startPprofServer(options) is called before options.Ctx is initialized (it’s set later via signal.NotifyContext). When --pprof is enabled, startPprofServer will block on <-opts.Ctx.Done() and will panic if Ctx is nil. Move the pprof startup to after options = ValidateAndRepairOptions(options) and after options.Ctx is set, or make startPprofServer tolerate a nil context (e.g., skip shutdown hook / use Background).

Copilot · 2026-03-05T15:29:01Z

+By default, the pprof server listens on `localhost:6060`. You can customize the address using the `--pprof-addr` flag:
+
+```bash
+./kyanos watch --pprof --pprof-addr="0.0.0.0:9090"


This example runs the pprof HTTP server with --pprof-addr="0.0.0.0:9090", which binds the profiling endpoint to all network interfaces with no authentication or TLS. In combination with the current --pprof implementation, any host that can reach this port can download heap/CPU profiles and goroutine dumps, which commonly contain sensitive in‑memory data. Consider keeping the example bound to localhost and explicitly warning that non-loopback addresses should only be used behind strong network controls (e.g. SSH tunnel or firewall) because they expose profiling data to the network.

Copilot · 2026-03-05T15:29:02Z

+默认情况下，pprof 服务器监听 `localhost:6060`。你可以使用 `--pprof-addr` 标志自定义监听地址：
+
+```bash
+./kyanos watch --pprof --pprof-addr="0.0.0.0:9090"


此示例使用 --pprof-addr="0.0.0.0:9090" 启动 pprof HTTP 服务器，会在没有任何认证或 TLS 的情况下监听所有网卡。结合当前 --pprof 实现，只要能访问该端口的主机就可以获取 heap/CPU profile 和 goroutine dump，这些数据中往往包含敏感内存信息。建议文档示例保持绑定在本机地址，并明确提示只有在有严格网络控制（如 SSH 隧道或防火墙）时才应使用非回环地址，否则会把 profiling 数据暴露到网络。

dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Mar 5, 2026

hengyoush requested a review from Copilot March 5, 2026 13:55

Copilot started reviewing on behalf of hengyoush March 5, 2026 13:56 View session

dosubot Bot added the enhancement New feature or request label Mar 5, 2026

Copilot AI reviewed Mar 5, 2026

View reviewed changes

hengyoush force-pushed the feat/pprof-endpoint branch from 4f664cc to a90b53e Compare March 5, 2026 14:15

dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Mar 5, 2026

vercel Bot deployed to Preview March 5, 2026 14:15 View deployment

vercel Bot deployed to Preview March 5, 2026 14:22 View deployment

vercel Bot deployed to Preview March 5, 2026 14:23 View deployment

hengyoush force-pushed the feat/pprof-endpoint branch from 7a4b04b to 87c75d6 Compare March 5, 2026 14:29

vercel Bot deployed to Preview March 5, 2026 14:29 View deployment

hengyoush force-pushed the feat/pprof-endpoint branch from 87c75d6 to 0448cea Compare March 5, 2026 14:30

vercel Bot deployed to Preview March 5, 2026 14:30 View deployment

hengyoush requested a review from Copilot March 5, 2026 15:20

Copilot started reviewing on behalf of hengyoush March 5, 2026 15:21 View session

Copilot AI reviewed Mar 5, 2026

View reviewed changes

Fix Redis long-run memory retention

7e06a51

vercel Bot deployed to Preview April 22, 2026 17:41 View deployment

hengyoush changed the title ~~feat: add pprof endpoint option for profiling~~ [codex] Fix Redis long-run memory retention Apr 22, 2026

hengyoush marked this pull request as draft April 22, 2026 17:41

-		if err := http.ListenAndServe(addr, nil); err != nil {
+		srv := &http.Server{
+			Addr:              addr,
+			Handler:           nil,
+			ReadHeaderTimeout: 5 * time.Second,
+			IdleTimeout:       60 * time.Second,
+		}
+		// Tie server shutdown to the agent's context, if provided.
+		if opts.Ctx != nil {
+			go func() {
+				<-opts.Ctx.Done()
+				shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+				defer cancel()
+				if err := srv.Shutdown(shutdownCtx); err != nil && !errors.Is(err, http.ErrServerClosed) {
+					common.AgentLog.Errorf("Error shutting down pprof server: %v", err)
+				}
+			}()
+		}
+		if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {

Conversation

hengyoush commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Root cause

Fix

Validation

Functional validation

Remote Linux validation

Fixed build, one-sided Redis long run, 10 minutes

Pre-fix build, same one-sided Redis long run, 10 minutes

Notes

Uh oh!

vercel Bot commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Issues-translate-bot commented Mar 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cloudflare-workers-and-pages Bot commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying kyanos with Cloudflare Pages

Uh oh!

Issues-translate-bot commented Mar 5, 2026

Deploying kyanos with Cloudflare Pages

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hengyoush commented Mar 5, 2026 •

edited

Loading

vercel Bot commented Mar 5, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Mar 5, 2026 •

edited

Loading