Skip to content

redis plugin: redis.raw_command retains the entire command argument via a V8 SlicedString, not just the 100-char truncation #10242

Description

@mykola-mokhnach-parloa

Summary

formatArg() in packages/datadog-plugin-redis/src/index.js truncates long command arguments to 100 characters for the redis.raw_command tag. Because String.prototype.slice() on a long string produces a V8 SlicedString — a view holding a pointer to its parent, not a copy — the resulting tag keeps the entire original argument alive for as long as the span is retained.

The tag reads as 100 characters and reports as 100 characters, but retains the full argument. For a SET of a 20 KB value that is ~200x more memory than expected.

Affected code (master, v6.15.0)

https://github.com/DataDog/dd-trace-js/blob/master/packages/datadog-plugin-redis/src/index.js#L100

function formatArg (arg) {
  if (typeof arg === 'string') {
    return arg.length > MAX_ARG_LENGTH ? arg.slice(0, MAX_ARG_LENGTH - 3) + '...' : arg
  }
  ...
}

Note the sibling truncation in formatCommand() (result.slice(0, MAX_COMMAND_LENGTH - 3), L92) is not affected: result is a ConsString, and slicing a ConsString forces V8 to flatten it first, which copies the leaf contents and releases the parents.

That means the leak occurs precisely when formatCommand returns without hitting MAX_COMMAND_LENGTH — e.g. SET <key> <large value>, which formats to ~110 characters and returns early, so the SlicedString from formatArg survives into the tag.

This was also present in datadog-plugin-openai and datadog-plugin-langchain in 5.31.0; both appear to have been refactored since, so this report is scoped to the redis plugin.

Impact

Retention lasts as long as the span is reachable. Spans stay in span._spanContext._trace.started[] until the trace finishes, so a service with long-running traces (or simply high span volume between flushes) accumulates one full command argument per instrumented Redis call.

Observed in a production Node service caching a ~21 KB JSON blob per request: heap snapshots showed ~145 live copies of the full 21 KB value, each reachable only through a redis.raw_command tag. Retaining path from the snapshot:

GC roots -> Global handles -> TCP -> AsyncContextFrame -> WeakMap
  -> .span            object:DatadogSpan
  -> ._spanContext._trace.started[16]  object:DatadogSpan
  -> ._spanContext._tags
  -> .redis.raw_command   concatenated string
     -> .second           concatenated string
        -> .first         sliced string          <- the 97-char truncation
           -> .parent     <the full 21 KB value>  <- retained

Reproduction

node --expose-gc repro.mjs

const MAX_ARG_LENGTH = 100

// verbatim from packages/datadog-plugin-redis/src/index.js @ v6.15.0
const formatArg = arg =>
  arg.length > MAX_ARG_LENGTH ? arg.slice(0, MAX_ARG_LENGTH - 3) + '...' : arg

// identical output, but the retained prefix is a fresh copy rather than a view on the parent
const formatArgCopied = arg =>
  arg.length > MAX_ARG_LENGTH
    ? Buffer.from(arg.slice(0, MAX_ARG_LENGTH - 3), 'utf8').toString('utf8') + '...'
    : arg

const N = 2000
const ARG_BYTES = 20 * 1024
const gc3 = () => { global.gc(); global.gc(); global.gc() }

const measure = (label, fn) => {
  gc3()
  const before = process.memoryUsage().heapUsed
  const tags = []                       // stands in for span._tags['redis.raw_command']
  for (let i = 0; i < N; i++) {
    const arg = `{"id":"${String(i).padStart(6, '0')}","payload":"` + 'x'.repeat(ARG_BYTES) + '"}'
    tags.push(fn(arg))                  // only the truncated tag is kept, exactly as the plugin does
  }
  gc3()
  const after = process.memoryUsage().heapUsed
  console.log(
    `${label.padEnd(22)} ${N} tags x ${tags[0].length} chars` +
    ` -> heap +${((after - before) / 1e6).toFixed(1)} MB` +
    ` (${((after - before) / N).toFixed(0)} bytes retained per 100-char tag)`,
  )
  return tags.length
}

console.log(`node ${process.version}; ${N} arguments of ~${(ARG_BYTES / 1024).toFixed(0)} KB each\n`)
measure('current formatArg', formatArg)
measure('with copied prefix', formatArgCopied)

Output (Node v24.14.1):

node v24.14.1; 2000 arguments of ~20 KB each

current formatArg      2000 tags x 100 chars -> heap +41.2 MB (20612 bytes retained per 100-char tag)
with copied prefix     2000 tags x 100 chars -> heap +0.4 MB (186 bytes retained per 100-char tag)

Suggested fix

Force the truncated prefix to be a standalone string rather than a view. Since the result is bounded by MAX_ARG_LENGTH, the copy is cheap and O(100):

if (typeof arg === 'string') {
  if (arg.length <= MAX_ARG_LENGTH) return arg
  return Buffer.from(arg.slice(0, MAX_ARG_LENGTH - 3), 'utf8').toString('utf8') + '...'
}

Any approach that copies works; the Buffer round-trip is just the least surprising way to express "flatten this" in portable JS. It may be worth a shared helper plus a regression test, since this is an easy pattern to reintroduce anywhere a plugin truncates a large value for a tag.

Versions

  • Observed in production: dd-trace 5.31.0 (via single-step APM library injection), Node v24.20.0
  • Confirmed still present on master / v6.15.0 by inspection
  • Reproduction run on Node v24.14.1

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions