[FEATURE]: Public API for removing tags from a Span (4.x lost this capability)

[jhaenchen]

Motivation

I maintain a TraceProcessor that acts as an allowlist-based redaction filter: it walks each span's tags, keeps the ones on a known-safe allowlist, transforms a few (e.g. redacts message bodies while keeping the command name), and drops everything else. This is a common shape for teams that need to guarantee no unexpected tag ever ships to Datadog — a defense-in-depth layer against an integration or a well-meaning set_tag() call leaking PII/PHI.

Under ddtrace 2.x / 3.x this was straightforward: the processor read span._meta, computed the sanitized dict, and assigned it back. _meta was a single-underscore internal attribute, but the pattern was stable and widely used.

In 4.x the Span was restructured (data moved into a native SpanData base + context) and _meta is no longer accessible on the Span. The public API kept the additive tag operations — get_tag, set_tag, get_tags, set_tags — but there is no public tag removal. set_tag(k, None) doesn't remove; it stores None. The only working call I've found is span._remove_attribute(k), which is defined on the internal SpanData base.

Ask

A supported public method for tag removal on Span. Something like:

span.remove_tag(key: str) -> None

and matching remove_metric(key) / remove_struct_tag(key) for the parallel storage. Alternatively, blessing _remove_attribute (etc.) as stable via docs would achieve the same practical outcome for consumers.

Current workaround

Falling back to the private methods:

• span._remove_attribute(key) — tag removal
• span._has_meta_structs() / span._get_meta_structs() / span._remove_struct_tag(key) — the same, for struct-typed metadata

These work in 4.11.x. It's a code smell, and it puts consumers on a manual-review cadence for every ddtrace bump — which is the opposite of what a public API should require.

Alternatives considered

• Redact at write time (subclass Span.set_tag) — inverts the design, wraps a hot path, and creates surface area for accidental bypass.
• Drop the whole trace when an unexpected tag appears — nuclear; auto-instrumentation adds runtime tags on every span, so this would drop most traces in practice.
• Skip 4.x — not viable long-term; 4.8.2 is the earliest release with the fix for CVE-2026-50271 (baggage-header DoS).

Happy to send a PR if there's alignment on the surface.

[brettlangdon]

hey @jhaenchen thanks for opening the issue.

EOD for me, but I'll need to think on this some more. I'd be fine with reviewing a PR, or any diff you wanted to just post here with your proposal for what would be helpful for you.

We do have these options available as well: https://docs.datadoghq.com/tracing/configure_data_security/ (replace tags was specifically what came to mind for me, this happens on the agent, and there is sensitive data scanner is also a good tool, but happens at ingestion side).

For context, the reason I am not agreeing 100% just yet is:

1. we want to be thoughtful of additions/changes to the public API since we'll then need to support them long term (even changing across major versions is a pain for users)
2. we've been talking internally about trying to move towards a Span model where they cannot be mutated, which would give us some flexibility in some performance optimizations. not set in stone, but any place we move in that direction is helpful, so again just want to be thoughtful

I think your use case makes sense to find a solution for, and we can try to figure out alternatives as well which might even be better.

Are there specific tags are want to remove, I see you mentioned "auto-instrumentation adds runtime tags on every span, so this would drop most traces in practice", are there tags we add by default you drop from every span?

[jhaenchen]

Thanks for engaging on this, Brett.

Allowlist vs Blocklist

The filter isn't "strip tag X." It's the inverse: keep only tags on a known-safe allowlist, drop everything else. In a compliance-sensitive environment I need a hard guarantee that a new integration, a new ddtrace default tag, or a stray set_tag() call in code I don't own can't exfiltrate data. If I have to enumerate what to redact, I've lost the property that makes the model safe.

That reframes the existing options.

replace_tags on the agent does per-tag regex replacement. It solves "I know this specific tag has emails and want them redacted." It does not give me "reject unknown-key by default." A tag added by a future integration goes through untouched.

Sensitive Data Scanner at ingestion is the same shape, downstream and post-hoc. For data protected under GDPR or HIPAA-equivalent regimes, "the PII briefly transited third-party infrastructure and was scrubbed on arrival" is materially different from "it never left our process." Ingestion-side mitigation is a good complement, not a substitute for source-side filtering.

Neither replaces a TraceProcessor running in the library.

Unknown Tags

The current ddtrace defaults (runtime-id, _dd.p.tid, _dd.p.dm, language, _dd.tags.process, and so on) are all allowlisted. The problem is the tags I don't know about yet. When 4.x added _dd.tags.process my allowlist gate reported it as unknown, I inspected, confirmed it was safe, and added it. Without the ability to drop unknown-key tags, that surface goes unmonitored.

Immutability

Fair concern, and I'd rather not fight the direction of the codebase. An immutability-friendly shape that solves the same problem:

class Span:
    def with_tags(self, tags: Mapping[str, str]) -> "Span": ...
    # or
    def rebuilt(self, *, tag_filter: Callable[[str, str], Optional[str]]) -> "Span": ...

A processor returns a new Span with different tags instead of mutating in place. Same behavior applies for _meta_struct. On my end it's the same allowlist code, just returning [new_span, ...] from process_trace instead of mutating. Happy to send a PR sketch of that shape if it sounds directionally useful.

[jhaenchen]

Here's a sketch of what an immutability-friendly shape could look like. Three methods, one per storage type, each returning a new Span with only that field replaced.

class Span:
    def with_tags(self, tags: Mapping[str, str]) -> "Span":
        """Return a new Span whose string tags are replaced by the given mapping.

        All other fields (trace_id, span_id, parent_id, name, service, resource,
        span_type, start_ns, duration_ns, error, metrics, meta_struct, events,
        links, context) are preserved.

        An empty mapping clears all tags.
        """

    def with_metrics(self, metrics: Mapping[str, Union[int, float]]) -> "Span": ...

    def with_meta_struct(self, meta_struct: Mapping[str, Mapping[str, Any]]) -> "Span": ...

Consumer code for the allowlist filter use case becomes:

class RedactingProcessor:
    def process_trace(self, trace: list[Span]) -> Optional[list[Span]]:
        return [
            span.with_tags({
                k: v for k, v in (span.get_tags() or {}).items()
                if k in ALLOWLIST
            })
            for span in trace
        ]

Properties this gives you:

• No mutation of an existing Span. Compatible with a future immutable Span model.
• No new "removal" API. The consumer expresses which tags it wants and the library constructs a new Span. Deletion falls out of the diff.
• Chainable: span.with_tags(...).with_metrics(...) if a processor filters more than one storage type.
• Symmetric across the three storage types. No special-casing.
• Trace-identity and telemetry fields are library-owned. Consumers can't accidentally rewrite trace_id, span_id, timing, etc.

If chaining feels wrong because each intermediate rebuilds a whole Span, an alternative is a single method with a sentinel:

class Span:
    _UNCHANGED = object()

    def replaced(
        self,
        *,
        tags: Union[Mapping[str, str], object] = _UNCHANGED,
        metrics: Union[Mapping[str, Union[int, float]], object] = _UNCHANGED,
        meta_struct: Union[Mapping[str, Mapping[str, Any]], object] = _UNCHANGED,
    ) -> "Span": ...

I lean toward the three-method variant because it's easier to type-hint and doesn't need a sentinel, but happy to work off whichever fits your codebase.

Happy to draft the internal changes if this shape sounds directionally right.

[brettlangdon]

Thanks, let me think on this. The cloning a span and just replacing it in the process_trace was one of the ideas that came to me as well.

I'll think about it some more and see if I can come up with any good ideas.

Tagging @mabdinur who might find this a fun problem to think through.