PII Redaction | Multi Language Support

[mishraomp]

PII Redaction Auto-Detect Language Design

Goal

Enable the PII redaction service to handle mixed-language chat payloads without requiring each tenant to declare one fixed detection_language value.

The design keeps the tenant-facing OpenAI request contract unchanged. Tenants still call the same API Management gateway endpoints with the same OpenAI-compatible payloads.

Current State

Today the platform uses one language code per tenant request:

• Tenant configuration sets apim_policies.pii_redaction.detection_language in tenant.tfvars.
• The API Management policy forwards that single value as piiDetectionLanguage.
• The redaction service receives config.detection_language as a single string.
• The language client stamps that same language onto every document sent to Azure Language Service.

This works for single-language tenants, but it is a poor fit for prompts that mix English, French, Spanish, or other languages in the same conversation.

Problem Statement

Changing detection_language from a string to a list does not solve the real problem.

The service still needs to decide which message or chunk uses which language. A tenant-level list only says that several languages are possible. It does not tell the service how to apply them to individual documents.

Because of that, language selection needs to move into the redaction service itself.

Recommended Design

Use automatic language detection inside the redaction service, but keep a short-term compatibility override.

Recommended request behavior:

• language_mode = "auto" means the service detects language per message or per chunk.
• language_mode = "fixed" means the service uses the configured detection_language for all documents.
• Existing tenants continue to work during rollout.
• Test can default to auto first.
• Production can later remove detection_language entirely if the override proves unnecessary.

Why This Design

This design gives the best balance of safety and usefulness.

• It supports true multilingual payloads.
• It does not change how tenants call OpenAI endpoints.
• It avoids forcing tenant administrators to manage a list of possible languages.
• It preserves a deterministic fallback during the test phase.
• It gives a clean path to removing the tenant variable later.

High-Level Flow

Current Flow

Tenant -> APIM -> /redact -> Azure PII detection with one fixed language -> APIM -> OpenAI backend

Proposed Flow

Tenant -> APIM -> /redact
                    -> build chunk manifest
                    -> detect language for each chunk
                    -> group chunks by language
                    -> run PII detection per language group
                    -> reassemble messages
                 -> APIM -> OpenAI backend

Request Contract

Tenant-Facing Contract

No change.

Tenant applications continue sending the same OpenAI-style request body to the same API Management gateway endpoint.

Internal APIM to Redaction-Service Contract

Recommended transition shape:

{
  "body": {
    "model": "gpt-4o",
    "messages": [
      { "role": "user", "content": "Bonjour, my passport is AB123456." }
    ]
  },
  "config": {
    "fail_closed": false,
    "excluded_categories": [],
    "language_mode": "auto",
    "detection_language": "en",
    "scan_roles": ["user", "assistant", "tool"],
    "correlation_id": "..."
  }
}

Notes:

• In auto mode, detection_language is ignored unless the detector cannot produce a supported answer.
• In fixed mode, detection_language remains required.
• After rollout, detection_language can become optional or be removed entirely.

Configuration Design

Short-Term Compatibility Model

Add one new tenant policy flag and keep the old one temporarily.

pii_redaction = {
  enabled             = true
  language_mode       = "auto" # "auto" or "fixed"
  detection_language  = "en"   # used only when language_mode = "fixed"
  fail_closed         = false
  excluded_categories = []
  scan_roles          = ["user", "assistant", "tool"]
}

End-State Model

After validation, the tenant configuration can simplify to:

pii_redaction = {
  enabled             = true
  fail_closed         = false
  excluded_categories = []
  scan_roles          = ["user", "assistant", "tool"]
}

In that final model, language selection is fully internal to the service.

Service Design

1. Extend Request Model

Add language_mode to the redaction config.

class RedactionConfig(BaseModel):
    fail_closed: bool = False
    excluded_categories: list[str] = Field(default_factory=list)
    language_mode: str = "auto"  # "auto" | "fixed"
    detection_language: str | None = None
    scan_roles: list[str] = Field(default_factory=lambda: ["user", "assistant", "tool"])
    correlation_id: str = ""

Validation rules:

• If language_mode == "fixed", detection_language must be present.
• If language_mode == "auto", detection_language is optional.

2. Add Language Detection Step

Introduce a small language detection client wrapper or a new method on the existing language client.

async def detect_languages(
    self,
    documents: list[dict[str, str]],
) -> dict[str, str]:
    payload = {
        "kind": "LanguageDetection",
        "analysisInput": {"documents": documents},
    }
    response = await self._http.post(
        f"/language/:analyze-text?api-version={self._api_version}",
        json=payload,
        headers=await self._get_auth_headers(),
    )
    response.raise_for_status()

    return {
        doc["id"]: doc["detectedLanguage"]["iso6391Name"]
        for doc in response.json().get("results", {}).get("documents", [])
    }

3. Detect Per Chunk, Not Per Whole Request

The detector should run on each chunk in the manifest, not on the whole payload.

Reason:

• One conversation can contain several languages.
• One long message can switch language partway through.
• Chunk-level language assignment lines up with the current batching model.

Suggested manifest extension:

class ChunkEntry(BaseModel):
    doc_id: str
    message_index: int
    chunk_index: int
    original_text: str
    detected_language: str | None = None
    redacted_text: str | None = None

4. Group PII Calls by Language

Once each chunk has a language, group by language and then batch within each group.

def group_entries_by_language(entries: list[ChunkEntry]) -> dict[str, list[ChunkEntry]]:
    grouped: dict[str, list[ChunkEntry]] = defaultdict(list)
    for entry in entries:
        grouped[entry.detected_language or "en"].append(entry)
    return grouped

This avoids mixing French and English documents in a single PII call when the Azure request expects one language per document processing path.

5. Add Fallback Logic

Auto-detect needs explicit fallback rules.

Recommended order:

1. If detector returns a supported language, use it.
2. If detector returns unsupported or unknown, use tenant detection_language if present.
3. If no tenant fallback exists, use en.
4. Log the fallback reason in diagnostics.

Example:

def choose_effective_language(
    detected_language: str | None,
    configured_language: str | None,
    supported_languages: set[str],
) -> str:
    if detected_language in supported_languages:
        return detected_language
    if configured_language in supported_languages:
        return configured_language
    return "en"

6. Preserve Timeout Budget

The design must fit inside the existing 55-second service budget and 60-second API Management timeout.

That means the language-detection pass must be budgeted explicitly.

Recommended rule:

• Reuse the existing batch size and concurrency controls.
• Detect language on the same chunk manifest used for PII processing.
• Reserve part of the total budget for detection before launching PII batches.

Suggested implementation sketch:

deadline = start + settings.total_processing_timeout_seconds

manifest = build_manifest(...)

if config.language_mode == "auto":
    await detect_manifest_languages(
        manifest=manifest,
        client=client,
        deadline=deadline,
        fallback_language=config.detection_language,
    )
else:
    for entry in manifest.entries:
        entry.detected_language = config.detection_language or "en"

batch_results = await run_language_grouped_pii_batches(
    manifest=manifest,
    client=client,
    config=config,
    deadline=deadline,
    settings=settings,
)

API Management Design

Transitional APIM Variables

API Management should send both language_mode and detection_language during rollout.

High-level fragment change:

<set-variable name="piiLanguageMode" value="${pii_language_mode}" />
<set-variable name="piiDetectionLanguage" value="${pii_detection_language}" />

And in the request body to the service:

new JProperty("language_mode",       languageMode),
new JProperty("detection_language",  language),

Final APIM State

Once the test rollout is complete, API Management can stop forwarding detection_language entirely if the service is fully auto-detect.

Diagnostics

Add explicit observability for language resolution.

Recommended new diagnostics fields:

• language_mode
• detected_language_count
• detected_languages
• language_fallback_count
• language_fallback_reasons
• language_detection_elapsed_ms

Example success diagnostics:

{
  "language_mode": "auto",
  "detected_languages": ["en", "fr"],
  "language_fallback_count": 1,
  "language_detection_elapsed_ms": 142.7
}

Risks

Low-Risk Areas

• No change to tenant OpenAI endpoint calls.
• No change to the public gateway contract.
• Test-only rollout reduces organizational risk.

Medium-Risk Areas

• Short chunks can be misdetected.
• Detection adds latency.
• Unknown-language fallback can hide quality issues if not logged clearly.
• Mixed-language chunking can split text in ways that reduce detector confidence.

Main Engineering Risk

The biggest risk is not the removal of the tenant variable. The biggest risk is adding a second analysis step while staying inside the current timeout budget.

Rollout Plan

Phase 1: Compatibility Rollout in Test

• Add language_mode.
• Default test tenants to auto.
• Keep detection_language as fallback.
• Add diagnostics for fallback frequency and latency.

Phase 2: Validation

Validate with real multilingual payloads:

• English only
• French only
• Spanish only
• English and French in separate messages
• English and French in the same long message
• Very short messages
• Messages with identifiers, names, and addresses only

Phase 3: Contract Simplification

If test results are good:

• Stop requiring detection_language in tenant config.
• Remove API Management forwarding of fixed language.
• Keep only internal fallback to en.

Testing Strategy

Unit Tests

Add tests for:

• language_mode = "fixed"
• language_mode = "auto"
• detector returns supported language
• detector returns unsupported language
• detector returns unknown language
• fallback to configured language
• fallback to English
• grouping by language
• reassembly after mixed-language batching

Integration Tests

Add or update test suites for:

• mixed English and French message payloads
• mixed role filtering and language detection
• timeout behavior with detection enabled
• fail-closed behavior when detection fails

Recommended Decision

Implement auto-detect now in test, but do not remove detection_language on day one.

Recommended policy:

• Add language_mode = "auto" | "fixed".
• Default test tenants to auto.
• Keep detection_language as a fallback and compatibility override.
• Remove the tenant variable later only after measured validation.

This gives you the auto-detect design you want without forcing an immediate one-way configuration break.