fix(lsp-server): handle unicode custom root diagnostics parity

Author: fengkx

AGENTS.md

@@ -8,3 +8,12 @@
 - When adding a new runtime dependency to `packages/lsp-server/package.json`, also review `packages/lsp-server/tsup.config.ts`.
 - If the dependency is imported by the server runtime code, add it to `nodeConfig.noExternal` unless there is a clear reason to keep it external.
 - Reason: `packages/lsp-server/scripts/prepare-client-assets.js` copies built server artifacts into `lsp-client`; required runtime dependencies should be bundled to avoid missing module errors at extension runtime.
+
+## Playground Browser Testing Notes
+- When editing files inside the VSCode Web playground, prefer whole-file replacement (`focus editor -> Select All -> type/paste full content`) over partial line edits.
+- Reason: editor language features (auto-indent, snippet/format behaviors, newline handling) can introduce unintended indentation or syntax changes during incremental typing.
+- Always verify editor focus before typing. VSCode Web frequently shifts focus to the command palette, Problems panel, diff editors, or other UI panels.
+- Prefer minimizing the repro fixture (for example, reduce `main.bean` includes to only the files under test) to reduce noise and focus-switch count.
+- After browser automation edits, verify actual file contents before trusting diagnostics. Useful pattern: add/call debug commands that copy active file content or a project snapshot.
+- Be careful when using command palette automation: fuzzy search can select similarly named commands (for example compare-with-clipboard flows) and open diff editors unexpectedly.
+- If a syntax error appears immediately after automated typing, first suspect accidental auto-indent / misplaced whitespace from editor input rather than parser/runtime incompatibility.

docs/specs/beancount-lsp-p0-correctness-implementation-summary-2026-02-25.md

@@ -258,6 +258,100 @@ Changes:
 - Reused shared config loading in both initial registration and config-changed path
 - Reworked `mergeAndDedupDiagnostics()` key to include:
   - start/end range
+
+## G. Post-Implementation Follow-up (2026-02-26)
+
+### G1. Root-name option changes now trigger diagnostics revalidation
+
+Problem observed in playground regression:
+
+- Editing `main.bean` to add/remove `option "name_assets"` (and related root-name options) updated effective options correctly, but existing diagnostics in other open files could remain stale until another trigger.
+
+Fix implemented:
+
+- `packages/lsp-server/src/common/features/diagnostics.ts`
+- Extended diagnostics revalidation trigger set (`REVALIDATE_ON_OPTION_CHANGE`) to include:
+  - `name_assets`
+  - `name_liabilities`
+  - `name_equity`
+  - `name_income`
+  - `name_expenses`
+
+Observed effect:
+
+- Adding a custom root (e.g. `Actifs`) produces expected root-account diagnostics.
+- Removing/restoring the option clears those diagnostics automatically without stale leftovers.
+
+### G2. Browser WASM custom non-ASCII root compat filter (beancheck parity workaround)
+
+Problem observed:
+
+- In browser WASM runtime, `beancheck` diagnostics could report `Invalid account name: <non-ASCII-root>:...` for custom non-ASCII root names, while local Python Beancount accepted equivalent minimal examples.
+- Tree-sitter parsing and local LSP account-root validation supported the Unicode roots, so this was not a parser/local-validation issue.
+
+Fix implemented:
+
+- `packages/lsp-server/src/common/features/diagnostics.ts`
+- Added a browser-WASM-only compatibility filter for beancheck diagnostics when all conditions hold:
+  - runtime mode is `wasm`
+  - custom root options are active
+  - at least one custom root contains non-ASCII characters
+- Suppresses beancheck diagnostics matching custom non-ASCII roots for:
+  - `Invalid account name: ...`
+  - `Invalid reference to unknown account '...'` (chained noise)
+- Shows a one-time warning that browser WASM diagnostics were partially suppressed and local Python runtime is authoritative.
+
+Observed effect in playground:
+
+- Russian custom roots (`Активы / Пассивы / Капитал / Доходы / Расходы`) can be used in `open` directives and transactions without beancheck noise flooding the Problems panel (browser WASM mode).
+
+### G3. Diagnostics source labeling and message language normalization
+
+Problem observed:
+
+- It was hard to distinguish which diagnostics came from local LSP checks vs runtime beancheck.
+- Root account local validation message was emitted in Chinese while surrounding diagnostics were otherwise English-first.
+
+Fix implemented:
+
+- `packages/lsp-server/src/common/features/diagnostics.ts`
+- Split `Diagnostic.source` values:
+  - local diagnostics: `beancount-lsp (lsp)`
+  - runtime beancheck diagnostics: `beancount-lsp (beancheck)`
+- Converted root-account validation message to English:
+  - `Invalid root account name "...". Valid root account names: ...`
+
+Reviewer note:
+
+- This intentionally keeps the existing validation strategy (open-document local validation does not require inclusion from `main.bean`).
+
+### G4. Playground debug commands for browser repro verification
+
+Problem observed:
+
+- VSCode Web focus/panel interactions can make browser automation appear successful while text was not actually written into the intended editor.
+
+Implementation:
+
+- `packages/playground/src/main.ts`
+- Added debug commands:
+  - `demo.copyActiveFileContent`
+  - `demo.copyAllProjectFilesSnapshot`
+
+Use:
+
+- Verify active editor content after each replacement (especially non-ASCII root tests).
+- Capture deterministic file snapshots (`path`, `length`, `sha256`, `head`) for repro handoff.
+
+### G5. Browser validation findings (manual playground runs)
+
+Validated behaviors in VSCode Web playground (WASM v3):
+
+- Custom root options in Cyrillic are recognized by local validation and root lists.
+- Account names using Cyrillic custom roots parse and render in editor/UI (outline/breadcrumb/codelens paths observed).
+- `F2` rename can be invoked on Cyrillic account names (prepare-rename succeeds and pre-fills full Cyrillic account name).
+- Opening a file not included by `main.bean` can still produce local diagnostics:
+  - this is expected because local LSP validation runs on open documents independently of beancheck include graph.
   - severity
   - code
   - message

packages/lsp-server/src/common/features/diagnostics.ts

@@ -32,19 +32,108 @@ interface DiagnosticsConfig {
 	warnOnIncompleteTransaction: boolean;
 }
 
+const DIAGNOSTIC_SOURCE_LOCAL = 'beancount-lsp (lsp)';
+const DIAGNOSTIC_SOURCE_BEANCHECK = 'beancount-lsp (beancheck)';
+
+const ROOT_OPTION_NAMES = [
+	'name_assets',
+	'name_liabilities',
+	'name_equity',
+	'name_income',
+	'name_expenses',
+] as const;
+
+const DEFAULT_ROOT_NAMES = new Set(['Assets', 'Liabilities', 'Equity', 'Income', 'Expenses']);
+const NON_ASCII_RE = /[^\x00-\x7F]/;
+const INVALID_ACCOUNT_NAME_RE = /^Invalid account name:\s*(.+)$/;
+const INVALID_UNKNOWN_ACCOUNT_RE = /^Invalid reference to unknown account ['"](.+?)['"]$/;
+
 function isNotGitUri(uri: string): boolean {
 	return URI.parse(uri).scheme !== 'git';
 }
 
+function extractAccountFromBeancheckMessage(message: string): string | null {
+	return message.match(INVALID_ACCOUNT_NAME_RE)?.[1]
+		?? message.match(INVALID_UNKNOWN_ACCOUNT_RE)?.[1]
+		?? null;
+}
+
+function shouldSuppressCustomRootParityDiagnostic(
+	diag: Diagnostic,
+	suppressibleRoots: Set<string>,
+): boolean {
+	const account = extractAccountFromBeancheckMessage(diag.message);
+	if (!account) {
+		return false;
+	}
+	const root = account.split(':')[0];
+	return root != null && suppressibleRoots.has(root);
+}
+
+interface BrowserCustomRootCompatFilterInput {
+	runtimeMode: 'off' | 'local' | 'wasm';
+	validRootAccounts: Set<string>;
+	customNonAsciiRoots: Set<string>;
+}
+
+interface BrowserCustomRootCompatFilterResult {
+	diagnosticsByUri: Record<string, Diagnostic[]>;
+	suppressedCount: number;
+}
+
+function filterBeancheckDiagnosticsForBrowserCustomRootCompat(
+	diagnosticsByUri: Record<string, Diagnostic[]>,
+	input: BrowserCustomRootCompatFilterInput,
+): BrowserCustomRootCompatFilterResult {
+	if (input.runtimeMode !== 'wasm' || input.customNonAsciiRoots.size === 0) {
+		return { diagnosticsByUri, suppressedCount: 0 };
+	}
+
+	const suppressibleRoots = new Set(
+		[...input.customNonAsciiRoots].filter(root => input.validRootAccounts.has(root)),
+	);
+	if (suppressibleRoots.size === 0) {
+		return { diagnosticsByUri, suppressedCount: 0 };
+	}
+
+	let suppressedCount = 0;
+	const next: Record<string, Diagnostic[]> = {};
+	for (const [uri, diagnostics] of Object.entries(diagnosticsByUri)) {
+		const kept = diagnostics.filter(diag => {
+			if (!shouldSuppressCustomRootParityDiagnostic(diag, suppressibleRoots)) {
+				return true;
+			}
+			suppressedCount += 1;
+			return false;
+		});
+		if (kept.length > 0) {
+			next[uri] = kept;
+		}
+	}
+
+	return { diagnosticsByUri: next, suppressedCount };
+}
+
 export class DiagnosticsFeature implements Feature {
 	private logger = new Logger('DiagnosticsFeature');
+	private static readonly REVALIDATE_ON_OPTION_CHANGE = new Set([
+		'infer_tolerance_from_cost',
+		'inferred_tolerance_multiplier',
+		'name_assets',
+		'name_liabilities',
+		'name_equity',
+		'name_income',
+		'name_expenses',
+	]);
 	private config: DiagnosticsConfig = {
 		tolerance: 0.005, // Default tolerance
 		warnOnIncompleteTransaction: true, // Default to show warnings for incomplete transactions
 	};
 	private diagnosticsFromBeancount: { [uri: string]: Diagnostic[] } = {};
 	private standaloneBeancountDiagnosticUris = new Set<string>();
 	private readonly validationTokenByUri = new Map<string, CancellationTokenSource>();
+	private connection: Connection | undefined;
+	private hasShownBrowserCustomRootParityWarning = false;
 
 	constructor(
 		private readonly documents: DocumentStore,
@@ -55,6 +144,7 @@ export class DiagnosticsFeature implements Feature {
 
 	async register(connection: Connection): Promise<void> {
 		this.logger.info('Registering diagnostics feature');
+		this.connection = connection;
 
 		// Register callback on global bus to update diagnosticsFromBeancount on save
 		const onBeancountDiagnosticsUpdated = async () => {
@@ -76,7 +166,7 @@ export class DiagnosticsFeature implements Feature {
 		});
 
 		this.optionsManager.onOptionChange(async e => {
-			if (['infer_tolerance_from_cost', 'inferred_tolerance_multiplier'].includes(e.name)) {
+			if (DiagnosticsFeature.REVALIDATE_ON_OPTION_CHANGE.has(e.name)) {
 				await this.validateAllDocuments(connection);
 			}
 		});
@@ -165,7 +255,7 @@ export class DiagnosticsFeature implements Feature {
 				severity,
 				range,
 				message,
-				source: 'beancount-lsp',
+				source: DIAGNOSTIC_SOURCE_BEANCHECK,
 			} as Diagnostic;
 
 			const uri = file.includes('://') ? file : `file://${file}`;
@@ -187,7 +277,37 @@ export class DiagnosticsFeature implements Feature {
 
 			addDiagnostic(DiagnosticSeverity.Warning, f.line, f.file, f.message);
 		});
-		this.diagnosticsFromBeancount = diagnosticsFromBeancount;
+		const { diagnosticsByUri, suppressedCount } = filterBeancheckDiagnosticsForBrowserCustomRootCompat(
+			diagnosticsFromBeancount,
+			this.getBrowserCustomRootCompatFilterInput(),
+		);
+		this.diagnosticsFromBeancount = diagnosticsByUri;
+
+		if (suppressedCount > 0 && !this.hasShownBrowserCustomRootParityWarning) {
+			this.hasShownBrowserCustomRootParityWarning = true;
+			const message = 'Browser Beancount WASM runtime may not fully support custom non-ASCII root account names. Diagnostics were partially suppressed; switch to Local (Python) runtime for authoritative checks.';
+			this.logger.warn(`${message} suppressed=${suppressedCount}`);
+			void this.connection?.window.showWarningMessage(message);
+		}
+	}
+
+	private getBrowserCustomRootCompatFilterInput(): BrowserCustomRootCompatFilterInput {
+		const runtimeMode = this.beanMgr?.getRuntimeStatus().mode ?? 'off';
+		const validRootAccounts = this.optionsManager.getValidRootAccounts();
+		const customNonAsciiRoots = new Set<string>();
+		for (const name of ROOT_OPTION_NAMES) {
+			const option = this.optionsManager.getOption(name);
+			const value = option.asString();
+			if (option.isDefault || !NON_ASCII_RE.test(value) || DEFAULT_ROOT_NAMES.has(value)) {
+				continue;
+			}
+			customNonAsciiRoots.add(value);
+		}
+		return {
+			runtimeMode,
+			validRootAccounts,
+			customNonAsciiRoots,
+		};
 	}
 
 	private async validateDocument(document: TextDocument, connection: Connection): Promise<void> {
@@ -274,7 +394,7 @@ export class DiagnosticsFeature implements Feature {
 						severity: DiagnosticSeverity.Warning,
 						range: transaction.headerRange,
 						message: `transaction flagged with "!": ${document.getText(transaction.headerRange)}`,
-						source: 'beancount-lsp',
+						source: DIAGNOSTIC_SOURCE_LOCAL,
 					});
 				}
 
@@ -314,7 +434,7 @@ export class DiagnosticsFeature implements Feature {
 						severity: DiagnosticSeverity.Error,
 						range: transaction.headerRange,
 						message: `Transaction does not balance: ${imbalanceMessages.join(', ')}`,
-						source: 'beancount-lsp',
+						source: DIAGNOSTIC_SOURCE_LOCAL,
 					});
 				}
 			}
@@ -348,7 +468,7 @@ export class DiagnosticsFeature implements Feature {
 						end: { line, character: Math.max(0, lineText.length) },
 					},
 					message: 'Balance line incomplete: quick fix can complete current account balance',
-					source: 'beancount-lsp',
+					source: DIAGNOSTIC_SOURCE_LOCAL,
 					code: 'balance-missing-amount',
 				});
 			}
@@ -627,8 +747,8 @@ export class DiagnosticsFeature implements Feature {
 				diagnostics.push({
 					severity: DiagnosticSeverity.Error,
 					range: asLspRange(accountNode),
-					message: `无效的根账户名称 "${root}"。有效的根账户名称: ${validRootsList}`,
-					source: 'beancount-lsp',
+					message: `Invalid root account name "${root}". Valid root account names: ${validRootsList}`,
+					source: DIAGNOSTIC_SOURCE_LOCAL,
 					code: 'invalid-root-account',
 				});
 			}

packages/playground/src/main.ts

@@ -71,6 +71,27 @@ function debugLog(...args: unknown[]) {
 	console.log('[playground]', ...args);
 }
 
+function hasNonAscii(text: string): boolean {
+	return /[^\x00-\x7F]/.test(text);
+}
+
+async function sha256Hex(text: string): Promise<string> {
+	const data = new TextEncoder().encode(text);
+	const digest = await globalThis.crypto.subtle.digest('SHA-256', data);
+	return Array.from(new Uint8Array(digest))
+		.map(byte => byte.toString(16).padStart(2, '0'))
+		.join('');
+}
+
+async function copyTextWithFallback(text: string, promptTitle: string, successMessage: string): Promise<void> {
+	try {
+		await navigator.clipboard.writeText(text);
+		void window.showInformationMessage(successMessage);
+	} catch {
+		globalThis.prompt(promptTitle, text);
+	}
+}
+
 // ---------------------------------------------------------------------------
 // File System Access API detection
 // ---------------------------------------------------------------------------
@@ -882,6 +903,39 @@ commands.registerCommand('demo.copyShareUrl', async () => {
 	}
 });
 
+commands.registerCommand('demo.copyActiveFileContent', async () => {
+	const editor = window.activeTextEditor;
+	if (!editor || !isProjectFile(editor.document.uri)) {
+		void window.showInformationMessage('Open a project .bean file editor first.');
+		return;
+	}
+	const text = editor.document.getText();
+	await copyTextWithFallback(
+		text,
+		`Copy active file content (${editor.document.uri.path})`,
+		`Copied active file content (${text.length} chars, non-ASCII: ${hasNonAscii(text) ? 'yes' : 'no'}).`,
+	);
+});
+
+commands.registerCommand('demo.copyAllProjectFilesSnapshot', async () => {
+	const projectFiles = Array.from(stateFiles.entries())
+		.filter(([path]) => path.startsWith(PROJECT_PATH_PREFIX))
+		.sort(([a], [b]) => a.localeCompare(b));
+	const snapshot = await Promise.all(projectFiles.map(async ([path, content]) => ({
+		path,
+		length: content.length,
+		hasNonAscii: hasNonAscii(content),
+		sha256: await sha256Hex(content),
+		head: content.slice(0, 160),
+	})));
+	const text = JSON.stringify(snapshot, null, 2);
+	await copyTextWithFallback(
+		text,
+		'Copy all project files snapshot',
+		`Copied project snapshot (${snapshot.length} files).`,
+	);
+});
+
 commands.registerCommand('demo.resetDemo', () => {
 	if (!isMemfsMode) {
 		void window.showInformationMessage('Reset is only available in memfs mode.');