refactor(sql-engine): rebuild parsing, behavior, lineage, and execution

[zycgit]

Summary

Rebuild the SQL processing stack from parser entry points through execution-time authorization.

This branch replaces the legacy coupling between statement splitting, query types, resource analysis, security domains, and Console-specific execution logic with a shared pipeline built around:

• dialect-neutral SQL splitting and parser parameters;
• behavior relations for authorization and rule evaluation;
• lineage analysis for result-column provenance;
• version-aware system-resource registries;
• one Console query-analysis flow used by interactive queries, approvals, CI/CD, and auto-execution.

Most of the diff is the regenerated and expanded cross-dialect SQL fixture corpus required to validate the new contracts. The production changes span the plugin SDK, SQL engines, datasource plugins, Console services, audit persistence, security rules, and database migration.

Related Issues

None.

Type of Change

• Bug fix
• Feature
• Refactor
• Documentation
• Test
• Build / tooling
• Other

Commit Evolution

1. Establish the new engine foundation (d3b0e00ab–489e14206)

• Regenerate and reorganize the SQL split, behavior, security-domain, column, and parser fixture baseline.
• Decouple datasource plugins from direct SQL-engine implementations.
• Add dedicated ISO SQL-92, SQL-99, and SQL-2003 engines.
• Unify SQL security-domain analysis and introduce dialect-neutral behavior objects and relations.
• Add version-aware resource registries, initially covering MySQL built-in functions, system objects, and administrative behavior.
• Consolidate statement splitting across dialects and migrate SQL engines to the behavior-analysis API.
• Remove obsolete SDK analysis and execution models after their callers move to the new contracts.

2. Make parsing session-aware and dialect-complete (ffa941c72–d79f39b1b)

• Add dedicated Doris and StarRocks CREATE TABLE metadata parsers.
• Introduce session-level parser parameters and propagate datasource version and session context into SQL engines.
• Upgrade SQL language completion and parser-backed editor services.
• Complete multi-version MySQL grammar support and remove datasource-level sql_mode branching.
• Remove the legacy supportMultiStatement path in favor of the shared splitter.
• Rename and reshape statement classification around SplitQueryType.

3. Move Console execution onto the shared analysis pipeline (bc5820c7c–f2fb1147e)

• Unify query and resource analysis behind one Console analysis service.
• Adapt interactive execution, CI/CD, auto-execution, rule checks, and approval preparation to multi-statement analysis results.
• Correct query preparation and execution ordering.
• Simplify SQL audit backfill and eliminate direct dependencies on legacy QueryType values.
• Represent statement classifications as collections so a statement can expose all relevant behaviors.

4. Rebuild security, approval, and audit integration (bfbe4dec4–76a236e1f)

• Isolate security-rule domains from SQL split types and relocate analysis types to their owning SDK packages.
• Prepare SQL audit rows before dispatch, keyed by query_id, then transition them through PENDING, RUNNING, and terminal states.
• Persist behavior relations for audits and approvals instead of reconstructing permissions from removed SQL-kind/resource fields.
• Unify behavior-based authorization across Console queries, approval analysis, and execution backfill.
• Preserve historical security-rule behavior while migrating handlers to the new relation model.
• Introduce QueryAnalysisOptions and remove redundant analysis parameters from callers.

5. Harden dialect behavior, lineage, and masking (2505b9bd4–5bd73592d)

• Exempt registered built-in objects from ordinary object-permission checks.
• Correct quoted-name parsing and normalization across dialects.
• Add extensive DM8 split, behavior, lineage, and compatibility coverage plus a versioned DM resource registry.
• Remove the unfinished mandatory select-column constraint feature.
• Rebuild lineage derivation and carry source-column metadata into query result configuration.
• Drive existing sensitive-data value processing from lineage-aware column configuration.
• Add the dm_ds_meta_config persistence model, DAO, mapper, and migration foundation for datasource object metadata such as masking configuration.
• Refine behavior permission derivation and complete the final cross-dialect lineage fixes.

6. Reconcile main and stabilize the merged result (1c2c60108, c133b8a96)

• Merge the current main branch into the feature branch.
• Preserve the pre-execution audit state machine and audit username fallback (username → account → bindAccount → UID).
• Resolve the merged SQL-audit mapper contract.
• Remove stale initialization references to the deleted query-constraint service.
• Restore successful compilation across all backend modules.
• Align the merged auto-execution provider test with the renamed execDal field.
• Add periodic s-test progress and heartbeat output for the large dynamic fixture suite.

Key Changes

SDK and plugin contracts

• Add behavior-analysis and lineage-analysis SPIs with neutral object, relation, action, and source-name models.
• Move security analysis types into the SQL analysis/security packages that own them.
• Replace single-value statement classification with collection-based SplitQueryType results.
• Pass parser/session options explicitly instead of reading dialect settings from unrelated datasource configuration.

SQL engines and dialect support

• Reorganize SQL analysis, rewrite, split, security, lineage, and metadata packages.
• Add ISO SQL-92, SQL-99, and SQL-2003 implementations.
• Expand MySQL version handling and built-in/system resource classification.
• Add DM8 versioned resources and broad split/behavior/lineage compatibility fixtures.
• Correct targeted MySQL, Doris, quoted-identifier, and lineage regressions found during migration.

Console analysis and execution

• Centralize feature selection through QueryAnalysisOptions and QueryAnalysisFeature.
• Reuse behavior relations for interactive authorization, approval, CI/CD checks, auto-execution, and execution backfill.
• Carry lineage-derived source columns into result metadata for masking/value processing.
• Remove legacy post-execution analysis and duplicated resource-resolution paths.

Audit and persistence

• Insert audit records before SQL dispatch and update them by stable query_id.
• Keep transaction/session confirmation and rollback handling compatible with the new audit lifecycle.
• Store behavior relations on audit and approval records.
• Resolve blank audit usernames through account and bound-account fallbacks before using the UID.
• Apply the consolidated migration that:
  • adds audit query_id, indexes, nullable execution fields, and behavior storage;
  • removes obsolete audit and auto-task classification columns;
  • adds approval behavior storage;
  • renames dm_exec_query_constraints to dm_ds_meta_config;
  • replaces the old constraint payload with the masking metadata column.

Compatibility and Scope Notes

• This is a breaking internal SDK refactor for datasource and SQL plugins that implement the removed or relocated analysis contracts.
• The generated SQL fixtures are intentional review artifacts and account for most of the branch size.
• dm_ds_meta_config establishes the datasource-object metadata persistence contract. This PR does not add a management UI or write API for that table.
• The removed mandatory select-column feature is intentionally not preserved.

Review Guide

Recommended review order:

1. cgdm-plugin-sdk analysis, behavior, lineage, parser, and security contracts.
2. sqlc-common plus the SQL-engine implementations and versioned resource registries.
3. Console QueryAnalysisServiceImpl and its authorization, approval, CI/CD, auto-execution, and query callers.
4. SQL audit lifecycle, MyBatis mappings, and V202607270001__sql_audit_ack.
5. Cross-dialect tests and generated expected-result fixtures.

Test Plan

• Not tested (explain why below)
• Unit tests
• Integration tests
• Frontend lint / build
• Backend build
• Manual verification

Details:

cd backend
./gradlew compileJava

Result: BUILD SUCCESSFUL
103 compile tasks completed or confirmed up-to-date across backend modules.

Current GitHub Actions status:

• CLA: passed.
• Frontend check: passed.
• Backend test: rerun pending for b9053c915; the stale field injection is fixed and s-test now emits progress heartbeats.

The full local unit and integration test suites were not rerun during the final merge-conflict repair. The branch itself adds and updates extensive parser, split, behavior, security-domain, lineage, and compatibility coverage.

git diff --check origin/main...HEAD reports whitespace in tracked SQL grammar and SQL fixture content. Those findings are in the large fixture corpus rather than uncommitted local files; the working tree is clean and the branch head matches origin/feat/sql_engine_optimization.

Checklist

• The PR title clearly describes the full change.
• The change follows the repository coding standards.
• Documentation was updated when behavior or usage changed. (No user-facing documentation is included in this branch.)
• Tests were added or updated for behavior changes.
• I checked that dependency directories, logs, credentials, local build outputs, and other local artifacts are not included.