Auto-regenerate derived tables via migrate_db.py --regenerate-derived-tables

migrate_db.py gains --regenerate-derived-tables (off by default) and
--derived-tables-sql. When passed, it compares generate_derived_tables.sql's
own header version against the database's current
info.derived_table_schema_version and regenerates derived tables whenever
they differ — covering both a base-table migration that requires a derived
rebuild and a derived-table-only version bump with no corresponding
migrate_schema.sql section. generate_derived_tables.sql now records its own
version into info.derived_table_schema_version at the end of the script, so
this works regardless of what invokes it (migrate_db.py, the docker-compose
init scripts, or metaImport.py's rebuild_derived_tables.py).

Off by default so institutional deployments that run derived-table
regeneration as a separate manual step (e.g. against ClickHouse Cloud) are
unaffected; cbioportal-docker-compose opts in explicitly.

Also:
- migrate_schema.sql: soften the idempotency rule from a hard requirement to
  a recommendation (not always achievable), and document the convention that
  a base-table migration must also bump generate_derived_tables.sql's
  version.
- test_db_version.sh: also check generate_derived_tables.sql's trailing
  info.derived_table_schema_version UPDATE against its own header version.
- README: document when derived tables get rebuilt across docker-compose and
  manual deployment scenarios, and describe the new flag.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: dippindots

docs/deployment/clickhouse/README.md

@@ -189,6 +189,14 @@ Without derived tables, every Study View page load would need to join across gen
 
 ### When Derived Tables Are Built
 
+| Scenario | Derived tables rebuilt? | Why |
+|---|---|---|
+| First-ever `docker compose up` (empty ClickHouse volume) | Yes | The fresh-install init scripts run `generate_derived_tables.sql` as part of first-time database setup. |
+| `docker compose up` on an existing, already-initialized database, no pending migration | No | Docker's init scripts only run once against an empty data volume; nothing else rebuilds derived tables on a plain restart. |
+| After importing a study (`metaImport.py`) | Yes, automatically | `metaImport.py` rebuilds derived tables after every successful import, unless you pass `--no-derive-tables` (see below). |
+| After `docker compose up` applies a pending schema migration | Yes, automatically | `migrate_db.py` is invoked with `--regenerate-derived-tables` in `cbioportal-docker-compose`, so it regenerates derived tables whenever `generate_derived_tables.sql`'s own version differs from the database's current version — whether or not the migration that triggered it touched base tables. See [§11 Version Migration](#11-version-migration). |
+| Manual/institutional deployments running `migrate_db.py` directly (no docker-compose) | No, unless you opt in | `migrate_db.py` does **not** regenerate derived tables by default — pass `--regenerate-derived-tables`, or rebuild them yourself as a separate step. See [§11 Version Migration](#11-version-migration). |
+
 By default, `metaImport.py` **automatically rebuilds derived tables** after every import. This ensures query performance stays fast after loading new studies.
 
 ### Skipping Derived Table Rebuild (`--no-derive-tables` and `derive-tables`)
@@ -268,17 +276,29 @@ Starting with `DB_SCHEMA_VERSION` `3.0.0`, in-place schema upgrades are handled
 sections) applied by `db-scripts/clickhouse/migrate/migrate_db.py`. The runner reads the current
 `db_schema_version` from the `info` table, skips sections already applied, and applies the rest in
 order. Derived table schema updates (tracked by `DERIVED_TABLE_SCHEMA_VERSION`) version
-independently and are applied by rebuilding your derived tables with `generate_derived_tables.sql`.
+independently, and are applied by rebuilding derived tables with `generate_derived_tables.sql`.
+
+**Version-bump convention:** any migration that changes base tables must also bump
+`generate_derived_tables.sql`'s version (even with no semantic change to that script), since a
+base-table change may affect derived tables in ways that aren't obvious to every contributor.
+`derived_table_schema_version` can also bump on its own, with no corresponding base-table
+migration.
 
 **Docker Compose deployments:** `git pull` the latest `cbioportal-docker-compose` master, then
 `docker compose up`. The migration step runs automatically before the `cbioportal` service starts;
-on a fresh install it's a safe no-op since `schema.sql` already seeds `info` at the current version.
+on a fresh install it's a safe no-op since `schema.sql` already seeds `info` at the current
+version. It also regenerates derived tables automatically whenever `generate_derived_tables.sql`'s
+version differs from the database's current `derived_table_schema_version` — you don't need a
+separate manual step.
 
 **Manual deployments (e.g. ClickHouse Cloud, Kubernetes, or any setup that doesn't go through
-`cbioportal-docker-compose`):** run `migrate_db.py` directly against your database, then rebuild
-derived tables, before deploying the new cBioPortal backend image. The backend refuses to start
-against a `db_schema_version` that doesn't match its build's `db.version` unless
-`db.suppress_schema_version_mismatch_errors=true` is set.
+`cbioportal-docker-compose`):** run `migrate_db.py` directly against your database before deploying
+the new cBioPortal backend image. By default `migrate_db.py` only touches base tables — pass
+`--regenerate-derived-tables` if you want it to also regenerate derived tables (using the same
+version-comparison logic described above) in the same run; otherwise, rebuild derived tables
+yourself as a separate step (e.g. if you run derivation through your own tooling against
+ClickHouse Cloud). The backend refuses to start against a `db_schema_version` that doesn't match
+its build's `db.version` unless `db.suppress_schema_version_mismatch_errors=true` is set.
 
 Upgrades from **before** `3.0.0` (i.e. the original v6→v7 migration, or any pre-migration-tooling
 ClickHouse deployment) still require the manual re-import process, since no migration path exists

src/main/resources/db-scripts/clickhouse/generate_derived_tables.sql

@@ -711,3 +711,8 @@ OPTIMIZE TABLE genetic_alteration_derived;
 OPTIMIZE TABLE generic_assay_data_derived;
 OPTIMIZE TABLE generic_assay_profile_entity_derived;
 OPTIMIZE TABLE generic_assay_meta_derived;
+
+-- Record the version of this script in info.derived_table_schema_version, so callers (e.g.
+-- migrate_db.py --regenerate-derived-tables) can detect whether derived tables are stale by
+-- comparing against this script's own header version, regardless of who/what last ran it.
+ALTER TABLE info UPDATE derived_table_schema_version = '2.0.0' WHERE 1;

src/main/resources/db-scripts/clickhouse/migrate/migrate_db.py

@@ -5,6 +5,12 @@
 sections and applies any section newer than the target database's current db_schema_version,
 strictly in ascending order. See the header of migrate_schema.sql for the section format.
 
+With --regenerate-derived-tables, also regenerates derived tables (generate_derived_tables.sql)
+whenever its version differs from the database's current info.derived_table_schema_version. This
+covers both a derived-table rebuild required by a base-table migration and a derived-table-only
+version bump shipped with no corresponding migrate_schema.sql section. Off by default, since
+institutional deployments may run derived-table regeneration as a separate manual step.
+
 ClickHouse connection is configured via environment variables, matching the convention used by
 cbioportal-core's rebuild_derived_tables.py:
     CLICKHOUSE_HOST, CLICKHOUSE_NATIVE_PORT, CLICKHOUSE_USER, CLICKHOUSE_PASSWORD, CLICKHOUSE_DB
@@ -29,9 +35,13 @@
 SECTION_HEADER_RE = re.compile(r'^##\s*db_schema_version:\s*(\S+)\s*$')
 DESCRIPTION_RE = re.compile(r'^##\s*description:\s*(.*)$')
 CUSTOM_RE = re.compile(r'^##\s*custom:\s*true\s*$', re.IGNORECASE)
+DERIVED_TABLE_VERSION_HEADER_RE = re.compile(
+    r'^-- version (\d+\.\d+\.\d+) of derived table schema and data definition\s*$')
 
 SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
 DEFAULT_MIGRATE_SCHEMA_SQL = os.path.join(SCRIPT_DIR, 'migrate_schema.sql')
+DEFAULT_GENERATE_DERIVED_TABLES_SQL = os.path.normpath(
+    os.path.join(SCRIPT_DIR, os.pardir, 'generate_derived_tables.sql'))
 
 # Hardcoded custom migration steps, keyed by db_schema_version. Each function receives the
 # ClickHouse connection properties dict and is responsible for whatever work that version's
@@ -94,6 +104,16 @@ def flush():
     return sections
 
 
+def get_derived_tables_sql_version(filepath):
+    with open(filepath) as f:
+        first_line = f.readline()
+    match = DERIVED_TABLE_VERSION_HEADER_RE.match(first_line.strip())
+    if not match:
+        raise RuntimeError(
+            f"Could not parse derived table schema version from the first line of {filepath}")
+    return match.group(1)
+
+
 def get_clickhouse_props():
     required_props = {
         'host': 'CLICKHOUSE_HOST',
@@ -175,6 +195,15 @@ def get_current_db_schema_version(ch_props):
     return version
 
 
+def get_current_derived_table_schema_version(ch_props):
+    version = run_query(ch_props, "SELECT derived_table_schema_version FROM info LIMIT 1")
+    if not version:
+        raise RuntimeError(
+            "Could not read derived_table_schema_version from info table. "
+            "Is the database initialized?")
+    return version
+
+
 def wait_for_mutations(ch_props, timeout_secs=300, poll_interval_secs=2):
     """Block until all ClickHouse mutations in this database have completed.
 
@@ -209,8 +238,42 @@ def apply_section(ch_props, section):
     print(GREEN + f"Applied db_schema_version {section.version}" + END)
 
 
-def run_migrations(migrate_schema_sql_filepath=None):
-    """Apply any pending migrations. Returns True on success, False on failure."""
+def maybe_regenerate_derived_tables(ch_props, derived_table_sql_filepath=None):
+    """Regenerate derived tables if generate_derived_tables.sql's own version differs from the
+    database's current info.derived_table_schema_version. Runs unconditionally as a blanket
+    policy whenever that's the case — covers both a derived-table rebuild required by a
+    base-table migration and a derived-table-only version bump with no corresponding
+    migrate_schema.sql section."""
+    filepath = derived_table_sql_filepath or DEFAULT_GENERATE_DERIVED_TABLES_SQL
+    if not os.path.exists(filepath):
+        raise RuntimeError(f"Could not find generate_derived_tables.sql at {filepath}")
+
+    expected_version = get_derived_tables_sql_version(filepath)
+    current_version = get_current_derived_table_schema_version(ch_props)
+    if expected_version == current_version:
+        print(f"Derived tables are already at derived_table_schema_version "
+              f"{current_version}. Nothing to do.")
+        return
+
+    print(f"Regenerating derived tables: derived_table_schema_version {current_version} -> "
+          f"{expected_version}")
+    optimize_backoff_secs = os.environ.get('CLICKHOUSE_OPTIMIZE_BACKOFF_SECS', '0')
+    _run_client(ch_props, [
+        '--multiquery',
+        '--queries-file', filepath,
+        '--param_optimize_backoff_secs', optimize_backoff_secs,
+    ])
+    # generate_derived_tables.sql ends with an ALTER TABLE info UPDATE for
+    # derived_table_schema_version, which is an async mutation.
+    wait_for_mutations(ch_props)
+    print(GREEN + f"Regenerated derived tables at derived_table_schema_version "
+                  f"{expected_version}" + END)
+
+
+def run_migrations(migrate_schema_sql_filepath=None, regenerate_derived_tables=False,
+                    derived_table_sql_filepath=None):
+    """Apply any pending migrations (and optionally regenerate derived tables). Returns True on
+    success, False on failure."""
     config_path = None
     try:
         filepath = migrate_schema_sql_filepath or DEFAULT_MIGRATE_SCHEMA_SQL
@@ -229,12 +292,14 @@ def run_migrations(migrate_schema_sql_filepath=None):
 
         if not pending:
             print(f"Database is already at db_schema_version {current_version}. Nothing to do.")
-            return True
+        else:
+            print(f"Current db_schema_version: {current_version}. "
+                  f"{len(pending)} migration(s) to apply.")
+            for section in pending:
+                apply_section(ch_props, section)
 
-        print(f"Current db_schema_version: {current_version}. "
-              f"{len(pending)} migration(s) to apply.")
-        for section in pending:
-            apply_section(ch_props, section)
+        if regenerate_derived_tables:
+            maybe_regenerate_derived_tables(ch_props, derived_table_sql_filepath)
 
         return True
     except Exception as e:
@@ -251,9 +316,24 @@ def main():
         '--migrate-schema-sql',
         default=None,
         help="Path to migrate_schema.sql (defaults to the file next to this script)")
+    parser.add_argument(
+        '--regenerate-derived-tables',
+        action='store_true',
+        help="After applying migrations, also regenerate derived tables if "
+             "generate_derived_tables.sql's version differs from the database's current "
+             "derived_table_schema_version. Off by default: institutional deployments that run "
+             "derived-table regeneration as a separate manual step should leave this unset.")
+    parser.add_argument(
+        '--derived-tables-sql',
+        default=None,
+        help="Path to generate_derived_tables.sql (defaults to the file next to this script's "
+             "parent directory); only used with --regenerate-derived-tables")
     args = parser.parse_args()
 
-    success = run_migrations(args.migrate_schema_sql)
+    success = run_migrations(
+        args.migrate_schema_sql,
+        regenerate_derived_tables=args.regenerate_derived_tables,
+        derived_table_sql_filepath=args.derived_tables_sql)
     sys.exit(0 if success else 1)
 
 

src/main/resources/db-scripts/clickhouse/migrate/migrate_schema.sql

@@ -9,8 +9,9 @@
 --   <SQL statements for this section>
 --
 -- Rules for writing a section:
---   1. Sections must be idempotent (IF EXISTS / IF NOT EXISTS everywhere) — ClickHouse has no
---      transactions, so a crash mid-section must be safe to re-run from the top.
+--   1. Sections should be idempotent wherever possible (IF EXISTS / IF NOT EXISTS) — ClickHouse
+--      has no transactions, so a crash mid-section ideally leaves it safe to re-run from the top.
+--      Not always achievable; use judgment, and note in the description when a section isn't.
 --   2. Never DROP a column in the same section that reads from it.
 --   3. For ORDER BY / primary-key changes: create a new table, INSERT ... SELECT, RENAME —
 --      ClickHouse cannot ALTER these in place.
@@ -22,6 +23,10 @@
 --      that must be computed outside ClickHouse). migrate_db.py runs this section's SQL (if any)
 --      first, then calls the matching hardcoded Python function registered in
 --      migrate_db.py's CUSTOM_MIGRATIONS dict, keyed by this section's version.
+--   6. Any section that changes base tables must also bump ../generate_derived_tables.sql's
+--      version (even with no semantic change to that file), since a base-table change may affect
+--      derived tables in ways that aren't obvious to every author. derived_table_schema_version
+--      may also bump on its own with no corresponding section here.
 --
 -- Sections for versions already recorded in the target database's info.db_schema_version are
 -- skipped automatically — do not remove or renumber old sections once released.

test/test_db_version.sh

@@ -99,12 +99,20 @@ migrate_schema_sql_db_version=$(grep -oE '^##[[:space:]]*db_schema_version:[[:sp
 generate_derived_tables_sql_version=$(head -n 1 ${GENERATE_DERIVED_TABLES_SQL} | \
     sed -E 's/^-- version ([0-9]+\.[0-9]+\.[0-9]+) of derived table schema and data definition/\1/')
 
+# --- generate_derived_tables.sql: its own trailing "ALTER TABLE info UPDATE
+# derived_table_schema_version = 'X'" statement, which is what actually gets written to the
+# database when this script runs — must agree with the header above ---
+generate_derived_tables_sql_info_update_line=$(grep 'ALTER TABLE info UPDATE derived_table_schema_version' ${GENERATE_DERIVED_TABLES_SQL} | tail -n 1)
+find_delimited_substrings "$generate_derived_tables_sql_info_update_line" "'" "'"
+generate_derived_tables_sql_info_update_version=${found_delimited_substrings[0]}
+
 echo "pom.xml db.version is $pom_db_version"
 echo "schema.sql db_schema_version is $schema_sql_db_version"
 echo "migrate_schema.sql highest db_schema_version section is $migrate_schema_sql_db_version"
 echo "pom.xml derived_table.version is $pom_derived_table_version"
 echo "schema.sql derived_table_schema_version is $schema_sql_derived_table_version"
 echo "generate_derived_tables.sql header version is $generate_derived_tables_sql_version"
+echo "generate_derived_tables.sql info UPDATE version is $generate_derived_tables_sql_info_update_version"
 
 if [ "$pom_db_version" == "$schema_sql_db_version" ] &&
         [ "$schema_sql_db_version" == "$migrate_schema_sql_db_version" ] ; then
@@ -113,7 +121,8 @@ else
     db_versions_all_match="no"
 fi
 if [ "$pom_derived_table_version" == "$schema_sql_derived_table_version" ] &&
-        [ "$schema_sql_derived_table_version" == "$generate_derived_tables_sql_version" ] ; then
+        [ "$schema_sql_derived_table_version" == "$generate_derived_tables_sql_version" ] &&
+        [ "$generate_derived_tables_sql_version" == "$generate_derived_tables_sql_info_update_version" ] ; then
     derived_table_versions_all_match="yes"
 else
     derived_table_versions_all_match="no"