Skip to content

ENRICHMENT_STATUS.md

Latest commit

 

History

History
107 lines (80 loc) · 2.45 KB

ENRICHMENT_STATUS.md

File metadata and controls

107 lines (80 loc) · 2.45 KB
Raw

Enrichment Status Report

Generated: $(date)

Current Status

Overall Statistics

Metric Count
Total Stocks 4,315
Enriched (completed) 38
Unenriched 4,277
With Key People 35
Enriched but no Key People 3
Need People Backfill 27
Coverage 0.9%

Enrichment Jobs

Status Count
Completed 23
Failed 22
Total 45

Issues

1. Enrichment Service Database Authentication

The enrichment processor service (enrichment-processor-pr-65-234770780438.australia-southeast2.run.app) is experiencing database authentication issues:

FATAL: Circuit breaker open: Too many authentication errors (SQLSTATE XX000)

This prevents the service from processing new enrichment jobs.

2. Low Coverage

Only 0.9% of stocks have been enriched, and 4,277 stocks still need full enrichment.

Scripts Created

1. scripts/batch_enrich_until_complete.py

Python script that:

  • Checks enrichment status from local database
  • Calls the enrichment service HTTP endpoint (/enrich-batch)
  • Runs continuously until all stocks have key people
  • Includes people backfill step

Usage:

# Check status only
python scripts/batch_enrich_until_complete.py --check-only

# Run enrichment (when service is healthy)
python scripts/batch_enrich_until_complete.py --batch-size 10 --max-batches 100

2. scripts/enrich_until_complete.sh

Bash script with similar functionality.

Usage:

./scripts/enrich_until_complete.sh 10 100

Next Steps

Immediate Actions

  1. Fix Database Authentication: The enrichment service needs its database credentials updated.

    • Check Cloud Run environment variables
    • Verify Supabase credentials
    • Reset connection pool

  2. Run Batch Enrichment: Once the service is healthy, run:

    python scripts/batch_enrich_until_complete.py --batch-size 10

Alternative Approaches

If the service can't be fixed immediately:

  1. Run Enrichment Processor Locally:

    cd services
make run.enrichment-processor

  2. Use Direct Database Enrichment:

    cd analysis
python enrich_database.py --limit 100

  3. Trigger Enrichment via API:

    make enrich-api BATCH=10

Estimated Completion Time

With 4,277 stocks to enrich:

  • At 10 stocks per batch: ~428 batches
  • At 2-3 minutes per batch: ~14-21 hours
  • Recommended: Run in batches of 50 overnight