domain-probe-ai

Extract structured organization details from any website URL — no third-party enrichment APIs. Combines schema.org parsing, regex heuristics, and Gemini 2.5 Flash-Lite for structured extraction.

Given a URL it returns a flat JSON ready to autofill an onboarding form:

{
  "brand_name":    "",
  "legal_name":    "",
  "company_phone": "",
  "company_email": "",
  "vat_number":    "",
  "tin_number":    "",
  "address_line":  "",
  "city":          "",
  "state":         "",
  "country":       "",
  "zip":           "",
  "country_iso2":  ""
}

Missing fields are returned as null — never an error, always a complete object.

Installation

Prerequisites

Requirement	Version	Check
Python	3.11 or higher	`python3 --version`
pip	any recent	`pip3 --version`
Git	any	`git --version`
Gemini API key	—	Get one free →

Step 1 — Clone the repository

git clone https://github.com/your-org/org-extractor.git
cd org-extractor

Step 2 — Install the package

pip3.11 install -e .

This installs the package and all its dependencies. The -e flag means "editable" — you can modify the source and changes take effect immediately without reinstalling.

To also install dev/test dependencies:

pip3.11 install -e ".[dev]"

Verify the install:

python3.11 -c "from org_extractor import extract_org_details; print('Installation OK')"

Step 3 — Get a Gemini API key

Go to aistudio.google.com
Sign in with a Google account
Click Get API key in the left sidebar
Click Create API key → copy it (starts with AIza...)

The free tier includes 20 requests per day which is enough for testing. For production use, enable billing at aistudio.google.com/plan_information.

Step 4 — Configure the API key

Option A — Environment variable (recommended)

export GEMINI_API_KEY=AIzaSy...your_key_here

To make it permanent (survives terminal restarts):

# macOS / Linux (zsh)
echo 'export GEMINI_API_KEY=AIzaSy...your_key_here' >> ~/.zshrc
source ~/.zshrc

# macOS / Linux (bash)
echo 'export GEMINI_API_KEY=AIzaSy...your_key_here' >> ~/.bashrc
source ~/.bashrc

Option B — .env file

A .env file is already provided in the project root. Open it and replace the placeholder with your real key:

GEMINI_API_KEY=AIzaSy...your_key_here

The library loads .env automatically on import — no manual export needed.

Step 5 — Verify everything works

org-extractor https://linguidoor.de

You should see extraction progress logs and a JSON result. Full output takes 5–15 seconds depending on the site.

Usage

Command line

# Full extraction (uses Gemini)
org-extractor https://example.de

# See tier-by-tier progress
org-extractor https://example.de --verbose

# Without Gemini — schema.org + regex heuristics only (no API key needed)
org-extractor https://example.de --no-gemini

# Save result to a file
org-extractor https://example.de > result.json

# Pass API key directly (overrides env var)
org-extractor https://example.de --api-key AIzaSy...

# Use custom config files
org-extractor https://example.de \
  --fields-config path/to/my_fields.json \
  --slugs-config  path/to/my_slugs.json

Python API

from org_extractor import extract_org_details

# Uses GEMINI_API_KEY from environment automatically
result = extract_org_details("https://example.de")

print(result["brand_name"])    # "Acme GmbH"
print(result["city"])          # "Berlin"
print(result["country_iso2"])  # "DE"

Pass the key explicitly:

result = extract_org_details(
    "https://example.de",
    gemini_api_key="AIzaSy...",
)

Use custom config files:

result = extract_org_details(
    "https://example.de",
    fields_config="path/to/fields_config.json",
    slugs_config="path/to/slugs_config.json",
)

Works in Jupyter notebooks — async context is handled automatically.

How it works

URL
 │
 ▼
Tier C — Homepage
  • Fetch homepage HTML
  • Parse schema.org / JSON-LD / Open Graph
  • Scan footer links for legal page keywords
  ✓ Early exit if all required fields found (~1.5s)
 │
 ├─ (fields missing) ─────────────────────────────────────┐
 ▼                                                         │
Tier A — Legal pages (parallel fetch)                      │ Tier B pages
  /impressum, /legal, /imprint, /legal-notice …            │ pre-fetched here
  • Clean text via trafilatura                              │ concurrently while
  • Send to Gemini → structured JSON                        │ Tier A Gemini runs
  • Merge into result (never overwrites filled fields)      │
  ✓ Early exit if all required fields found (~4–6s)         │
 │                                                         │
 ├─ (fields missing) ◄────────────────────────────────────┘
 ▼
Tier B — Contact / About pages (already pre-fetched)
  /contact, /about, /about-us …
  • Combine text: Tier B + Tier A + homepage
  • Send to Gemini for remaining fields only
 │
 ▼
Validate & normalize
  • Phone → E.164 format (+49301234567)
  • Email → normalized lowercase
  • Country name → ISO-2 code
  • VAT / TIN validated per country
 │
 ▼
Flat JSON — all 11 fields, null for anything not found

Stop-early rule: The moment all required fields are non-null, extraction stops — Tier B is skipped if Tier A already has everything.

Tier B overlap: Tier B page fetches run in parallel with the Tier A Gemini call, so network latency doesn't stack.

Expected latency

Scenario	Time
schema.org complete on homepage	~1.5s
Tier A (Impressum) has all fields	~4–6s
Tier A + Tier B both needed	~10–14s

Configuration

`fields_config.json`

Controls which fields are extracted and the description fed to the LLM. Edit without touching any code.

{
  "required": {
    "brand_name":    "The commonly used trade name or brand...",
    "legal_name":    "The full registered legal entity name...",
    "company_phone": "The primary business contact phone number...",
    "company_email": "The primary business contact email...",
    "vat_number":    "The VAT identification number...",
    "tin_number":    "The Tax Identification Number...",
    "address_line":  "Street address excluding city/state/zip/country.",
    "city":          "The city or town.",
    "state":         "State, province, or Bundesland. Null if not applicable.",
    "country":       "Country name in English.",
    "zip":           "Postal code or ZIP code."
  },
  "optional": {}
}

required fields drive the stop-early rule. optional fields are extracted if found but never block completion.

`slugs_config.json`

Controls which URL paths are probed for legal and contact pages.

{
  "tier_a_slugs": ["/impressum", "/imprint", "/legal", "/legal-notice", "..."],
  "tier_a_link_keywords": ["impressum", "imprint", "legal notice", "..."],
  "tier_b_slugs": ["/contact", "/about", "/about-us", "..."]
}

tier_a_link_keywords match footer link text (case-insensitive) to discover legal pages with non-standard URLs.

Output fields

Field	Example	Notes
`brand_name`	`"Acme Corp"`	Trade name — from logo, title, or OG tag
`legal_name`	`"Acme Corporation Ltd"`	Full registered name
`company_phone`	`"+12025550147"`	E.164 format
`company_email`	`"info@acmecorp.com"`	Prefers `info@`, `contact@`, `hello@`
`vat_number`	`"US123456789"`	Validated via python-stdnum
`tin_number`	`"12-3456789"`	Country-specific (Steuernummer, EIN, PAN…)
`address_line`	`"123 Main Street"`	Street only
`city`	`"Springfield"`
`state`	`"Illinois"`	`null` for countries without states
`country`	`"United States"`	English name
`country_iso2`	`"US"`	Derived from `country` field + TLD fallback
`zip`	`"62701"`

Project structure

org-extractor/
├── pyproject.toml                  # Package definition and dependencies
├── README.md                       # This file
├── .env                            # Add your GEMINI_API_KEY here (auto-loaded)
├── fields_config.json              # Root copy (for development/CLI)
├── slugs_config.json               # Root copy (for development/CLI)
└── src/org_extractor/
    ├── __init__.py                 # Public API: extract_org_details()
    ├── cli.py                      # CLI entry point (org-extractor command)
    ├── orchestrator.py             # Tier loop + stop-early logic
    ├── fetcher.py                  # Async parallel HTTP fetcher
    ├── url_discovery.py            # Tier A/B URL discovery from homepage
    ├── extruct_checker.py          # schema.org / JSON-LD / OG extraction
    ├── text_extractor.py           # HTML → clean text via trafilatura
    ├── gemini_caller.py            # Gemini call + regex heuristic fallback
    ├── schema_builder.py           # Dynamic Pydantic model builder
    ├── validators.py               # Phone/email/VAT/country normalization
    ├── config_loader.py            # JSON config loader
    ├── fields_config.json          # Bundled default fields config
    └── slugs_config.json           # Bundled default slugs config

Running tests

python3.11 -m pytest tests/ -v

71 passed in 0.7s

Run a specific module:

python3.11 -m pytest tests/ -v -k "url_discovery"
python3.11 -m pytest tests/ -v -k "validators"
python3.11 -m pytest tests/ -v -k "extruct"

Troubleshooting

Installation OK not printed after install Python version may be below 3.11. Check with python3 --version and use python3.11 explicitly.

No Gemini API key — heuristics only GEMINI_API_KEY is not set in the current terminal session. Run export GEMINI_API_KEY=... and try again.

All fields null even with Gemini The site is JavaScript-rendered (React, Angular, Next.js). HTML-only fetching cannot see JS-rendered content — out of scope for v1.

robots.txt disallows for all pages The site blocks automated crawlers. Nothing can be done without violating ToS.

Pages 404 but site has a contact/legal page The site uses non-standard URL paths. Add the correct paths to tier_a_slugs or tier_b_slugs in slugs_config.json.

Gemini timed out after 15s Gemini API was slow. Falls back to heuristics only. Usually transient — retry.

Dependencies

Package	Purpose
`httpx[http2]`	Async parallel page fetching with HTTP/2
`trafilatura`	HTML → clean text extraction
`extruct`	schema.org / JSON-LD / Open Graph parsing
`beautifulsoup4`	Footer link parsing
`google-genai`	Gemini 2.5 Flash-Lite structured extraction
`pydantic`	Dynamic schema model builder
`phonenumbers`	Phone parsing and E.164 normalization
`python-stdnum`	VAT and TIN validation per country
`email-validator`	Email normalization
`pycountry`	Country name → ISO-2 conversion
`lxml`	HTML parser for BeautifulSoup
`nest_asyncio`	Async compatibility for Jupyter notebooks

Out of scope (v1)

JS-rendered pages (Playwright / Selenium)
Optional fields (logo, social handles, founded year, registration number)
Caching layer for repeat domains
Batch URL processing
Confidence scores per field

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

domain-probe-ai

Installation

Prerequisites

Step 1 — Clone the repository

Step 2 — Install the package

Step 3 — Get a Gemini API key

Step 4 — Configure the API key

Step 5 — Verify everything works

Usage

Command line

Python API

How it works

Expected latency

Configuration

`fields_config.json`

`slugs_config.json`

Output fields

Project structure

Running tests

Troubleshooting

Dependencies

Out of scope (v1)

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
src/org_extractor		src/org_extractor
tests		tests
.gitignore		.gitignore
README.md		README.md
cli.py		cli.py
fields_config.json		fields_config.json
pyproject.toml		pyproject.toml
setup.py		setup.py
slugs_config.json		slugs_config.json

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

domain-probe-ai

Installation

Prerequisites

Step 1 — Clone the repository

Step 2 — Install the package

Step 3 — Get a Gemini API key

Step 4 — Configure the API key

Step 5 — Verify everything works

Usage

Command line

Python API

How it works

Expected latency

Configuration

fields_config.json

slugs_config.json

Output fields

Project structure

Running tests

Troubleshooting

Dependencies

Out of scope (v1)

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`fields_config.json`

`slugs_config.json`