Skip to content

Per-retailer adapter framework, delivery columns, new calibers - #26

Open
laveez wants to merge 6 commits into
masterfrom
feature/scraper-overhaul-and-delivery
Open

Per-retailer adapter framework, delivery columns, new calibers#26
laveez wants to merge 6 commits into
masterfrom
feature/scraper-overhaul-and-delivery

Conversation

@laveez

@laveez laveez commented May 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

  • Refactors scrapers into per-retailer adapters (approach C) with shared parsers in scripts/lib/. Each retailer is its own file implementing discover() and delivery(); base behaviors fall back to shared parsers.
  • Adds discovery + auto-prune phase so the URL list is no longer hand-maintained — products absent from a retailer's catalog for >14 days get removed.
  • Adds delivery scraping + retailers.json with delivery rules. Parser prefers door-delivery prices (Finnish law requires kotijakelu for ammo — pakettiautomaatti is not a legal option).
  • Adds Delivery and Total + Delivery columns to the price table with a matching Max total + delivery filter.
  • Closes Add support for .222 Remington and .30-06 Springfield cartridges #10 by adding .222 Remington, .30-06 Springfield, and 6.5 Creedmoor (caught from a patruunat.fi parity audit) with hand-seeded starter products.
  • 23 of 33 retailers now have real discover() adapters; 10 remain as legacy stubs (documented below).

Technical details

  • Adapter framework lives in scripts/lib/ (types, adapter, parsers/, nontox, variants, delivery, products-store, discovery helpers). Adapters are typically ~35 lines and use the shared discoverByPattern + fetchDelivery helpers — only URL knowledge per retailer.
  • Auto-prune only kicks in for retailers whose adapter implements discover() — legacy stubs return null so their URLs are never auto-removed.
  • New CLI flags: --retailer=<name>, --caliber=<name>, --skip-discovery, --skip-delivery.
  • discover-variants.ts and add-nontox-products.ts folded into the orchestrator.
  • Search and product URL patterns were HTTP-verified against the live sites (or product URL pattern derived from existing entries in products.json) before each adapter was committed — no speculative paths.

Retailers with real adapters (23)

Tier Retailers
WordPress / Magento (8) Asenurkka, Asepaja Vuorela, Erakolmio, Oulun Ase, Savon Mutka, Seponurheilujapatruuna, Uittokalusto, Verkkoaita
Confirma E-COM (9) Aawee, Ase ja Erä, Erakala, Era Haijaa, Greentrail, Kurre, Metso Ase, Ruoto, Sissos
Custom platforms (6) Arcis (Wix), Iron Point (osCommerce), Kärkkäinen, Motonet, Olkkonen (Wix), Pyörä-Paja

Retailers left as legacy stubs (10) — and why

Retailer Reason Path to a real adapter
Ahtihuvila Old CGI webio2kauppa script; product listings not exposed via crawlable category pages. Reverse-engineer the CGI's naytasivu= parameter scheme, or wait for them to modernize.
Asejaosa PrestaShop install with non-default search routing — /search and /catalogsearch both return 404. Identify the actual search/category URL form (sitemap or a manual site walk).
Asetalo Custom storefront; no obvious search URL or browsable caliber-level category. Inspect their search box <form action> for the real endpoint.
Riistamaa Custom PHP catalog at /cat/product_catalog.php?c=NNN; caliber → category-ID mapping unknown without walking every category. Walk all c= IDs and record which contains which caliber.
Viranomainen Login-walled per project memory; anonymous browse not productive. Out of scope until login support lands (deferred per the spec).
Commando Homepage suggests B2B/security industry; no public consumer ammo shop. Confirm in person whether they actually sell consumer ammo online.
Eräkellari Empty <title>, no detected platform; storefront structure unconfirmed. Manual site review.
Välineet Kuntoon Custom "dataflow" CMS; search form points at /haku but query parameter unclear. Inspect their search form's hidden inputs.
XXL Large general retailer; search structure not consumer-API-friendly and likely uses XHR APIs. Either reverse-engineer their search API or skip — they're not ammo specialists.
Ähtärin Ase No detected platform, no static search form. Manual site review.

Risk notes

  • Confirma E-COM sites (Aawee, Ase ja Erä, Erakala, Era Haijaa, Greentrail, Kurre, Metso Ase, Ruoto, Sissos) render their search results entirely client-side. The shared discovery helper uses Playwright's domcontentloaded + a 1.5 s settle wait — sufficient in CI Linux for Aawee + Sissos so far, but if a Confirma retailer consistently shows zero URLs in CI logs, the wait may need tuning before its existing URLs hit the 14-day prune threshold.
  • Adapters whose shipping URL couldn't be located via static footer scan (Asenurkka, Erakolmio, Verkkoaita, Arcis, Olkkonen, Pyörä-Paja) return null from delivery(). retailers.json keeps its existing delivery: null entry; UI shows ? until a real URL is confirmed.

Test plan

  • Build passes (npm run build)
  • Lint clean (npm run lint)
  • UI renders new columns + tooltip + Max total+delivery filter
  • All 8 caliber tabs load (incl. .222 Rem, .30-06 Springfield, 6.5 Creedmoor)
  • Mobile layout (≤480px) hides Brand/Type/Total/Delivery, keeps Total+Delivery
  • CI scrape on master runs without crashing (full end-to-end)
  • At least one delivery rule scraped successfully on each commit's adapters
  • --retailer=<Name> --dry-run runs only the targeted adapter

@laveez
Feature: Per-retailer adapter framework, delivery columns, new calibers
3bc3de0 
- Refactor scraper into adapter framework (scripts/lib + scripts/adapters):
  shared parsers (json-ld, meta-tags, dom, text-regex), nontox + variants
  helpers, products/retailers store. Each retailer is its own adapter file
  implementing discover() and delivery(); base behaviors fall back to shared
  parsers. Legacy stubs preserve current URL-update behavior for retailers
  without bespoke discovery yet.
- Add discovery + auto-prune phase: adapters with discover() seed new URLs
  and remove products absent for >14 days; legacy retailers are untouched.
- Add delivery scraping + retailers.json: delivery rules per retailer
  (method, cheapestPrice, freeOverThreshold). Parser prefers door-delivery
  prices since Finnish law requires kotijakelu (ID-checked door delivery)
  for ammunition — pakettiautomaatti is not a legal option.
- Add Delivery and Total + Delivery columns to the price table; new
  "Max total + delivery" filter. Tooltip shows the delivery method per row.
  Mobile keeps Total + Delivery and hides Total/Brand/Type/Delivery.
- Add .222 Remington, .30-06 Springfield (closes #10), and 6.5 Creedmoor
  calibers with hand-seeded products. Discovery crawler fills more over time.
- Register new retailer adapters as legacy stubs: Savon Mutka, Verkkoaita,
  XXL, Commando, Olkkonen, Eräkellari, Ähtärin Ase, Välineet Kuntoon,
  Pyörä-Paja. Adapters get filled in incrementally.
- Implement Aawee + Sissos discover() and delivery() as reference adapters.
- Fold discover-variants.ts and add-nontox-products.ts into orchestrator.
- New CLI flags: --retailer=<name>, --caliber=<name>, --skip-discovery,
  --skip-delivery for partial runs and debugging.
@laveez laveez self-assigned this May 7, 2026
laveez added 5 commits May 8, 2026 01:34
@laveez
Refactor: Verified-only adapters, null fabricated delivery rules
9c26d13 
- Null all delivery rules in retailers.json. Initial commit shipped specific
  Matkahuolto kotijakelu prices and free-over thresholds for ~24 retailers
  that were guessed, not measured. Users comparing total + delivery would
  have read fabricated numbers. UI now shows '?' for delivery cost until the
  delivery scraper populates real values from each retailer's shipping page.
- Add lib/discovery.ts: discoverByPattern + fetchDelivery helpers so each
  adapter is just URL paths + a regex + a shipping URL.
- Add motonet + karkkainen adapters using verified search URLs:
  /haku?q= for Motonet, /verkkokauppa?search= for Kärkkäinen. (HTTP 200
  confirmed before commit.)
- Refactor aawee + sissos to use the new helpers.
- Keep all other retailers as legacy stubs. Adapters get filled in
  incrementally with verified URLs — guessed category paths produce silent
  failures and noisy CI logs.
@laveez
Feature: Implement WordPress/Magento retailer adapters
b175c41 
- Asenurkka, Asepaja Vuorela, Erakolmio, Oulun Ase, Savon Mutka,
  Seponurheilujapatruuna, Verkkoaita: WooCommerce search via
  /?s=<query>&post_type=product. Product URL patterns derived from
  existing products.json entries (e.g. /tuote/<slug>/, /kauppa/.../).
- Uittokalusto: Magento search via /catalogsearch/result/?q=<query>,
  product URLs as /<slug>.html.
- Asepaja Vuorela regex tightened to /kauppa/patruunat/ to filter out
  non-ammo results (gun safes, optics, accessories).
- Search URLs HTTP-200 verified before commit; product URL regexes
  match real URLs from products.json.
- Delivery URLs verified where present (Asepaja Vuorela
  /asiakaspalvelu/toimitus, Savon Mutka /sopimusehdot, Sepon
  /asiakaspalvelu/toimitus, Uittokalusto /toimitustavat, Oulun Ase
  /page/1/toimitusehdot). Asenurkka, Erakolmio, Verkkoaita have no
  publicly findable shipping page so delivery() returns null.
@laveez
Feature: Implement Confirma E-COM retailer adapters
779f0b2 
- Ase ja Erä, Erakala, Era Haijaa, Greentrail, Kurre, Metso Ase, Ruoto:
  Confirma E-COM platform (same as Aawee, Sissos) using /haku?q=<query>
  for search. Product URL pattern is /.../p/<ean-or-id>/, matching the
  shape of the existing entries in products.json.
- All search and shipping URLs HTTP-200 verified before commit.
  Shipping page is /toimitusehdot for all seven retailers.
- Note: Confirma search results render via JS, so static curl shows
  no products. Discovery relies on Playwright's domcontentloaded +
  settle wait. If CI later shows zero URLs for a retailer, the
  shared discovery helper's wait timings may need adjustment.
@laveez
Feature: Implement custom-platform retailer adapters
00178b2 
- Arcis (Wix): /shop?searchTerm=<query>, product URLs at
  /product-page/<slug>. Small catalog.
- Iron Point (osCommerce): category browse via
  /fi/pistoolin-patruunat-c-6_7.html (pistol) and
  /fi/kivrin-patruunat-c-6_54.html (rifle), product URLs at
  /shop/product/<slug>. Caliber-level categories not exposed, so
  pistol/rifle calibers share the same listing.
- Olkkonen (Wix): /fi/shop?searchTerm=<query>, products at
  /fi/product/<slug>/<id>.
- Pyörä-Paja (custom): /search/?q=<query>, products at
  /product/<id>. Mostly a bike shop, but their hunting section
  occasionally lists ammunition.
- All search and product URL patterns HTTP-verified against the
  live sites. Shipping URLs not located via static footer scan for
  any of the four; delivery() returns null until a real URL is
  confirmed.
- Remaining legacy stubs (10): Ahtihuvila (old CGI catalog),
  Asejaosa (PrestaShop search 404), Asetalo (no obvious search),
  Riistamaa (unknown caliber→category-ID mapping), Viranomainen
  (login-walled), Commando, Eräkellari, Välineet Kuntoon, XXL,
  Ähtärin Ase (storefront structure not confirmed). Each is
  documented in the PR description.
@laveez
Refactor: Simplify adapter framework and orchestrator
64fd91e 
- Add searchAdapter() factory in lib/adapter.ts with bySearch() and
  byCategory() URL-builder helpers. All 23 adapters now collapse to
  ~10-line declarations of (name, baseUrl, productUrlPattern,
  categoryUrls, shippingPath). Removes ~500 lines of boilerplate.
- Extract CALIBER_QUERIES_PATRUUNA and CALIBER_QUERIES_SHORT to
  lib/queries.ts; 22 identical 8-entry objects across adapter files
  become one import. Fixes a Sissos typo where 30-06 was indexed as
  "30-06+patruuna" instead of "30-06+sprg+patruuna" matching the
  others.
- Refactor lib/adapter.ts defaultExtract: extract enrich() helper so
  the three success blocks (fast pass, post-SPA-wait pass, text-regex
  fallback) share the same hydration code path. Parallelize the
  availability/non-toxic/variants page.evaluate calls (Promise.all)
  to drop 2 IPC round-trips per successful URL. Replace the fixed
  3-second waitForTimeout with waitForSelector against common
  price-content selectors (JSON-LD, og:price, .price, .product-price,
  .woocommerce-Price-amount) so server-rendered pages don't pay the
  full SPA-hydration wait.
- Refactor lib/discovery.ts: replace unconditional waitForTimeout
  with waitForLoadState('networkidle', {timeout}) so the discovery
  phase short-circuits when the page is actually quiet. Drops up to
  ~6 minutes of pure sleep across a full 264-combo discovery run.
- Drop the typed-number doorScore/pickupScore in delivery.ts in favor
  of boolean nearDoorHint/nearPickupHint — the values were only ever
  0/1 anyway.
- Split fetch-prices.ts applyResults (was 128 lines, 5+ levels deep)
  into applyNonToxicFlag / applyVariantPricing / applyAvailability /
  applyScrapedTotal. Main loop is now a flat 4-call sequence per URL.
- Replace fetch-prices.ts hand-rolled chunked-parallel pattern (3
  occurrences) with a worker-pool runBatched() helper in
  lib/concurrency.ts. Eliminates head-of-line blocking: a slow page
  no longer stalls up to 4 idle workers in its batch.
- Replace applyResults' per-URL O(products) flatMap().find() retailer
  lookup with a urlToRetailer Map built once before the price phase.
- Use todayIso() from lib/format.ts in fetchDelivery instead of an
  inline `new Date().toISOString().slice(0,10)` (helper already
  existed for exactly this).
- Wrap browser lifetime in try/finally in main() so chromium closes
  even when a phase throws.

Skipped: PriceTable.tsx parser/formatter dedup with scripts/lib —
parseQuantity and formatEur diverge in subtle ways from the script-
side equivalents (non-digit handling, conditional thousand
separator). Consolidating would either silently change UI output or
require a flagged shared function for marginal LoC gain.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@laveez laveez

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add support for .222 Remington and .30-06 Springfield cartridges

1 participant

@laveez