# Attribute Extraction — shared foundation (design)

> **STATUS: DESIGN ONLY, NOT SCHEDULED (written 2026-05-29).**
> A forward-looking foundation doc. Nothing here is built. It exists so
> that when we DO pick up cross-retailer merge / compatibility checks /
> price-anomaly work, we start aligned instead of re-deriving this.

## What it is

A single deterministic layer that reads structured **attributes** out of
a part's messy retailer title (`Product.description`) plus its `type`,
and stores them so multiple features can consume them. Think
"platform = AR-15, caliber = 5.56, length = 16in, finish = FDE" pulled
out of `"Aero Precision 16\" 5.56 NATO M4E1 Barrel, FDE Cerakote"`.

It is the **shared keystone** under several features we already want.
Build it once, reuse it everywhere (including the sister sites,
[[sister-sites-shared-infrastructure]]).

## Why it exists — the consumers

| Consumer | What it needs from this layer |
|---|---|
| **Cross-retailer merge** ([[feature-cross-retailer-merge]]) | The distinguishing attributes (caliber, length, color/finish, config) that decide two listings are the SAME product vs different variants. Title alone over-merges variants (16" vs 18" barrel) and under-merges reworded dupes. This is the hard guard that makes merge safe. |
| **Build-compatibility checker** ([[feature-build-compatibility-checker]]) | Platform + caliber + buffer/config to warn on mismatches (.308 mag + 5.56 upper, 9mm lower + 5.56 upper, AR-10 BCG in an AR-15). Its memory already names "rule-based attribute extraction" as Phase 1. |
| **Price-anomaly categorization** ([[feature-price-anomaly-categorization]]) | Attribute context sharpens per-category outlier detection (a "complete upper" with no barrel-length/caliber is suspect; a $5 item tagged as an upper is mis-binned). |
| **"Most Popular" / product identity** ([[feature-popular-in-builds-sort]]) | A product-group key of (brand, type, attributes) so popularity counts per PRODUCT, not per retailer-row. |

Three-plus features lean on the same extraction. That is the whole
argument for treating it as a foundation rather than building a
one-off matcher inside the merge feature.

## Design principles (inherited from the rule engines we already have)

This is a close sibling of `scanner-ng/ngscan/reclassify.py` (the
category reclassifier) and `hide_rules.py`. Follow their proven shape:

1. **Deterministic, explainable rules over fuzzy/ML.** Discerning users;
   precision over recall. A wrong attribute becomes a wrong merge or a
   wrong compatibility warning, which erodes trust. Every extracted
   value should trace to a named rule (same as `reclass_rule`).
2. **Suggest-first, never fabricate.** "Unknown" is a first-class value.
   Better to leave an attribute null than to guess. This is what keeps
   merge safe: **merge fires only when the distinguishing attributes are
   KNOWN and equal; an unknown blocks the merge** (conservative by
   default, widen recall later).
3. **Whole-word / phrase matching.** Reuse the tested tokenizer from
   `reclassify.py` (`_has` / `_norm`) and heed
   [[reclassify-has-multiword-bug]] (the Py3.7 `re.escape` multi-word
   break) — escape per-token, not the whole phrase. There is already a
   unit-test suite pattern to copy (`scanner-ng/tests/`).
4. **Per-type schemas.** Attributes are type-specific. Do not force one
   flat schema. A barrel has length/caliber/profile/gas-length/thread; a
   magazine has caliber/capacity; a trigger has stage/pull-weight; an
   optic has type/magnification/reticle. The [[scope-ar15-and-ar10]]
   rule applies: platform is AR-15 AND AR-10, never caliber-gated.

## The attribute taxonomy (first cut, per type)

Types are fixed (`scanner-ng/ngscan/parttypes.py`). High-value
attributes by type:

- **Universal (extract for everything first):** `platform`
  (ar15 / ar10 / ar9 / multi / na), `caliber`/chambering, `brand`
  (reuse the manufacturer list from
  [[manufacturer-direct-prepend-brand]]), `finish`/`color`.
- **barrel:** length, caliber, gas-system length (pistol/carbine/mid/rifle),
  profile (gov/pencil/heavy/SOCOM), thread pitch, material.
- **bcg:** platform (AR-15 vs AR-10/.308), coating (nitride/phosphate/NiB/Cerakote),
  full-auto vs semi profile.
- **buffer:** weight (H1/H2/H3/...), length (carbine/rifle/A5).
- **rail (handguard):** length (in), mount (M-LOK / KeyMod / quad / picatinny),
  free-float vs drop-in, barrel-nut style.
- **mag:** caliber, capacity (10/20/30/...), material (polymer/steel/aluminum).
- **muzzle:** kind (brake/comp/flash-hider), thread pitch, caliber.
- **trigger:** stage (single/two-stage), pull weight, curved/flat.
- **optic / sight:** kind (red-dot/LPVO/prism/iron), magnification, reticle.
- **stock / grip / ch / gastube / lpk / lbk / tool / clean:** mostly
  platform + brand + finish; lighter schemas.

Start with the **smallest set that unlocks the consumers**: `platform` +
`caliber` (drives both merge guards and the compatibility checker),
then `length` for barrels and `finish`/`color`. Everything else is
incremental.

## Where it runs, and where the output lives (the storage decision)

This is the one place it deliberately **diverges** from the reclassify
engine, so call it out (cf. [[hide-vs-reclassify-storage-split]]: storage
location follows who QUERIES it).

- **Engine:** a scanner-ng module, sibling to `reclassify.py`. Runs over
  `Product.description` + `type`. Cheapest to enrich **at scan time**
  (no extra pass; attributes stay fresh with the row), with a
  re-derivable nightly sweep as the fallback for rule changes.
- **Output storage:** unlike reclassify (Pi-side SQLite, no FK because
  nothing on prod queries it), the extracted attributes **must be
  prod-side and queryable**, because the consuming features (merge,
  compatibility) read them at request time joined to `parts`. Proposal:
  a `parts_attrs` table on the prod DB with a few **indexed scalar
  columns** for the hot join keys (`platform`, `caliber`) plus a typed
  **JSON blob** for the long tail of per-type attributes. Key it for
  durability on the stable scanner upsert `url` (parts.id churns on
  OOS-purge / URL change, same lesson as the popularity sort), with
  parts.id available for live joins.
- **Rules:** live in code/files, versioned, self-bootstrapping into the
  dashboard for a **suggest-first preview** (exactly the
  `reclassify_rules.py` / `hide_rules.py` pattern: see what a rule would
  extract on a sample before any write, human can correct/override).

## Identity / keying

Re-uses the popularity-sort lesson: key by the stable `url` where
durability matters; use parts.id for live request-time joins. Once this
layer exists, the **product-group** derivation for
[[feature-cross-retailer-merge]] is `(brand, type, platform, caliber,
+distinguishing attrs)` -> a `product_group` id.

## Phasing

- **P0 (this doc):** taxonomy + storage decision agreed. Done when picked up.
- **P1:** engine + the universal pair (`platform`, `caliber`) extracted
  at scan time -> `parts_attrs`; dashboard suggest-first preview.
  Immediately usable as the compatibility checker's .308-vs-5.56 guard
  and as the conservative merge guard.
- **P2:** per-type attributes (barrel length/profile/gas, mag capacity,
  finish/color), normalization, a curated-override table for what rules
  can't get.
- **P3:** `product_group` derivation -> feeds cross-retailer merge and
  per-product popularity.

## Open questions (deferred until we pick this up)

- Scan-time enrich vs nightly sweep (lean scan-time, sweep as fallback).
- `parts_attrs`: indexed columns + JSON (lean) vs fully columnar.
- **Multi-fit parts:** a part that fits 5.56 AND .300 BLK, or a
  platform-agnostic grip. Attributes are sometimes a SET, and
  `multi`/`na` must be valid values that do NOT block a merge or trip a
  compatibility warning. Get this wrong and we over-merge or cry wolf.
- Confidence threshold for auto-store vs suggest-only.
- Does the engine live in scanner-ng (with the other rule engines) while
  its output table is prod-side and its consumers are site-ng? (Current
  lean: yes; this doc sits in site-ng/docs because the consumers are
  site features, but the engine is scanner-ng-flavored.)

## Cross-references

- **Code siblings:** `scanner-ng/ngscan/reclassify.py` (rule engine to
  mirror), `reclassify_rules.py` + `hide_rules.py` (runner + storage
  split + dashboard preview), `parttypes.py` (type vocab),
  `models.py` (`Product`, and the `reclass_suggested`/`reclass_rule`
  suggest-only precedent).
- **Memories:** [[feature-cross-retailer-merge]],
  [[feature-build-compatibility-checker]],
  [[feature-price-anomaly-categorization]],
  [[feature-popular-in-builds-sort]],
  [[hide-vs-reclassify-storage-split]],
  [[reclassify-has-multiword-bug]],
  [[manufacturer-direct-prepend-brand]], [[scope-ar15-and-ar10]],
  [[sister-sites-shared-infrastructure]].
