# Loom Supplier data acquisition and enrichment for PIM and ecommerce. Loom retrieves what a supplier publishes — product detail pages, series pages, listings, catalogs, datasheets — reads specifications out of it, normalizes units and wording onto a canonical dictionary, and holds everything behind a review gate until a person accepts it. The design constraint is that **no value can exist without provenance, and no value can leave without a decision**. Both are enforced in the schema and the code paths, not by convention. --- ## Why it is built this way Scraping supplier data is easy. Scraping supplier data you are willing to put in front of customers is not. The difference is entirely in what happens between "we read a number off a page" and "that number is now our published specification". Loom is mostly that middle part: | Question a buyer or a customer will eventually ask | Where Loom answers it | |---|---| | Where did this figure come from? | Every value stores the source document, its sha256, the retrieval time, the extraction method, and a locator inside the document (CSS path, or page and line for PDFs). | | Was it converted? From what? | The normalization note records the conversion, e.g. `converted 0.138in -> 3.5052mm`. The supplier's original text is retained verbatim. | | Who decided this was correct? | Review state, reviewer name, timestamp and note on every value. Bulk decisions record the filter that selected them. | | What did we choose not to publish, and why? | The export manifest lists every product held back with its reasons. | | What if the supplier changes it? | A re-crawl that finds a changed value clears the earlier review decision and returns it to the queue. A reviewer's typed correction is preserved and the change is recorded in the audit log. | | Are we allowed to collect this? | robots.txt is honoured by default; an override requires a written justification stored on the supplier and logged per request. Each supplier carries a "permission of record" field. | There is deliberately no automatic mapping of supplier wording onto canonical attributes. Suggestions are scored and displayed; a person confirms them. That one boundary is what keeps "Voltage Rating: 250 VAC" from silently becoming a DC rating in the catalog. --- ## Running it Requires Node 22.5 or newer (it uses the built-in `node:sqlite`). ```bash npm install ``` ```bash npm start ``` Then open . To see it working immediately against the bundled fixture supplier site: ```bash npm run demo ``` That registers a fictional supplier, crawls it end to end, and prints what the pipeline produced — documents by type, products, extraction methods, unmapped labels, validation findings, and the unit conversions it performed. Then start the server and browse the result. ### Tests ```bash npm test ``` 41 tests. The unit tests cover unit resolution, conversion and value parsing. The integration test boots the app, crawls the fixture site over real HTTP, parses real HTML and real PDFs, and asserts on what lands in the database — including that robots.txt was honoured, that nothing was auto-approved, that an ambiguous label stayed unmapped, and that the export gate held back everything until a person acted. --- ## The fixture supplier site `fixtures/build.js` generates a small, self-contained supplier site served at `/fixture`. Northwind Interconnect is invented, and every page says so. It exists so the pipeline can be exercised honestly without sending traffic to a real supplier's servers. It is deliberately awkward in the ways real sites are: - MX150 product pages publish JSON-LD **and** a specification table. - DT product pages publish microdata and a definition list, no JSON-LD. - One page states the same specifications in different words and imperial units (`Contact Spacing: 0.138 in`, `Temperature Range: -40 to +257 °F`). - Series pages publish a part matrix; each row becomes a product. - The MX150 datasheet says 25 mating cycles while the product page says 30. - One page uses `SMT`, which is not in the dictionary's allowed values for mounting type. - `Voltage Rating` appears with no AC/DC qualifier. - `robots.txt` disallows `/fixture/internal/`. Each of those produces a specific, visible outcome rather than a quiet wrong answer. That is the point of the fixture. --- ## How a value travels ``` robots check → rate-limited fetch → blob store (sha256) → classify document ↓ supplier rules → JSON-LD → microdata → spec tables → part matrices → PDF text ↓ merge readings per label (keep all, rank by trust, flag disagreements) ↓ crosswalk lookup: supplier label → canonical attribute [human-curated] ↓ parse value: qualifier, range, condition, unit → canonical unit ↓ validate against the dictionary: type, unit family, range, allowed values ↓ review: accept / reject / edit [human decision] ↓ export gate: approved product + accepted values + no open errors ``` ### Extraction tiers and confidence Confidence is a fixed weight per method, defined in one place (`src/config.js`). It describes how certain we are that we **read the source correctly**. It is never a claim that the supplier's published figure is right. | Method | Weight | | |---|---|---| | Manual entry | 1.00 | a reviewer typed it | | Supplier rule | 0.98 | a person wrote the selector for this site | | JSON-LD | 0.95 | the supplier published it for machines | | Microdata | 0.90 | machine-readable but mixed into presentation | | Definition list | 0.85 | explicit term/definition markup | | Spec table / meta | 0.80 | two-column table under a specifications heading | | Generic table | 0.72 | two-column table with no context | | Classified link | 0.70 | a datasheet or CAD link identified by its text | | Inline `Label: value` | 0.62 | text pattern inside a specifications block | | PDF table | 0.55 | column-gap split in a datasheet | | PDF text line | 0.45 | `Label: value` line in a datasheet | | Heuristic | 0.35 | anything else | Where the same label is read from several documents, all readings are kept. The highest-trust one is current; the others are shown as alternatives, and a reviewer can promote one, because a review decision outranks trust. ### Units `src/lib/units.js` is the part most likely to be wrong in a system like this, so it is strict: - Symbol lookup is **case-sensitive first**. `mΩ` is a milliohm and `MΩ` is a megohm; lowercasing them together is a real defect that this code avoids. - Context-dependent tokens (`oz`, `lb`) resolve only when the attribute declares a unit family. `oz` in a force context is ounce-force; on its own it is ounce-mass, which is documented rather than guessed. - Temperature is affine and never travels through a multiplicative factor. - A unit from the wrong family is refused with a note, not coerced. - AWG is handled properly, including aught sizes (`4/0` → gauge −3). ### What "completeness" means Required dictionary attributes holding a non-rejected value, divided by the number of required attributes. Nothing else. The dashboard shows the definition next to the number. ### What "export ready" means The product is approved or published, has no open error-severity findings, and every required attribute has an accepted value. --- ## Layout ``` server.js Express app, static hosting, fixture site src/ config.js Ports, paths, politeness defaults, confidence weights db/ schema.sql Tables and the ranked-value view dictionary.js 58 canonical attributes, 184 seed crosswalk entries seed.js Idempotent, non-destructive seeding lib/ units.js Unit registry, resolver, converter, AWG normalize.js Label and value parsing robots.js RFC 9309 parsing and matching fetcher.js Polite acquisition blobs.js Content-addressed store classify.js Document type classification html.js Parsing helpers and CSS locators extract/ jsonld.js microdata.js tables.js links.js pdf.js rules.js index.js Layer merge, corroboration, conflict detection pipeline/ runner.js Crawl orchestration and logging persist.js Writes, including the review-protection rules mapping.js Crosswalk resolution and suggestions quality.js Validation, completeness, readiness review.js Review actions and mapping backfill export.js Gated export and manifest suppliers.js Supplier records and validation routes/api.js REST API public/ Front end: vanilla ES modules, no build step fixtures/ Fixture site generator and a small PDF writer test/ Unit and end-to-end tests data/ SQLite, blob store, exports (gitignored) ``` The front end has no build step and no framework by choice: it is a dense data tool, the views are tables, and being able to read the shipped source is worth more here than a component library. --- ## Configuration Environment variables, all optional: | Variable | Default | | |---|---|---| | `LOOM_PORT` | `8477` | | | `LOOM_HOST` | `127.0.0.1` | Binds to loopback by default | | `LOOM_DATA_DIR` | `./data` | SQLite, blobs and exports | | `LOOM_USER_AGENT` | `LoomSupplierBot/0.1 …` | Set a real contact before crawling anything external | | `LOOM_RATE_LIMIT_MS` | `1500` | Minimum gap between requests to one host | | `LOOM_MAX_PAGES` | `250` | Page budget per run | | `LOOM_MAX_DEPTH` | `3` | Link depth | | `LOOM_FETCH_TIMEOUT_MS` | `25000` | | | `LOOM_MAX_BLOB_BYTES` | `41943040` | Largest response retained | Per-supplier settings override the rate limit, page budget and depth. --- ## Before pointing this at a real supplier 1. Set `LOOM_USER_AGENT` to something identifying, with a real contact. 2. Record the permission of record on the supplier. Loom does not check it, but it is quoted in export manifests and it is the field someone will ask about. 3. Leave `robots_policy` on `enforce` unless you have a specific agreement, in which case write the justification down — the field is required. 4. Start with a small page budget and `discover` mode to see how the site classifies before reading anything. 5. Expect the first run to produce a full unmapped-label queue. That is the system working: it is asking what this supplier's wording means before putting it in your catalog. ## Known limits - Single-tenant. The operator name is a header, not an authenticated identity. - Assets are recorded by URL and classified, but binary files other than crawled PDFs are not downloaded into the blob store yet. - No JavaScript rendering: sites that build specifications client-side will need a supplier rule against their JSON payload, or a different fetcher. - The mapping suggester ranks by label token overlap. It is deliberately dumb, because it is only ever a suggestion. - Export targets a generic CSV/JSON shape. A specific PIM's import format is a thin adapter over `src/pipeline/export.js`, not a rewrite.