Product pipeline

Every product passes through the same stages, in the same order, whenever it is imported (store sync, CSV, feed), created or edited in the admin app, or corrected in the partner app. Each stage records what it read and what it decided, and is skipped next time when nothing it reads has changed. Every stage is safe to run again.

The code is in packages/core/src/server/pipeline/. The avatar function is packages/core/src/shared/avatar.ts.

#The stages

The avatar runs twice because it depends on the details, but every product should have one at once. It is the same pure function both times.

#Stage 1: validate

Version 1. Deterministic rules; nothing is fetched.

"Blocked" is used only for these two hard failures; it hides nothing that was not hidden before. Warnings never hide a product. Validate owns the codes above in products.issues: it adds them and removes them when they stop applying, and leaves the codes other code writes there (the sync's notes, such as colour_from_name).

A wide, low chest of drawers ("Kummut Softoak", 129 × 39 × 84 cm) is often a sideboard, but 124 of the 173 real dressers in October 2026 had that shape: Estonian shops call sideboards "kummut" too. Size alone cannot tell them apart, so this is left to stage 3's coherence check.

#Stages 2 and 4: the avatar

avatarFor(piece, { corrected, details }) returns an AvatarSpec:

Values are merged with this precedence: a correction from the store or staff > the store's own data > a photo reading at or above 0.7 confidence (PHOTO_CONFIDENCE) > a default.

The 3D room (buildPiece in core/client/room3d.ts) and product cards (toScenePiece in packages/ui/src/ProductCard.tsx) draw from the avatar; no builder reads raw attributes or picks its own defaults. A scene piece may carry a stored avatar; it is used when its builderVersion is current, and otherwise worked out again from the piece's data. Today the shop works the avatar out in the browser from the product's data, which gives the same result as stage 2. Drawing the stored stage 4 avatar in the shop is planned, for when stage 3 adds photo readings. Staff see the stored avatar on the review page and the product page.

Where details come from is one function (facts() in avatar.ts): look columns, products.specs and its corrections, and stage 3's readings. A new source of details plugs in there, without touching the builders.

#Stage 3: evaluate the details (contract)

Planned; another session builds it. Until then the stage records noop with no output, and stage 4 equals stage 2.

It is given a function through PipelineDeps.evaluate({ name, category, imageUrl, product }) (PIPELINE_DEPS in run.ts), so tests can fake it and no network call happens in other stages. It must return EvaluatedDetails:

{
  fields: {                     // keys: look attributes (VOCAB keys, "colour" as #RRGGBB) or detail keys (shared/taxonomy.ts)
    [key: string]: { value: string | number | boolean; confidence: number }   // confidence 0–1
  },
  coherence: {                  // one call also answers "does the photo show what the name and category say?"
    matches: boolean | null,    // null: can't tell
    category: string | null,    // the category the photo shows, when it differs
    confidence: number,
    note: string | null         // short, for staff
  } | null
}

• Values must be valid for the product's category: a vocabulary value, or a detail value parseSpecs() accepts. Anything else is ignored.
• Readings below 0.7 confidence are kept in the record but not drawn; the stage notes low_confidence with the field names.
• A confident matches: false notes photo_mismatch with the category and note.
• The stage's input hash covers the name, category and photo address, so it re-runs only when one of them changes or its version is bumped.
• It never writes products columns; corrections stay with stores and staff.

#Stage 5: review

The admin app's Products → Review page (/products/review, permission products.manage) lists products whose review is open, oldest first. Each shows the store photo, the stored avatar drawn with productThumb, and the flags. Staff can:

• Approve ("Looks right"): the review becomes approved and the product leaves the queue until its flags change.
• Use the suggested category for kind_mismatch: saved as a correction (products.overrides.category), never in the synced column. The pipeline then runs again.
• Edit the product in the admin form, whose changes to a synced product are corrections too.

Every decision is audited (review_approve, review_set_category). Only flags that need a decision open a review: size_implausible, kind_mismatch, no_price, photo_invalid, low_confidence, photo_mismatch and stage_failed (REVIEW_CODES in stages.ts). Missing data a store can add (no_colour, no_photo, no_dimensions) shows on the product and in the partner app instead: two in five real products had no colour. Example products are left out of the queue.

#Stage 6: watch

The runner itself. A stage runs when it has no record, when its input hash changed, or when its version is newer than the record's; otherwise it is skipped. Because each stage's hash includes what the earlier stages decided, a change flows on to the stages that depend on it and no further. A forced run keeps an approved review while its flags are the same.

#The stage interface

interface Stage<O> {
  name: StageName;                   // "validate" | "avatar" | "evaluate" | "avatar_refined" | "review"
  version: number;                   // bump to re-run it for every product
  inputHash(input, ctx): string;     // hash of everything the stage reads
  run(input, ctx): { output: O; issues: StageIssue[]; status: StageStatus } | Promise<…>;
}

• input is { product, corrected, colourHex }: the product as the shop sees it (rowToProduct), the keys of its corrections (details as specs.<key>), and the colour as stored.
• ctx is { earlier, stored, deps }: the records of the stages before it in this run, the stage's own last record, and the injected side effects.
• A StageIssue is { code, severity: "warn" | "block", data? }.
• Stages are pure where possible. Anything with side effects (network, Claude) comes in through deps.

#Statuses

#The runner and the record

runPipeline(productIds, { from?, only?, force?, deps? }) in run.ts runs the stages in order for each product. from starts at a stage, only runs the named stages, and force (true or a list of stage names) runs them even when nothing changed. It never throws to the caller; a product whose run breaks is counted as failed. runPipelineSafely() is the version for saves and syncs. Products are read and their records written 200 at a time, one statement per batch: on 10,000 products a first run took 30 seconds with a write per record and takes 6 seconds now (in-memory database; a database over the network gains more, since about 60,000 queries became about 150).

Each run is recorded in product_pipeline, one row per product and stage (migration 021_product_pipeline):

#Triggers

Pipeline errors never break a sync or a save: they are logged, and the product is saved as before.

The backfill job is POST /api/cron/pipeline in the partner app (GET also works), with Authorization: Bearer $CRON_SECRET, the same secret as the scheduled syncs. ?force=validate,avatar or ?force=all re-runs those stages for every product. A call goes through the products in id order and stops after ?seconds= (default 240, under a hosted function's time limit); its answer then has next, and ?after=<next> goes on from there. next is null when every product has been through. npm run pipeline:backfill calls it on a running partner app until it is done (-- --force <stages> passes force). Call it after every deploy and once a day.

#Adding or changing a stage

• A new rule in validate, or a change to a stage's output: change the stage and bump its version. The next backfill runs it for every product.
• A change to the 3D builders that changes how existing products look: bump AVATAR_BUILDER_VERSION in shared/avatar.ts. It is the version of both avatar stages, and drawings cached under the old version are not reused.
• A new stage: add it to STAGES in stages.ts at its place in the order, with a unit test in packages/core/tests/pipeline.test.ts, and a row on this page.

#Who owns what

#Related pages

• Platform architecture
• Product import, management and search: diagrams of the whole path, from a store's data to search
• Furniture details by category
• How a piece is drawn
• Partner integrations guide