Core Architecture Cost Mapping Systems

Unit Conversion & Canonicalization

Every reliable food-cost calculation rests on an assumption that is almost never true in the raw data: that a quantity means the same thing everywhere. This guide, part of the core architecture and cost mapping systems framework, isolates the sub-problem of making that assumption real — converting every weight, volume, and count to a single canonical base unit at ingestion so that no downstream stage ever performs arithmetic on ounces, cups, and “each” in the same expression. The failure this eliminates is the silent unit mismatch: an invoice in pounds, a recipe in grams, and a POS line in “each” that reconcile to three different quantities and quietly corrupt every margin figure built on them.

Concept Definition and Data Contract

Canonicalization is a pure transform from a (quantity, unit) pair plus an optional ingredient context to a canonical quantity in a fixed base unit. The base units are chosen once for the whole platform — grams for mass, milliliters for volume, and a discrete count that is never coerced into either. The input contract carries raw_quantity (a Decimal), raw_unit (a canonical unit string, itself normalized from aliases), and ingredient_id (needed only when a volume must become a mass). The output contract is canonical_quantity, canonical_unit, and a status of CONVERTED or QUARANTINE.

Two units are never silently bridged: a volume and a mass require a density, and a count and a weight require a piece-weight. When the required constant is absent, the row is quarantined, not estimated. The normalization of messy unit strings — fl oz, floz, fluid ounce — into canonical unit tokens is a prerequisite handled in resolving regional unit aliases; this guide assumes the token is already clean and focuses on the physics of the conversion.

Architecture Decision Rationale

Convert at the edge, never at calculation time. The single most important decision is where conversion happens. Doing it at ingestion means every downstream consumer — the BOM cost roll-up, the yield factor frameworks, the variance engine — reads a single unit system and never re-implements a conversion matrix that could drift. Converting at calculation time scatters the same table across every consumer, and they inevitably diverge.

Explicit dimensions over string heuristics. A unit is mapped to a physical dimension (mass, volume, count) and a factor to the base unit, in a typed table. Guessing dimension from a string (“oz could be weight or fluid”) is where silent corruption enters. The converter refuses to proceed on an ambiguous unit unless the ingredient context disambiguates it.

Quarantine over imputation. A missing density is a data gap, not a rounding problem. Imputing “about a gram per milliliter” is how a batch of olive oil gets costed as if it were water. The pipeline quarantines and asks a human, because a fabricated conversion is worse than a delayed one. The runnable matrix is built step by step in building a unit-conversion matrix in Python.

Phase 1 — The Typed Conversion Contract

The first phase encodes units, dimensions, and base-unit factors as a typed, immutable table. Money is not involved here, but exactness still matters: quantities are Decimal so a conversion factor never injects float drift into a value that will later be multiplied by a cost.

from __future__ import annotations

from dataclasses import dataclass
from decimal import Decimal
from enum import Enum


class Dimension(str, Enum):
    MASS = "mass"
    VOLUME = "volume"
    COUNT = "count"


@dataclass(frozen=True)
class UnitDef:
    unit: str
    dimension: Dimension
    to_base: Decimal          # multiply raw_quantity by this to reach the base unit


UNITS: dict[str, UnitDef] = {
    "g":   UnitDef("g",  Dimension.MASS,   Decimal("1")),
    "kg":  UnitDef("kg", Dimension.MASS,   Decimal("1000")),
    "oz":  UnitDef("oz", Dimension.MASS,   Decimal("28.349523125")),
    "lb":  UnitDef("lb", Dimension.MASS,   Decimal("453.59237")),
    "ml":  UnitDef("ml", Dimension.VOLUME, Decimal("1")),
    "l":   UnitDef("l",  Dimension.VOLUME, Decimal("1000")),
    "each": UnitDef("each", Dimension.COUNT, Decimal("1")),
}

Keeping to_base as an exact Decimal literal (28.349523125, not a float 28.35) means the conversion is auditable and reproducible — the factor is a definition, not an approximation.

Phase 2 — Dimension Resolution and Density Bridging

The second phase converts a row, bridging dimensions only when it can. A volume-to-mass conversion multiplies by a density (grams per milliliter) looked up per ingredient; a missing density quarantines the row.

from decimal import Decimal

import pandas as pd


def canonicalize(df: pd.DataFrame, densities: dict[str, Decimal]) -> pd.DataFrame:
    """Convert (raw_quantity, raw_unit) to a base-unit canonical_quantity."""
    df = df.copy()
    out_qty, out_unit, status = [], [], []
    for row in df.itertuples():
        unit = UNITS.get(row.raw_unit)
        if unit is None:
            out_qty.append(None); out_unit.append(None); status.append("QUARANTINE")
            continue
        base = Decimal(str(row.raw_quantity)) * unit.to_base
        if unit.dimension is Dimension.VOLUME:            # bridge to mass if we can
            rho = densities.get(row.ingredient_id)
            if rho is None:
                out_qty.append(None); out_unit.append(None); status.append("QUARANTINE")
                continue
            base = base * rho                             # ml * (g/ml) -> g
        out_qty.append(base); out_unit.append("g" if unit.dimension is not Dimension.COUNT else "each")
        status.append("CONVERTED")
    return df.assign(canonical_quantity=out_qty, canonical_unit=out_unit, status=status)

The row loop here is over unit definitions, not a hot data path — in production the same logic is expressed as a vectorized map/merge once the density join is materialized, exactly as the core calculation engine prefers. The important property is that an unknown unit or a missing density becomes a QUARANTINE status rather than a None that silently zeroes a cost.

Phase 3 — Idempotent Canonical Column and Handoff

The canonical quantity is written back as a stable column keyed on the source row, so re-running canonicalization is a no-op and downstream joins always find the same value.

UPDATE ingest_lines AS l
SET canonical_quantity = c.canonical_quantity,
    canonical_unit     = c.canonical_unit,
    unit_status        = c.status
FROM canonicalized AS c
WHERE l.line_id = c.line_id
  AND l.unit_status IS DISTINCT FROM c.status;   -- idempotent: only touch changed rows

Because the update is keyed on line_id and guarded by IS DISTINCT FROM, a retried canonicalization pass writes nothing new and the column is stable. Downstream, the BOM cost roll-up and POS taxonomy mapping join only on canonical_quantity, never on raw_quantity, which is what guarantees the whole platform does arithmetic in one unit system.

Production Hardening

Exact factors. Store conversion factors as Decimal definitions, not float approximations, so a converted quantity round-trips and reconciles.
Density as versioned data. Keep ingredient densities in a reviewed table with an effective date; a density that changes (a reformulated product) mints a new version rather than silently altering historical conversions.
Quarantine, never impute. An unknown unit or missing density routes to a review queue with a reason code; the pipeline never guesses a bridge constant.
Idempotent writes. Key the canonical column on the source row and guard updates so retries are no-ops and the value is stable across runs.
Count is not mass. Never coerce each into a weight without an explicit, versioned piece-weight; treating a count as a gram is a category error that corrupts both inventory and cost.

Failure Modes and Troubleshooting

Symptom	Likely cause	Detection / fix
Oil costed like water	Volume-to-mass bridged with a default density	Require a per-ingredient density; quarantine when absent, never default to 1.0.
Converted quantity drifts on re-run	Float conversion factor	Store `to_base` as an exact `Decimal` literal.
A dish’s cost silently halved	`each` coerced to grams	Keep count as a distinct dimension; require an explicit piece-weight to bridge.
Unknown unit produced a zero	Missing unit mapped to `None` then summed as 0	Route unknown units to quarantine with a status, exclude from aggregation.
Two consumers disagree on a quantity	Conversion done at calculation time in two places	Convert once at ingestion; downstream reads `canonical_quantity` only.

FAQ

Why convert units at ingestion rather than when the cost is calculated?

Because converting at calculation time forces every consumer — the roll-up, the yield stage, the variance engine — to re-implement the same conversion matrix, and they inevitably diverge. Converting once at the edge means the entire platform operates on one unit system, removing a whole class of double-conversion and ambiguity bugs from every hot path, and making every stored quantity directly comparable.

How are volumes converted to weights?

Through a per-ingredient density in grams per milliliter, stored as reviewed, versioned data. A volume quantity is converted to the base volume unit, then multiplied by the ingredient’s density to reach mass. If no density exists for that ingredient, the row is quarantined rather than bridged with a default — assuming water’s density for oil or syrup would silently misstate cost.

Why keep conversion factors as Decimal instead of float?

A conversion factor is a definition (an ounce is exactly 28.349523125 grams), and storing it as a float approximation injects binary-rounding error into every converted quantity. Because those quantities are later multiplied by costs, that error compounds. A Decimal literal keeps the conversion exact and auditable, and the converted quantity reconciles against the source.

What happens to a unit the system does not recognize?

It is quarantined with a reason code, not guessed. An unrecognized unit token usually means an upstream alias was missed or a new supplier introduced a unit; routing it to review keeps a single bad unit from producing a None that a later aggregation treats as zero. The alias-normalization step upstream reduces how often this happens.

For a reference on quantity dimensions, see the official Pint documentation.