Data Platform

Architecture

How the system is shaped. Full implementation stays private.

1. Step 1

   Bronze

   Land toy CRM order and service case CSV extracts as raw bronze files.

2. Step 2

   Silver

   Type rows and apply named quality rules. Failed rows go to quarantine and never reach gold KPIs.

3. Step 3

   Gold

   Trusted facts in DuckDB plus CSV export. Certified regional measures feed KPI snapshot and Ask numbers mode.

4. Step 4

   Contracts

   pytest locks accepted and quarantined counts. Optional PySpark evidence on this machine matches the same totals.

5. Step 5

   Console

   FastAPI UI with Take a tour, Run pipeline, Quality report, KPI snapshot, and Ask. The public case study ships a frozen quality and gold chip demo. Full source stays on request.

Frozen quality and gold contract (toy extracts)

Frozen pytest contract on toy CRM and service extracts. Optional Spark on this machine matched the same counts. Not a vendor lakehouse. Airflow is not claimed.

Algorithm

Rule-based quality quarantine then gold KPIs

Evaluate each bronze row against named rules. Keep only clean rows in silver. Aggregate certified measures from gold.

for row in bronze:
  failures = evaluate_rules(row)
  if failures: quarantine(row, failures)
  else: silver.append(row)
gold = load_duckdb(silver)
kpis = certified_measures(gold)
return kpis, quarantine_report

Key logic

Compact illustrative snippet (Quality split sketch (illustrative)). Not the full codebase.

def split_quality(rows, rules, key):
    accepted, quarantined, seen = [], [], set()
    for row in rows:
        failures = [name for name, check in rules if not check(row)]
        if row.get(key) in seen:
            failures.append("duplicate_key")
        elif row.get(key):
            seen.add(row[key])
        if failures:
            quarantined.append({"row": row, "failures": failures})
        else:
            accepted.append(row)
    return {"accepted": accepted, "quarantined": quarantined}

Stack