HaciGroup
Back to Public Spending Explorer

Technical Documentation

Full data schema, ingestion pipelines, and technical safeguards. This page is for developers, auditors, and researchers who need implementation details.

Public overview →Signal methodology →

Data schema (v1)

v1 scope centers on California state-level spending. No local, county, or federal data is ingested until provenance and governance are approved.

Agencies

v1

State-level departments, agencies, and control entities issuing payments or contracts.

  • agency_id (string, source key)
  • name (official agency name)
  • category (department / control agency / constitutional office)
  • parent_agency_id (nullable)
  • source_dataset
  • source_url
  • source_updated_at

Vendors

v1

Suppliers receiving state payments or purchase orders.

  • vendor_id (string, source key)
  • legal_name
  • doing_business_as
  • uei / duns (when present)
  • address_city, address_state, address_zip (from source, no enrichment)
  • naics / category (when provided)
  • source_dataset
  • source_url
  • source_updated_at

Payments / Expenditures

v1

Individual state payments with evidence links and traceability.

  • payment_id (source record id)
  • agency_id, vendor_id (foreign keys)
  • amount, currency, fiscal_year, payment_date
  • object_code / purpose (as provided)
  • description (source text only)
  • source_dataset
  • source_url
  • source_updated_at
  • ingested_at (etl timestamp)
  • checksum (hash of raw row for drift detection)

Contracts / Purchase Orders

v1

Awarded contracts or purchase orders when published alongside payments.

  • contract_id (source key)
  • vendor_id, agency_id
  • award_amount, ceiling_amount, obligated_amount
  • start_date, end_date
  • solicitation / contract_number
  • commodity_code / category
  • source_dataset
  • source_url
  • source_updated_at

Ingestion & provenance design

Only public, legally published datasets are used. Pipelines emphasize reproducibility, data freshness, and clear provenance so reviewers can verify every value.

Sources (public & state-published)

  • California State Controller's Office Open Data (State Expenditures, Payees, and legacy FI$Cal exports)
  • Open FI$Cal / FI$Cal transparency portal published CSVs
  • California Department of General Services Cal eProcure open data (contracts / POs)

Pipeline

  • Pull only published files or APIs; no credentials or scraping of non-public endpoints.
  • Record `source_dataset`, `source_url`, `source_updated_at` on every row; keep raw file checksum for audits.
  • Normalize IDs without altering source values; do not fabricate missing IDs.
  • Store import logs with row counts, hash, and ingest timestamp for replayability.

Provenance & evidence

  • Every UI record must show dataset name, record id (when available), last updated date, and a link back to the original publisher.
  • Evidence panel links directly to the exact file/API URL and, when possible, an anchor or query to the record id.
  • Surface data freshness (source_updated_at) and ingest run time (ingested_at).

Safeguards

  • No enrichment with non-public data; no geocoding of addresses beyond what the source publishes.
  • Suppress or redact PII if present in free-text fields when not essential to public accountability.
  • Respect publisher rate limits; cache downloads with ETags/Last-Modified to avoid unnecessary load.

Checksums & replayability

Every ingestion run produces a checksum of the raw input file and stores it alongside row counts and timestamps. This allows auditors to:

  • Verify that the source file has not been altered since ingestion
  • Replay the exact ingestion with the same input to reproduce results
  • Detect drift between our copy and the publisher's live dataset

Import logs include: source URL, file checksum (SHA-256), row count, ingest timestamp, and any errors or warnings.

Legal & ethical safeguards

Built-in guardrails keep the tool neutral, verifiable, and safe for public use.

  • Confirm every dataset is explicitly labeled public/open by the State of California.
  • Do not display SSNs, bank details, or personal addresses if they appear in free-text fields; mask or omit while keeping a note of the redaction.
  • Honor takedown or correction notices from publishers; log the request and patch history without deleting provenance.
  • Avoid implying wrongdoing; all copy should state facts and methods only.
  • Show data freshness and version so readers know if a discrepancy could be timing related.
  • Ensure accessibility: keyboard navigation, focus states, and readable captions for every table and chart.
This tool presents publicly available records for transparency and research purposes. It does not allege wrongdoing.

API & data exports

Currently, data is accessible through the web interface only. Future releases may include:

  • Bulk CSV exports with full provenance metadata
  • Public API for programmatic access (with rate limits)
  • Nightly snapshots for researchers

Interested in API access? Contact transparency@hacigroup.org