From scattered sources to one source of truth
Most valuable datasets are assembled, not downloaded. Results from one site, form from another, official documents from a regulator that blocks automated access, prices from a CSV feed, sensor data from a vendor API, and your own records in a spreadsheet. A data platform is the system that gathers all of that on a schedule, matches the same entity across sources even when it is spelled three ways, stores it in a well-designed database and makes it available through a website and an API.
The largest platform we run holds roughly 489,000 races, 2.2 million runner results and 124,000 reconciled horse identities, assembled from a dozen sources by seventeen scheduled jobs, with a public read-only API and a web front end whose pages load in under a second. Read the horse racing database case study for the full picture.
What we build
- Web crawlers and scrapersPolite, rate-limited collectors built with httpx and Playwright, with retry, change detection and recovery from archived captures when a source goes dark.
- Scheduled ETL pipelinesCron-driven jobs on Render or GitHub Actions: daily refresh, backfill, enrichment, reconciliation and heal steps, each idempotent and logged.
- Entity reconciliationMatching the same horse, person, product or address across sources: exact-first, fuzzy-fallback, with a dedupe routine and an alarm when duplicates start growing.
- Database designPostgreSQL schemas built for the questions you will ask, materialised views for the slow ones, DuckDB for staging and analytics, weekly backups to cloud storage.
- Read-only APIsJSON endpoints for search, entities and analytics, exempt from the site login, documented, and consumed by native apps and partner systems.
- Data-quality monitoringInvariant checks that run daily (row counts, coverage, freshness, backlog growth), heartbeat alarms and a nightly reconcile-and-heal job.
Data quality as an engineering problem
Data platforms rot quietly. A source changes its page layout, a job dies without an error, a feed starts sending the same race twice. We treat this as an engineering problem with engineering answers: every job writes a heartbeat; a monitor records nineteen invariants every day and raises an alarm when growth-based thresholds are crossed; a weekly dedupe pass runs with a census of what it merged; and an append-only log keeps each morning's model outputs so nothing can be rewritten after the fact.
We also document what the data cannot do. Coverage that thins out before a certain year, a jurisdiction that is only partly captured, a feed with a known drift between seasons: these go in the documentation and on the page, so the people using the platform make decisions with their eyes open.
Built to be cheap to run
None of this needs a data-engineering team or a six-figure cloud bill. The platforms we build run on a one-gigabyte PostgreSQL plan, a handful of cron jobs and a small web service. Where heavy computation is needed, it runs as a batch on spot GPU instances or a serverless function and is billed by the minute. Typical total hosting cost is tens of dollars a month.