1. Home
  2. Services
  3. Data Platforms, Pipelines and APIs

Service

Data Platforms, Pipelines and APIs

When the data you need lives in six places, none of them yours, we build the platform that collects it, cleans it, reconciles identities, monitors itself and serves it to people and applications.

From scattered sources to one source of truth

Most valuable datasets are assembled, not downloaded. Results from one site, form from another, official documents from a regulator that blocks automated access, prices from a CSV feed, sensor data from a vendor API, and your own records in a spreadsheet. A data platform is the system that gathers all of that on a schedule, matches the same entity across sources even when it is spelled three ways, stores it in a well-designed database and makes it available through a website and an API.

The largest platform we run holds roughly 489,000 races, 2.2 million runner results and 124,000 reconciled horse identities, assembled from a dozen sources by seventeen scheduled jobs, with a public read-only API and a web front end whose pages load in under a second. Read the horse racing database case study for the full picture.

What we build

  • Web crawlers and scrapersPolite, rate-limited collectors built with httpx and Playwright, with retry, change detection and recovery from archived captures when a source goes dark.
  • Scheduled ETL pipelinesCron-driven jobs on Render or GitHub Actions: daily refresh, backfill, enrichment, reconciliation and heal steps, each idempotent and logged.
  • Entity reconciliationMatching the same horse, person, product or address across sources: exact-first, fuzzy-fallback, with a dedupe routine and an alarm when duplicates start growing.
  • Database designPostgreSQL schemas built for the questions you will ask, materialised views for the slow ones, DuckDB for staging and analytics, weekly backups to cloud storage.
  • Read-only APIsJSON endpoints for search, entities and analytics, exempt from the site login, documented, and consumed by native apps and partner systems.
  • Data-quality monitoringInvariant checks that run daily (row counts, coverage, freshness, backlog growth), heartbeat alarms and a nightly reconcile-and-heal job.

Data quality as an engineering problem

Data platforms rot quietly. A source changes its page layout, a job dies without an error, a feed starts sending the same race twice. We treat this as an engineering problem with engineering answers: every job writes a heartbeat; a monitor records nineteen invariants every day and raises an alarm when growth-based thresholds are crossed; a weekly dedupe pass runs with a census of what it merged; and an append-only log keeps each morning's model outputs so nothing can be rewritten after the fact.

We also document what the data cannot do. Coverage that thins out before a certain year, a jurisdiction that is only partly captured, a feed with a known drift between seasons: these go in the documentation and on the page, so the people using the platform make decisions with their eyes open.

Built to be cheap to run

None of this needs a data-engineering team or a six-figure cloud bill. The platforms we build run on a one-gigabyte PostgreSQL plan, a handful of cron jobs and a small web service. Where heavy computation is needed, it runs as a batch on spot GPU instances or a serverless function and is billed by the minute. Typical total hosting cost is tens of dollars a month.

Have something you need built?

Tell us the problem, the systems you already use and what a good outcome looks like. You will get a plain-language reply within one business day, usually with a first sketch of how we would build it.