top of page

Two Years Rebuilding a Data Icon: Lessons from Leading Foursquare Engineering

Aug 15, 2023
5 min read

How to turn a collection of acquired products into a single coherent platform without stopping the business.

Everyone has experienced the quiet frustration of a map app lying to them. You walk into a coffee shop on the ground floor of a dense Manhattan office tower, open your phone, and the blue dot insists you are across the street at a dry cleaner. Or worse, you drive twenty minutes to a restaurant that Google or Apple says is open, only to find a "For Lease" sign taped to the inside of the glass.

Consumers see a slightly annoying blue dot. Engineers who build location systems see three completely different physics and data problems colliding at once: noisy mobile sensor fusion on the device, continuous entity resolution across millions of physical venues, and high-scale spatial analytics to make sense of the movement.

Over the past two years leading engineering, data science, infrastructure, design, and technical product at Foursquare, our central challenge has been unifying those three layers. Through a series of acquisitions and organic bets, we had world-class products in each category: the Places graph, the Pilgrim mobile SDK, and Unfolded Studio for GPU-accelerated spatial visualization.

On a slide deck, they looked like one platform. In the repository and the data warehouse, they were three separate companies exchanging giant flat files.

Two Years Rebuilding a Data Icon: Lessons from Leading Foursquare Engineering Architecture Explainer

A Portfolio of Products vs. A Unified Platform

When companies grow through M&A, the default trap is treating a data platform transformation as a branding and single-sign-on exercise. You put the same navigation bar on top of three web apps, unify the billing system, and declare victory.

A shared login does not make a platform. A shared data primitive does.

Before we could unify our product experience, we had to look honestly at why our systems were fighting each other. Engineering architecture always mirrors your org chart, and our pipelines reflected three distinct engineering cultures:

  1. The Places Graph: A massive batch and streaming entity-resolution engine ingesting hundreds of sources to deduplicate venues, verify hours, and predict when a storefront has permanently closed.

  2. Pilgrim (Movement SDK): An edge engine living inside iOS and Android apps, restricted by strict OS background location limits, fusing GPS, Wi-Fi, Bluetooth, and accelerometer signals to detect when a phone actually stops inside a venue rather than just passing by in traffic.

  3. Unfolded Studio (now Foursquare Studio): A browser-based visualization and analytics tool built on Kepler.gl and deck.gl, designed to render millions of geospatial data points at 60 frames per second on a client GPU.

The problem was not that any of these systems were poorly built. The problem was the impedance mismatch at the boundaries. Connecting a point-in-polygon stop detection event from Pilgrim to a venue polygon in Places, and then joining that against macroeconomic census data in Studio, required brutal, expensive PostGIS or Spark spatial joins.

That is not a UI problem. It is an abstraction problem.

The Common Currency: Moving to H3 Hexagons

The Common Currency: Moving to H3 Hexagons

To turn a portfolio of products into a single platform, you have to eliminate the translation tax between them. If every team stores and queries the physical world using a different geometry (raw latitude/longitude points, arbitrary administrative zip codes, and complex multi-ring building polygons), every cross-product feature requires a custom ETL pipeline.

Our turning point was standardizing on a shared spatial index: Uber's H3 discrete global grid system, which we extended into our own Hex Tiles format.

Instead of forcing every analytical query to compute point-in-polygon math over irregular shapes at runtime, H3 partitions the entire planet into a hierarchy of hexagons down to the square meter. When you snap movement telemetry, POI footprints, and third-party raster data into H3 cell IDs early in the ingestion pipeline, a punishing two-dimensional geometric join collapses into a standard one-dimensional integer join on an h3_index column.

When we unified the Unfolded Studio visualization engine with our core Places and Pilgrim pipelines, we stopped shipping raw multi-gigabyte polygon files across internal boundaries and standardized on H3 Hex Tiles. That single architectural shift turned twelve-hour Spark spatial joins into sub-second browser queries running directly on the client GPU via deck.gl.

This created immediate, compounding trade-offs across the stack:

  • Storage vs. Compute: Pre-indexing data into hierarchical H3 tiles increases storage footprint and upfront ingestion work, because a single polygon overlaps multiple hexagons at fine resolutions. However, it slashes downstream warehouse compute costs and makes interactive exploration possible. We applied the same principles of moving compute from cloud warehouses to the client edge to keep both latency and AWS bills predictable.

  • Schema Stability vs. Internal Velocity: We could not break thousands of existing enterprise customers and indie developers relying on our Places and Pilgrim APIs while we rewired the core. Just like Stripe manages multi-year API versions through compatibility layers, we had to design our developer APIs to survive schema evolution, keeping the public contract rock-solid while completely replacing the spatial engine underneath.

  • Precision vs. Privacy and Battery: On mobile devices, sampling raw GPS every five seconds destroys the user's battery and triggers iOS permission warnings. Pilgrim has to do the heavy math on the edge, waking up only when the accelerometer and Wi-Fi environment shift, resolving the stop locally, and emitting clean semantic events upstream.

Rebuilding the Engine While Flying the Plane

In GitHub's recent write-up on rebuilding their core Git infrastructure while the platform keeps running, they highlighted a truth every infrastructure leader knows in their bones: you rarely get the luxury of a clean-room rewrite. You have to swap out the foundation while serving billions of requests a day.

Over these past two years, I reviewed dozens of architectural proposals across our data science, infrastructure, and product teams. Whenever a migration stalled, it was almost never because the target architecture was flawed. It stalled because the team tried to boil the ocean in a single eighteen-month phase, rather than finding the narrowest seam where the new primitive could deliver immediate customer value.

Don't ask for two years of darkness to build a platform. Ship the shared primitive inside a real customer workflow in ninety days, then migrate the rest of the stack behind it.

We only succeeded once we dissolved the separate "platform unification" workstreams and put the engineers who owned the Places entity-resolution models, the Pilgrim mobile SDK, and the Studio WebGL renderer into the same release cadence against shared customer workflows.

When a data scientist working on venue deduplication can drop their output directly into Foursquare Studio via Hex Tiles and immediately see where a model is hallucinating closed storefronts across Tokyo, the feedback loop drops from weeks to minutes. The platform stops being an executive mandate and becomes the fastest way for an engineer to get their job done.

Two years in, the biggest lesson from leading this team is that a data platform transformation is never really about data warehouses. It is about picking the right foundational primitive, aligning your team boundaries to match it, and having the discipline to retire the glue code that used to hold your silos together.

When you look at the products your teams maintain today, are they actually sharing a common data primitive underneath, or are your engineers paying a hidden translation tax at every team boundary?

 
 
 

Comments


bottom of page