Customer data lives everywhere: your website, your mobile apps, and internal tools.
That’s why collecting and processing all of it is a tricky problem. Segment has built libraries, automatic sources, and functions to collect data from anywhere—hundreds of thousands of times per second.
We’ve carefully designed each of these areas to ensure they’re:
- Performant (batching, async, real-time, off-page)
- Reliable (cross-platform, handle rate-limits, retries)
- Easy (setup with a few clicks, elegant, modern API)
Here’s how we do it.
You need to collect customer data from your website and mobile apps tens of thousands of times per second. It should never crash and always work reliably. Here’s how we do it…
To fully understand your users, you’ll have to mirror the databases of your favorite SaaS tools (Adwords, Stripe, Salesforce, and more) into your systems of record. Here’s how we do it…
Often, you’ll want to pull in data from arbitrary web services or internal sources. They’ll have webhooks for triggering new data entries, but no standardized way of getting that data. Here’s how we do it…
To collect all this data, you need an API that never goes down. We’ve ensured that our high-performance Go servers are always available to accept new data. With a 30ms response time, and “six nines” of availability, we’re collecting 1M rps, handling spikes of 3-5x in minutes.
Data can be messy. As anyone who has dealt with third-party APIs, JSON blobs, and semi-structured text knows that only 20-30% of your time is spent driving insights. Most of your time is spent cleaning the data you already have.
At minimum, you’ll want to make sure your data infrastructure can:
- Handle GDPR suppressions across millions of users
- Validate and enforces arbitrary inputs
- Allow you to transform and format individual events
- Deduplicate retried requests
Here’s how we do it.
Validation and Transformation
Sometimes the data you’re collecting isn’t always in the format you want. It’s important that anyone—PMs, marketers, engineers—be able to help clean the data you’re collecting to match your needs.
What’s worse than missing data? Duplicated data. It’s impossible to trust your analysis if events show up 2, 3, or more times. That’s why we’ve invested heavily in our Deduplication Infrastructure. It reduces duplicate data by 0.6% over a 30d window.
GDPR Suppression and Deletion
If you’re working with data today, you must be compliant with the GDPR and CCPA. Both grant individual users the ability to request that their data be deleted or suppressed. It’s sort of like finding 100k needles in 100 billion haystacks—but here’s how we do it.
Individual events don’t tell the full story of a user. For that, you need to combine that data all into the notion of a single profile.
This is where most systems hit their scaling limits. Not only do they need to process tens of thousands of events per second, but they need to route that data to a single partition. This requires:
- Querying a single profile in real-time in milliseconds
- Scanning across millions of user histories to find a small group
Here’s how we do it.
Emails, device IDs, primary keys in your database, account IDs—a single user might be keyed in hundreds of different ways! You can’t just overwrite those ties or endlessly traverse cycles in your identity graph. You need a bullet-proof system for tying those users together. Here’s how we do it.
Personalizing your website isn’t easy. You need millisecond response times to be able to get the page to load quickly. You need a full event history of your user. And you need to be able to make those decisions as the page is loading. Here’s how we do it.
Many customers have mission-critical data which is siloed within an internal database or data warehouse. We built SQL traits to automatically pull in this data via a customized SQL query. JOIN together rich data processed via your own ETL jobs, then send it to hundreds of downstream tools.
Searching through billions of user actions to find who loaded a particular page more than 3 times in a month isn’t easy. You have to run complex aggregations, set up your data structure so they can be quickly and efficiently queried, and run it all on core infra.
Once you finally have data in one spot, there’s one last step—using that data. That means getting it into all of the different consumers and end tools.
In a controlled environment, this is easy. But over the open internet, it’s anything but.
You’re connecting to dozens of APIs. What if one fails? What if it gets slow? What if an API mysteriously changes? How do you ensure nothing goes missing?
You have all sorts of semi-structured data. How do you map it into structured forms like a data lake or a data warehouse.
It’s not easy, but here’s how we’ve built it.
APIs fail. At any given time, we see dozens of endpoints timing out, returning 5xx errors, or connection resets. We built Centrifuge, infrastructure to reliably retry and deliver messages even in cases of extreme failure. It brings up our message delivery by 1.5% on average.
Writing integrations is fussy. There’s new APIs to learn, tricky XML and JSON parsing, authentication refreshes, and mapping little bits of data. Often these APIs have little documentation and unexpected error codes. We’ve built out our destinations to handle all of these tiny inconsistencies at scale.
Warehouses / ETL
Most companies load data into a data warehouse. It’s the source of truth for various analytics deep dives. Getting data in there can be challenging, as you need to consider schema inference, data cleaning, loading at off peak hours, and incrementally syncing new data. Here’s how we do it.