Tinybird Customer Story

Audiense evolves its social media timeline with Tinybird

Learn how a leader in social audience intelligence leveraged Tinybird to unlock the incredible value in the data it had collected from over 1.5 billion X users.
Start building
No credit card required

Data Stack

Audiense collects data from X in a compliant, privacy-first manner, then emits changes in user profile data to a Kinesis stream. Audiense then ingests data into Tinybird from Kinesis using the Tinybird Events API.

As we make the transition to streaming and real-time data, Tinybird gives us a massive leap forward. We put in very little effort and get a huge boost toward becoming a leader in real-time social media consumer insights.

Alfredo Artiles

Alfredo ArtilesCTO and Co-founder at Audiense

830TBprocessed per month

700,000requests per month

Audiense is a one-stop shop for audience research needs powered by social data, giving marketers and consumer researchers intelligence for audience and marketing strategies by capturing high-value, high-granularity social data and connecting it with any tech stack.

Audiense maintains a suite of social data providers to power their consumer insights. One of these providers is X, as Audiense is an enterprise customer of the well-known platform's data division. This relationship with X provides Audiense with access to massive amounts of data, including billions of posts and interactions from X users over the years.

Audiense goes the extra mile to capture this data in a compliant, privacy-first manner, leveraging it to extract meaningful, anonymized consumer behavior insights for its customers.

1. A MySQL cluster that could no longer scale

Audiense stores its X data in a proprietary database containing trillions of data points. They have maintained this database since 2014, updating it daily with new information about almost every single X user in existence. This massive database sits at the heart of their analytics service, giving their users access to high-granularity consumer trends on their target audiences and communities, a feature that Audiense calls "X Evolution."

Up until recently, Audiense stored this data in a massive MySQL cluster with trillions of rows sharded across 8 tables totaling over 20 terabytes, and they were adding new rows every day.

While the company and its users benefited from this treasure trove of X data, they started to struggle with performance and cost associated with it. The MySQL cluster itself cost them $3,000 a month just to maintain, and performance began to suffer under the on-demand queries that Audiense's growing user base was making against the data.

We got to the point where we just couldn't scale the MySQL cluster anymore. It was costing us $3,000 a month and was a huge pain to maintain. And in the end, it didn't even provide a good experience for our users.

Javier Baena

Javier BaenaHead of Data Platform at Audiense

2. Deprecate a valuable feature? Or...

Audiense had a choice to make. They could either assess alternative technologies and make a migration, or they could reduce the load by deleting old data points and reducing the granularity of their dataset.

They considered moving the data to Snowflake, but it was overkill for what they needed and would be difficult to engineer for the low-latency, high-concurrency requirements of user-facing analytics.They also considered some cloud time series databases, but the costs for storage and compute were too high, making a migration unjustifiable.

Left with few technological options, all of which were undesirable, Audiense considered deprecating X Evolution. They understood the value of this carefully curated data set and the opportunities that it could create in the future, but they were unsure of how to cost-effectively maintain it.

The MySQL cluster was really expensive, yes. But we also struggled because we had no control over when and how often our customers would hit the database. We wanted our users to have a first-class analytics experience, but our cluster - and thus our team - was suffering badly under the load.

Alfredo Artiles

Alfredo ArtilesCTO and Co-founder at Audiense

Audiense needed to discover a new angle; a fresh solution that would allow them to retain their prized dataset, develop new functionality on top of it, and fit within their budget.

3. Discovering a powerful real-time data platform

After running a proof of concept with Tinybird, the data team at Audiense had the "aha" moment they were looking for.

They could move the data from MySQL to Tinybird, completely offloading their massive MySQL costs while shifting the analytical workloads to a platform purpose-built for serving the user-facing analytics they wanted to build.

During the proof of concept, Audiense's data team discovered how easy it was to write SQL queries on top of the dataset, and then instantly publish those queries as low-latency, scalable APIs.

We knew we had value in our X Evolution data. The challenge was getting that data out and into the hands of our users so that we could generate revenue from it. Once we started building with Tinybird, we realized all of these new revenue-generating use cases that we could explore.

Alfredo Artiles

Alfredo ArtilesCTO and Co-founder at Audiense

The Audiense data team chose to move the data from their MySQL cluster over to Tinybird to support their X Evolution feature. Migrating a 20 TB dataset is no easy task, however, so the team at Audiense turned to Tinybird's Customer Support team, which is made up of experienced, professional data engineers.

The team at Tinybird assisted Audiense in exporting the data from their MySQL cluster as a series of Parquet files, which they were able to then import into Tinybird.

To be honest, the migration was the most challenging part, but Tinybird helped us solve it. Their customer support is incredible. They know what they're doing, and they take a proactive approach to support whatever you need from start to finish.

Javier Baena

Javier BaenaHead of Data Platform at Audiense

4. Evolving to a streaming data architecture

Audiense's adoption of Tinybird is part of a broader initiative within the company to shift to a streaming data platform. The data team and the company at large recognize the importance of real-time, and they're committed to creating internal infrastructure and systems that will capture the data they need and make it available to end users much more quickly.

Tinybird sits at the center of the platform, ingesting X updates as they stream in and exposing SQL-based metrics as real-time APIs that power the Audiense user experience.

Sometimes we forget that we use Tinybird. It is so stable, and we're able to delegate so many things that would normally require people's time directly to the Tinybird platform.

Alfredo Artiles

Alfredo ArtilesCTO and Co-founder at Audiense

Audiense uses Tinybird's Events API, a simple but scalable HTTP streaming endpoint, to stream change data from their X sources into Tinybird.

Once the data lands in Tinybird, they use a Materialized View to build pre-aggregated metrics that get calculated incrementally, as new data is ingested. On top of that Materialized View, they've built a parameterized API Endpoint using SQL to power their user-facing features.

They can pass time- and user-based filters as query parameters within their frontend code and return specific X trends based on user activity in the platform.

After making the transition to Tinybird, they've not only reduced overall storage from 20 TB to less than a terabyte, but they're maintaining static storage amounts by discarding unnecessary stale data, instead maintaining aggregate states in a much smaller Materialized View.

And with Tinybird's API layer, they can easily handle fluctuations in customer usage while still maintaining the sub-second API latency that promotes a positive user experience.

20Xless storage

4. Creating an ROI from data

For almost a decade, Audiense has been sitting on an incredibly valuable data asset: detailed consumer insights about all X users and how they've evolved over the last 10 years.

Before Tinybird, they suffered under an intractable MySQL cluster that had grown too big to manage. They considered deleting data or hiring more team members to set up and maintain new infrastructure.

But after switching to Tinybird, they have fully tapped into the value of this asset, discovering new ways to leverage their dataset to explore, test, and deploy revenue-generating use cases.

Without Tinybird, we would have needed people to set up and maintain ClickHouse, people to manage the API layer, people to manage the ETLs. Tinybird has easily saved us from having to hire like 3 to 5 more engineers.

Javier Baena

Javier BaenaHead of Data Platform at Audiense

Do you like this content?

Build fast data products, faster.

Try Tinybird and bring your data sources together and enable engineers to build with data in minutes. No credit card required, free to get started.
Need more? Contact sales for Enterprise support.