Team integration and data governance¶

In this guide you'll learn about how different teams in a single organization usually work with Tinybird, and how data is best managed and governed.

Tinybird supports a wide range of industries and products - and this spectrum expands every day. Our customers organize themselves and their businesses in different ways, but there are over-arching principles you can adopt and adapt. Knowing how to integrate your team with Tinybird (and vice versa) is important to getting the most value out of the platform.

Foundational concepts¶

What you'll need for this guide¶

You don't need an active Workspace, you just to be familiar with the following Tinybird concepts:

What Tinybird is (and isn't)¶

Tinybird is not a data governance tool, but - on top of all the other cool things it does - it provides a way to manage who uses the data. This also helps ensure data quality and enables visibility about everything that's happening at the Organization level.

Roles and responsibilities¶

It's not possible to provide a single, clear-cut "Data Engineers do X, and Software Devs do Y" - it really depends on the company structure, preferences, and Tinybird use case. There is no single solution, and sometimes it can be a controversial topic. The good news, though, is that there are a lot of possibilities and your use case is likely to be fairly straightforward. We know, because we have customers who do it every day! (If it turns out to be unusual, don't worry: We have an amazing Customer Support team to help you iron out any issues and get you going fast 🚀).

Responsibilities: Who does what?¶

It's not really about job roles or professional personas: It's about what the different responsibilities are.

Let's step back for a moment. Tinybird allows you to share Data Sources across Workspaces. This means you can create Workspaces that map your organization, and not have to duplicate the Data Sources. In general, most Tinybird users have an ingestion Workspace where the owners of the data ingest the data, clean the data, and prepare it for onward consumption. They then share these ready-to-use Data Sources with other internal teams using the Tinybird feature Sharing Data Sources.

You can have as many ingestion Workspaces as you need; bigger organizations group their Workspaces by domain, some organizations group them by team. For instance, in a bank, you'd find different teams managing their own data and therefore several ingestion Workspaces where data is curated and exposed to other teams. In this case, each team maps to a domain:

However, in other companies where data is centralized and managed by "the data platform", you'd find just one ingestion Workspace. Here, all the data is ingested and shared with other onward Workspaces where specific domain users build their own use cases:

Some organizations rely on a hybrid solution, where certain data is provided by the data platform but each domain group also ingest their own data:

Whatever your approach, it's an established pattern to have an ingestion or "data-platform" Workspace or team who own ingestion and data preparation, and share the desired Data Sources with other teams. These downstream, domain-specific teams then create the Pipe logic specific to their own area (usually in a Workspace specifically for that domain). That way, the responsibilities reflect a manageable, clear separation of concerns.

Enforcing data governance¶

No matter which industry you're in, good data governance is essential. Tinybird isn't a data governance tool, but is built to support you in the following ways:

• Availability is assured by the platform's uptime SLA (see Terms and conditions). Tinybird wants all your teams to be able to access all the data they need, any time they need. You can monitor availability using our monitoring tools (see Monitoring below), which includes but is not limited to monitoring ingestion, API Endpoints, and quarantined rows. Compared to other tools, Tinybird offers a really straightforward way to reingest quarantined rows and maintain Materialized Views to automatically reingest data, which we know is a problem for users wanting to maintain maximum data availability.

• Control over data access is managed through a single Organization page in Tinybird. You can enforce the principle of least privilege by assigning different roles to Workspace members, and easily check data quality and consumption using Tinybird's Service Data Sources. Tinybird also supports schema evolution and you can keep multiple schema versions running at the same time so consumers can adjust at their own pace.

• Usability is maximized by having ingestion Workspaces. They allow you to share cleaned, curated data, with specific and adjusted schemas giving consumers precisely what they need. Workspace members have the flexibility to create as many Workspaces as they need, and use the Playground feature to sandbox new ideas.

• Consistency is a similar topic: Data owners have responsibility over what they want to share with others. We cannot block teams from ingesting in their own Workspaces but the Organization has the tools to monitor who (which Workspace) is ingesting data.

• Data integrity/quality, especially at scale and at speed, is simply essential. Just like availability, it's a perfect use case for leveraging Tinybird's monitoring capabilities (see Additional ecosystem tools below). Ingestion teams can build Pipes to monitor everything about their inbound data and create alerts. These alerts can be technical or business-related - see the Monitor your ingestion guide for an example.

• Data security: This information is available at the top-level Organizations page and also in individual Workspaces.

Additional ecosystem tools¶

Tinybird is built around the idea of data that changes or grows continuously. We provide additional tools (as standard) as part of Tinybird - more "process governance" tools than data governance, but still very useful. They help you get insights on and monitor your Workspaces, data, and resources.

Operations log¶

The Operations log shows information on each individual Data Source: Its size, the number of rows, the number of rows in the quarantine Data Source (if any) and when it was last updated. The Operations log contains details of the events for the Data Source, which are displayed as the results of the query. It allows you to see every single call made to the system (API call, ingestion, jobs). This is really helpful if you're concerned about one specific Data Source, and you want to see under the hood.

Monitoring¶

Enterprise customers can also use the Organizations UI for managing Workspaces and Members, and monitoring their entire consolidated Tinybird consumption in one place. For example, you can track costs and usage for each individual Workspace.

Testing¶

To ensure that all production load is efficient and accurate, all Tinybird API Endpoints that you create in your Workspaces can be tested before going to Production. You can do this by using version control - just like how you manage your codebase.

Alerts and health checks¶

To ensure everything is working as expected once you're in production, any team can create alerts and health checks on top of Tinybird's Service Data Sources.

Next steps¶

• Watch this video to understand how Factorial sets "domain boundaries" and organizes their teams.
• Build something fast and fun - follow the quick start tutorial.