How does the analysis of live data work?
How can extremely high data volumes be processed?
How is a cloud-based technology stack put together?
How does the analysis of live data work?
How can extremely high data volumes be processed?
How is a cloud-based technology stack put together?
As a result of the digital transformation, data is being generated in all areas and is available for analyzing performance, trends and developments. The use of player performance data by the coaching staff has also become established in professional soccer in recent years. The particular challenge here lies in the live analysis of match data.
For our customer, a top club in the German Bundesliga, we have created a high-performance tech stack based on Azure and Snowflake. It forms the core of the taod Football Data Platform. Thanks to the special data preparation, the analysis can be started directly in Tableau and thus quickly generate added value for the coaching staff and team alike.
The Bundesliga provides over 3.6 million data points per match. These need to be retrieved live and stored in a database for further analysis. The calculated KPIs are then visualized. By setting up its own data platform, a club can lay the foundation for the successful work of its own analytics department and thus help the team to perform better.
A highly optimized ETL process developed in Python is used to retrieve the data from the interface as well as quality checks and initial transformations. The hosted code versioned on Azure DevOps is orchestrated by Logic apps and executed in Docker containers using Azure Container Registry/Azure Container Instances. This enables consistent deployment and on-demand scalability to meet the low data availability latencies required for live games in particular.
The central basis of the data platform is Snowflake. In combination with Azure, the Snowflake Data Cloud provides a stable and low-maintenance foundation. Data is stored in Snowflake across several levels in order to be able to map different versions. Data processing itself is divided into the processing of live data and the import of quality-assured raw data. Thanks to automatic scaling, peak situations on match day, when millions of data records are transferred within a few minutes, can be handled without any problems.
dbt Cloud is used to manage the services and develop the data models. All transformations are written uniformly in SQL so that individual steps can be saved using code versioning and the execution of the scripts can be orchestrated. Thanks to the integration in Snowflake, it is possible to work directly on the database and quickly create a data model in an agile implementation. In addition, dbt enables the establishment of tests and QA checks as well as the creation of documentation and data lineage.
The interactive analyses and dashboards can be carried out by providing data on a Tableau Server. As a self-service tool, Tableau offers the option of data discovery. Thanks to the direct connection to Snowflake, data can be visualized and analyzed live in Tableau. The intuitive user interface enables all employees to create their own dashboards for individual data analysis.
Data, and the associated football analytics, have long been a decisive and value-adding component, both for the club's playing success and for its business success. That is why every game is translated into data and analyzed. With the new technology stack, the Bundesliga club is now able to translate every game into data and analyze it. It can also enrich the data and convert it into animations in order to keep the communication of the evaluations to the team particularly intuitive.
Arjan van Staveren
Country Lead Germany/ Snowflake
From analysis to cloud implementation: we boost your data management strategically and technologically.
We'll show you how easy it is to manage your data with Snowflake.
What components does a professional tech stack for analytics consist of?
