Five steps to a modern technology setting
Many a company plans its technological environment required for data analysis on a so-called greenfield site. This phrase is a symbol for a concept that can be implemented without having to consider major framework conditions. On the one hand, it sounds like freedom and endless possibilities. On the other hand, it often poses a major dilemma for companies that are newcomers to data analysis and comprehensive cloud architecture: They want to use their data quickly and profitably. But how, with little or no adequate infrastructure? After all, high-quality data analysis is one of the most important growth factors for companies. However, this requires a carefully planned and, above all, reliable architecture. So where to start?
Modern Data Stack: Everything can, some must
The simple and important solution to this dilemma is the Modern Data Stack. A multi-layered system of tools in the cloud that process data in three phases:
- the central storage and basis in a warehouse,
- the connection of the raw data,
- and analyzing the transformed data.
These data tools are time-saving and cost-effective. In addition, the required infrastructure can be set up with little prior technological knowledge.
Tool-Time: Orientation in the tech stack
First of all, one thing should be mentioned: There is now a wide range of different tools that cover many different needs. As the technologies within the stack are decisive for the success of all data management measures, they must be selected carefully, differentiated and individually. There is also no single technology that is suitable for all planned activities. Every company needs its own data stack, which can always be adapted to current requirements. It is therefore difficult at this point to name or even recommend universal technologies that meet all the requirements of every company.
The following recommendations therefore include tools that have proven successful in recent years and take care of many of these technological necessities in a highly innovative way. taod also draws on a wealth of experience and we have developed a good feeling for which tools should be considered at least once within a modern data stack. Therefore, in addition to the respective strategic step, this article recommends a basic technology that can be used sensibly in many cases - and that every company should definitely have heard of at least once. Here is our Modern Data Stack guide in five steps:
Modern Data Stack Step 1: Cloud Data Warehouse
One thing is clear: any raw data must be filed and stored centrally before it can be processed further. We recommend a cloud data warehouse that scales automatically in order to be able to react flexibly to large volumes of data. If new data sources are added, they are simply fed in, stored and processed there.
Our tool recommendation: Snowflake. The implementation of the warehouse is very simple and requires little or no engineering effort. The cloud tool is a top performer in terms of storage scaling. Per-second, consumption-based pricing guarantees that you only pay for the service you need.
Modern Data Stack Step 2: Connection through connectors
Once it has been determined on which platform all the data is to be collected, the connection must be secured. Traditional data stacks used to require programming by analytics engineers. They had to write complex codes that connected the data sources to the warehouse. There are now many tools that take over this work with automated connectors. This means that all members of a data team are able to add sources in a few simple steps without spending a lot of time or incurring high costs.
Our tool recommendation: Fivetran. The automated data integration tool, which is based on a fully managed ELT architecture, offers maintenance-free pipelines and query-ready schemas. Over 200 connectors are available to connect analytics, CRM or marketing data, for example.
Modern Data Stack Step 3: Business Intelligence
The data from the warehouse can now be used. With a business intelligence tool, companies can analyse their data and then visualize it using dashboards. It is important that all employees in the company have access to the data. Data-based decisions must be possible for everyone. The desired self-service mentality in the company is massively supported by appropriate BI tools.
Our tool recommendation: Tableau or Power BI. In our day-to-day project work, both BI tools have proven to be excellent for visualizations. Tableau shows its strengths particularly in map views. Power BI can be easily embedded in Microsoft environments.
Modern Data Stack Step 4: Data transformation
Theoretically, an initial data stack would already be functional once the first three steps have been completed. In practice, data transformation is still required. This is necessary because raw data is generally not yet suitable for reporting. Raw data often contains irrelevant data, for example duplicates, test data records or metadata that is only relevant for the original production system.
Our tool recommendation: dbt. Data Building Tool, dbt for short, is a command line tool that transforms data using simple SQL commands. The application is intuitive to use so that even non-engineers can quickly familiarize themselves with it.
Modern Data Stack Step 5: Data Science
Data science describes measures that derive further insights from the data and thus provide recommendations or business assessments for the future, for example. Methods and knowledge from the fields of mathematics, statistics, stochastics and computer science as well as industry knowledge are used for this.
This makes data science the supreme discipline for companies. However, this project can only succeed if the previous steps have been carefully considered and the technology stack is functioning properly. Companies should always keep this in mind for all their data projects.
Conclusion: always keep moving
These five steps lead from the green field mentioned at the beginning to an open and adaptable modern data stack. However, the stack only partially reduces the wide range of solution approaches and the associated latent pressure to make decisions based on the extensive possibilities in the area of data management.
In a constantly changing, technology-driven world of decision-making, a tool may be the perfect choice today. However, it may prove to be no longer sufficiently value-adding tomorrow or the day after tomorrow. The good news describes the irreplaceable advantage of the Modern Data Stack: all tools can be replaced quickly if necessary. If the need for new technologies is identified, they can be integrated quickly and flexibly.
The Modern Data Stack solves countless technological challenges in one fell swoop by constantly combining and interlinking different building blocks of data management. Which insights and results ultimately grow on the greenfield site will always depend on which elements are placed where and how companies use them for themselves.
