Controlling costs in the cloud

Manage cloud spending with Snowflake, Databricks, or Microsoft Fabric

Cloud platforms are real game changers for modern data projects. We'll tell you how to scale cleverly, avoid cost traps and keep an optimal grip on your cloud resources.

In the modern world of data management, hyperscaler platforms have established themselves as game changers. Snowflake, Databricks and Microsoft Fabric are pioneers when it comes to the flexible, scalable and efficient handling of data in the cloud. But this flexibility often comes with an unpleasant side effect: skyrocketing costs. The attractive pay-as-you-go model in particular can quickly become a cost trap if there is no clear strategy for cost control.

In this article, we'll look at how you can use proven strategies to keep your budget under control and make the most of the benefits of the cloud. Let's first take a look at the common payment models of the individual providers.

Understanding cost models: Capacity vs. consumption

The various cloud platforms Snowflake, Databricks and Microsoft Fabric pursue individual approaches in their payment and cost models, which is crucial for cost planning and control. Microsoft Fabric uses a capacity model in which companies pay to provide a specific amount of resources. This capacity is then consistently available for all processes on the platform. Managing peak loads is a crucial aspect of this. Organizations must ensure that these spikes can be absorbed by existing capacity while avoiding the risk of permanent underload that leaves resources unused. Short peak loads, on the other hand, can be automatically absorbed by fabric through bursting, throttling or smoothing.

Databricks is characterized by one of the most flexible billing models. On the one hand, this has advantages, can quickly overwhelm inexperienced users. Billing is based on the use of compute and storage resources and offers an almost infinite number of instance types, including standard and premium tier services with different levels of performance and price points. Reserved instances and Spot instances in Databricks also make it possible to realize savings through long-term commitments or flexible pricing options. Serverless computing also offers the opportunity to react particularly flexibly to fluctuating and unpredictable workloads. At this point, understanding the underlying cost structure is essential in order to make cost-effective decisions.

Snowflake uses a credit-based billing model in which companies consume credits to the second depending on the service, which can enable a high level of precision in cost control. Snowflake's serverless architecture allows computing capacity to be flexibly scaled, but requires in-depth knowledge of workload characteristics to control cloud spending. Strategic planning is essential here in order to use all platform advantages cost-effectively.

When it comes to storage costs, Microsoft Fabric, Databricks, and Snowflake have similar price points, typically between 18 and 23€ per terabyte. While this appears moderate at first glance, storage costs can increase significantly over time, particularly with large volumes of data or inefficient data management. A common cost driver is redundant or outdated data storage, which is rarely actively used. However, these costs often only account for a comparatively small proportion of total expenditure, as the majority often comes from compute resources. Nevertheless, it remains essential for companies to regularly review and optimize their storage strategies in order to ensure cost efficiency here as well.

Strategic Capacity Planning: Start Small — Scale Big

Finding the right scale for cloud resources is one of the hardest challenges when implementing data warehousing solutions in the cloud. Especially on platforms such as Databricks, which offer an enormous range of configuration options for compute instances, it can be difficult to find the right balance. This is all the more critical as insufficient or excessive capacity planning can lead to inefficient use of resources and thus to unnecessary costs.

In this context, the “Start small — scale big” strategy is gaining in importance. It offers a proven approach to first develop a sense of the actual requirements using smaller initial resources. Especially for companies that are new to the cloud environment, this method offers clear advantages by first testing which resources and capacities are really necessary. In this way, investments can be scaled up gradually and in accordance with actual demand without being burdened by oversized spending right from the start. This flexible approach makes it possible to react dynamically to growing requirements while increasing operational efficiency.

With Snowflake, it's a good idea to start with affordable sizes. At the start, a small xs warehouse may be enough to meet basic requirements and test application performance. This minimizes the risk of excessive initial costs and allows the configuration to be adjusted as soon as larger amounts of data are required.

Databricks, with its choice between classic and serverless compute, also offers a variety of options for adapting to different workloads. Classic clusters offer versatile customization options and are perfect for predictable workloads, while serverless clusters excel when demands are unpredictable.

Microsoft Fabric also makes it possible to select the initial capacity moderately and increase this as the project grows continuously. This is supported by well-thought-out license planning, such as taking into account the required Power BI “Viewer” licenses, efficient cost control.

However, initial investments should be realistically calculated to prevent unexpected cost overruns. The “Start small — scale big” strategy helps companies assess the services they need and lays the foundation for sustainable growth and success in the cloud.

Efficient use of resources: Only pay for actual use

The efficient use of resources in the cloud is a key lever for controlling costs. Computing power is often overused because compute clusters remain active after a workload is completed, for example. This results in unnecessary expenditure, which can be avoided by taking appropriate measures. One essential approach is to use auto-suspend features. These ensure that compute instances are shut down promptly after the workloads have been processed. This is particularly true for non-serverless components, which continue to run for a longer period of time and can therefore generate costs without being productive.

In addition, Autoscaling can provide a flexible solution to dynamically adapt available resources to actual needs. Automatically scaling performance up or down ensures that there are no over-provisioning or bottlenecks. This not only contributes to cost efficiency, but also optimizes application performance. Regular reviews of execution times and performance reports are also essential. Through continuous monitoring and analysis, potential problems can be identified and resolved at an early stage, which subsequently leads to savings and increases in efficiency.

Microsoft Fabric offers an interesting way to optimize costs by increasing or reducing capacities via API or scaling to the right size. Since Fabric is billed by the minute, there are savings opportunities here that many users are not fully aware of, as the focus is often on monthly prices. However, it should be recognized that this strategy also involves compromises: Certain functionalities, such as viewer licenses starting at F64 capacity, may be limited.

It is therefore worthwhile to know the available tools and their extensive configuration options in detail. The providers provide a wealth of information on best practices that are specifically designed for cost-effective use. If you master the specific tricks and functions of your platform, you can achieve significant savings — in line with a “only pay for what you really need” strategy.

Storage optimization: Reduce data volumes and storage costs

When developing a data warehouse, it is inevitable that more and more data will be aggregated over time. Data is continuously retrieved from source systems and stored in a landing zone. A central point is the efficient handling and storage of these volumes of data. In order to optimize the amount of data to be stored here, it is advantageous to load tables incrementally, if possible. This significantly reduces the number of lines to land and helps to reduce storage costs. Implementing lifecycle policies pays off, especially when larger amounts of data are regularly generated. These make it possible to automatically transfer files from expensive hot storage to cheaper cool or archive storage. A well-thought-out deletion and archiving strategy is essential, especially for streaming data, as large volumes of data are quickly accumulated here.

Snowflake and the Delta architecture in Databricks and Microsoft Fabric offer useful features, such as the time-travel feature. This is particularly useful for viewing past database states, but requires additional storage space, which increases costs. For this reason, time travel should be targeted where it is really necessary and configured carefully to avoid costly storage overloads.

Databricks and Fabric regularly offer the opportunity to optimize tables. With the Delta Open-Table format, unnecessary storage volumes can be cleaned up using commands such as Optimize and Vacuum in order to eliminate superfluous data and thus free up storage space. In recent Databricks versions, and also with Snowflake, some of these optimizations are already done automatically. The user benefits from continuous improvements without manual intervention.

It is crucial to understand and utilize the specific features and benefits of the chosen technology. Snowflake, for example, offers zero-copy cloning, which makes it possible to duplicate tables without additional storage requirements. This can be useful in development and test environments to avoid unnecessary storage costs. By integrating such best practices into their storage strategies, companies not only reduce costs, but also promote efficient and sustainable use of their data resources.

Proactive cost management through monitoring and alerts

Effective cost monitoring and setting up alerts are essential in cloud management. All major cloud platforms offer built-in tools to monitor ongoing costs. Azure provides cost management for this, AWS provides Cost Explorer, Snowflake has its own cost management system and the account console is available in Databricks, although the Azure Cost Management Tool can also be used equally.

A key aspect of cost management in the cloud is setting budgets and corresponding alerts. These alerts notify you when defined budgets are met or when there is an exceptionally high daily consumption. In Snowflake, resource monitors can even set hard limits to prevent costly overruns.

Another step towards transparency is the systematic use of tags, which allows spending to be assigned to specific departments, projects, or teams. This makes it easier to clearly identify cost sources and allows consistent cost management.

To gain even deeper insights into cost drivers, usage data can be integrated directly into tools such as Tableau or Power BI. Such dashboards enable a detailed analysis of usage patterns and help identify inefficiencies and obtain a real-time overview of the cost situation.

In addition, it is important to strictly regulate access rights. Only authorized individuals should be able to make changes to resource deployment or budget settings. This not only protects against unwanted costs, but also ensures that all measures are taken in accordance with company guidelines.

The combination of these tools and strategies enables proactive cost management that avoids financial surprises while optimizing resource utilization.

Performance tuning: optimize workloads and reduce costs

Regardless of the tool, optimizing workloads is critical to increasing performance and reducing operating costs in the cloud. Efficiency is the key here: By specifically improving the way data is processed, significant savings and performance improvements can be achieved. ETL processes themselves have great potential for optimization. Here, it is important to design processes in such a way that throughput times are minimized and resources are used optimally. This includes the practice of loading data incrementally whenever possible to reduce the amount of processing. The focus should also be on proven development paradigms in SQL and Python (Spark) and computationally intensive functions should be avoided when they are not required.

Each platform also offers specific benefits that can be used in a targeted manner. Snowflake is already making many performance improvements on its own. Query acceleration or result caching can also be used in a targeted manner to reduce query times and thus also effectively reduce costs. Another advantage is the use of zero-copy cloning, which allows tables to be duplicated without using up additional storage space.

Databricks users can benefit from activating the Photon Engine, which significantly improves the performance of Apache Spark workloads. Managing tables via the Unity Catalog makes it possible to benefit from automatic optimizations that further increase efficiency. In addition, Spot instances (remaining stocks of unused instances) can be used to carry out non-time-critical transformations cost-effectively.

Spot instances can also be used in Microsoft Fabric. In addition, a “scale-out” strategy makes it possible to distribute development tasks to smaller, time-controlled capacities. This enables flexible resource planning and can help to reduce costs with clever allocation. Unlike Databricks or Snowflake, partition optimization must be carried out manually and plays an important role in maximizing performance while minimizing costs.

These specific optimization options for the respective platforms not only support better performance, but also ensure sustainable cost management. By using these technologies in a targeted manner, companies create the basis for more efficient data operations and thus avoid unexpected increases in costs. This opens up ways to successfully overcome technical and financial challenges while fully exploiting the potential of the cloud.

Long-term savings through Reserved Instances and Commitments

After you have found the right cloud platform for your own needs and have successfully implemented the first steps, it is possible to secure significant cost advantages through longer commitments. This strategic approach makes it possible to benefit from long-term savings that can be achieved through forward-looking planning.

Take Microsoft Fabric as an example: By setting a capacity for 12 months, you can save up to 41%. This provides an excellent opportunity to significantly reduce costs. If required, the platform makes it possible to supplement this reserved capacity with additional pay-as-you-go resources. However, there is a disadvantage in the rigid obligation, particularly when workloads change unpredictably, which can make adjustments difficult.

Databricks also offers the option to reserve units, known as Databricks Consume Units, for a period of one or three years in advance. As a result, costs can be reduced by between 4 and 20%. Instances can also be reserved for this period, which promises savings of between 6 and 18%. However, it is important to remember that these reservations are beneficial if the instances are used regularly, as there are still costs for unused time.

Snowflake offers a comparable system with its Capacity Commitment Contracts. With a one or three-year commitment, fixed credit quotas can be purchased. This not only enables predictable costs, but also offers flexibility: Additional credits can be added as needed and unused credits can even be transferred to the next billing period.

In each of these cases, knowing how much power is required is critical. A reliable forecast of the resources required over the commitment period is essential in order to get the maximum benefit from the reservation and commitment models. With proper forecasting and planning, significant cost benefits can be achieved, which both increase operational efficiency and save budgets.

Without a clear cost control strategy, the “cloud dream” can quickly become a financial nightmare. It is important to know exactly the various optimization options and use them in a targeted manner to maximize the efficiency and profitability of data workloads.

This article was first published in our magazine data! Issue 5. Read now for free.

‍