What Is the Modern Data Stack and Why Should You Be Excited About It?

As more and more organizations move their data and analytics workloads fully to the cloud, the term modern data stack continues to gain momentum in the industry—and for good reason. We’re seeing lower barriers of entry for what skillsets can be enabled to be productive, effective data engineers. We’re seeing time and cost efficiencies created through automation and through the flexibility of elastic workloads. We’re seeing openness to architectures that embrace modular components to avoid the dreaded bottlenecks that come with single-vendor lock. This confluence of benefits is creating a new wave of innovation and productivity with data and analytics, and we’re really excited about it.

In this blog, we will discuss the modern data stack, why you should consider it for your organization, and how you can get started with your modern data stack implementation.

What Is the Modern Data Stack?

Modern data stack is used to describe the combination of tools that are adopted to meet the demands of the different phases of the data lifecycle in the cloud. It’s not about specific tools; what makes a stack modern is its ability to meet the different demands caused by modern data problems at each phase of the data lifecycle—what happens to data from when it is created to when it is ready to be actioned on as information.

Technology Should Address Each Stage of the Data Lifecycle

For this discussion, we generally find it helpful to talk about the modern data stack by considering the data lifecycle in phases. Part of the difficulty of treating this discussion from a technology standpoint is that each technology has competencies that address different parts of the data lifecycle, and there’s a lot of overlap when you combine the technologies together into a stack. Here are some of the competencies in the context of analytics:

1. Data Extraction: Getting data out of systems, sources, and people’s heads.
2. Data Replication: Extracting data in a way to ensure that it matches its data source.
3. Data Ingestion: Putting data into a centralized location to be made available for analytics.
4. Data Storage: Keeping data in a centralized location to be made available for analytics.
5. Data Integration: Bringing data from different places together.
6. Data Transformation: Manipulating and combining data so that it is fit for purpose.
7. Data Cleansing: Fixing issues in data that inhibit the value of information generated from it.
8. Data Augmentation: Creating data to fill gaps or supplement other data.
9. Data Validation: Ensuring that information is accurate and precise.
10. Data Curation and Presentation: Featuring data as information for an intended audience.

If you’ve worked long enough in the data and analytics industry, you are probably already saying to yourself, “Yeah, but none of this is new.”—and in some ways, you’re right. That said, what is indeed new is in how the technologies that make up the modern data stack address these challenges and activities in the data lifecycle.

Why Should I Consider Leveling Up to A Modern Data Stack?

I recently participated in a discussion that was focused on the advantages of the modern data stack over a more traditional approach. Out of all the assertions collected and discussed in the group, a few significant ones stood out to me as valid reasons to modernize your data stack approach.

1.) Spend your time and money on data engineering—turning data into valuable information—rather than on database administration and performance tuning.

Ok, ok, you can’t eliminate all administration, especially for security, but you can simplify life by migrating into a fully managed cloud database. The main takeaway here is in time savings. You can instantly scale up workloads to speed up development and testing, driving lower time-to-value in delivering valuable information to your organization. When evaluating the cost of operating in the cloud, don’t forget to consider the lessened cost of administration and the added benefit of faster time-to-value (which is harder to quantify, but you know it is there).

2.) Data replication can be more easily automated, sometimes even including automated handling of schema drift.

If you deal with source data changes constantly disrupting your data analytics workflows, then it is time to evaluate advancements in data extraction, replication, and ingestion technology. Change data capture (CDC) and log-based replication are not new capabilities, but this space is maturing in a way where you can fast-track initial setup and automate a good portion of the maintenance required to keep these source connections up to date.

3.) Data transformation can be virtualized and calculated at run-time, enabling near real-time analytics and faster development lifecycles.

I really do not want to oversell this point, because there are a lot of valid use cases for maintaining instantiated, historical tables in dimensional data warehouses. Realistically, however, there’s probably a set of your data transformation workloads for analytics that can be updated to be based on views on top of your replication layer by using one of the modern data transformation tools, which can create faster development lifecycles, be easier to maintain in the long run, and move organizations closer to near real-time analytics capabilities. Recognize again that you are paying almost entirely for database compute in this new cloud paradigm, so a fully virtualized data warehouse may incur more infrastructure costs than a traditional stack. The idea here is that you could still end up ahead by lessoning the cost of development and maintenance of data engineering tasks.

How Do I Get Started on a Modern Data Stack Implementation?

The modern data stack has many advantages over traditional monolithic approaches, but the complexity of this current, diverse technology landscape makes it hard to understand what the right combination of data tools is for your organization. Here are some tips on how to get started on your modern data stack implementation:

1. Evaluate how much time you’re spending on maintaining data extraction and ingestion workloads. There are tools available like Fivetran, Stitch, Azure Data Factory, Matillion, Talend, and AWS Kinesis that can help simplify and even automate maintenance of these workloads to allow you to focus your time on things you can’t automate as easily.
2. Calculate how much time your data team is spending on administering, updating, upgrading, maintaining, and scaling databases. Moving to a managed cloud database platform is the best way to keep your staff and partners focused on delivering high value solutions to your organization rather than wasting time on server upgrades.
3. Consider how difficult it is to develop, test, and deploy changes in your data stack. If you are not able to make changes easily or quickly to your data integration and transformation layer, then it is time to make a change to a tool such as dbt (data build tool) that allows you to rapidly develop, then automate elements of testing and deployment of your solution.
4. Think about how easy it is for you to take advantage of new technology and tool advancements in the industry. Adopting a modular approach to technology for each phase of the data lifecycle allows you to take advantage of disruptors quickly as opposed to waiting on a monolithic approach to catch-up.
5. Look into your ability to scale up and scale down cloud resources easily. Sometimes you need horsepower to finish a job quickly to respond to an urgent need, other times you need your cloud services to shut off to save valuable resources. If this is not an easy and simple process, then it is time to re-evaluate your cloud strategy and move into highly elastic modern data platforms and services.