Data Lakes: What They Are and Why Companies Use Them

What is a data lake?

Data lakes are central data repositories used to store any and all raw data. A data lake has no predefined schema, so it retains all of the original attributes of the data collected, making it best suited for storing data that doesn’t have an intended use case yet.

James Dixon, founder at Pentaho, who coined the term “data lake,” explains the concept like this: “If you think of a datamart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state. The contents of the data lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples.”

A data lake allows for the easy, flexible storage of different types of data because it doesn’t have to be processed on the way in. It’s important, however, to have good data quality and data governance practices in place. Otherwise, you can end up with a data swamp, making it hard to access data and get real value out of it.

The difference between a data lake vs. a data warehouse

Storage is the key difference between a data lake and a data warehouse. A lake can store any type of data, from images to PDF files. A warehouse is made for storing metadata and structured and unstructured data.

Data lakes and warehouses also differ in accessibility. Data warehouses support self-service, meaning that any user can easily find the data they need without IT support. Lakes are more complex since they store unprocessed data in its original format, which limits access to expert users.

Why do companies use data lakes?

Companies collect more data by the year, requiring a scalable database where they can store their data until it’s ready for use. Data lakes have emerged as a cost-effective solution for big data that provides many other benefits as well.

1. Cost savings

Data lakes are able to store a large amount of data at a relatively low cost, making them an ideal solution to house all of your company’s historical data. A data lake offers companies more cost-effective storage options than other systems because of the simplicity and scalability of its function. For companies storing vast amounts – sometimes petabytes – of data, using a data lake results in significant cost savings for data storage.

2. Prevents silos

A data lake gives you a central repository for your data, making data available across the organization. When you store data in individual databases, you create data silos. Data lakes remove those silos and give access to historical data analysis so every department can understand customers more deeply with the same data.

3. Supports advanced analytics

By combining all of your data into a data lake, you can power a wide range of functions, including business intelligence, big data analytics, data archiving, machine learning, and data science. With predictive analytics, for example, you can use the data inside the lake to forecast future trends and prepare accordingly. Companies leverage predictive analytics to offer customers personalized product recommendations, forecast staffing needs, and create high-performing marketing campaigns.

4. Flexibility

Data lakes are schema-free, giving you the flexibility to store the data in any format. Because they keep all data in its native form, you can send the data through ETL (extract, transform, load) pipelines later, when you know what queries you want to run, without prematurely stripping away vital information.

5. Improved business performance

Thanks to the advanced analytics enabled by centralized data, companies that effectively implement a data lake surpass those that do not in several key business metrics. According to Aberdeen research, 24% of data lake leaders report “strong” or “highly effective” organic revenue growth compared to 15% of data lake followers or companies with an ineffective data lake. Similarly, 15% of leaders see growth in operating profit compared to 11% of followers.

6. Simplifies data collection

Data lakes can ingest data of any format without it needing to be structured as it flows in. The flexibility allows you to easily collect as much data as you want and process it later for a specific use case. This puts more data at your disposal to run advanced analytics.

Common challenges with data lakes

The benefits of data lakes, especially regarding business performance, are undeniable. But there are a handful of challenges that you might need to navigate as you build a data lake.

1. Security and compliance

Data management grows more complex with the volume of collected data. Without strong security measures, your company data (including your customer’s personally identifiable information) could end up in the wrong hands and cause lasting damage to your reputation. 

Considering many organizations invest in data lakes to collect more customer data for their marketing campaigns, compliance with data privacy laws like the GDPR or CCPA is also critical. For example, the GDPR provides data subjects (your customers) the right to be forgotten. To comply with a request for data erasure, you must be able to find the customer’s data in your lake and show proof that you’ve removed their information from all of your databases. A data lake that doesn’t support these actions will make compliance difficult. 

Your exact data lake security strategy will depend on whether you’re using a cloud-based, on-premise, or hybrid architecture. Cloud-based data lakes are particularly susceptible to threats – between 2021 and 2022, attacks on cloud-based networks increased by 48%. For effective cloud data security, you need strong encryption protocols that protect the data when it’s in transit and at rest. Likewise, controls are needed to govern who can access the data lake and when.

2. Data governance

Good data governance encompasses security and compliance, but it also addresses all of the data-related roles and responsibilities in your organization across the data lifecycle. If you invest in a data lake without a data governance framework in place, it opens up issues with data quality, security, and compliance. 

In scenarios where several departments work off of the same data lake, the lack of governance could lead to conflicting results and create issues around data trustworthiness. But a data governance framework will ensure all teams follow the same rules, standards, and definitions for data analysis that produce consistent results. 

3. Data quality

Managing data quality in your lake is challenging because it’s easy for poor quality data to slip in undetected until it snowballs into a larger issue. Without a way to validate the data in your lake, it will turn into a swamp, introduce dirty data into your pipeline, and create data quality issues that could impact important business activities.

One method of data validation is to create data zones that correspond with the degree of quality checks the data has undergone. For example, freshly ingested data arrives at the transient zone. Once it passes quality control and is stripped of personal information and other sensitive data, you can label it as trusted and move it further down the pipeline.

4. Costs

If you build your data lake in the cloud, be aware that the costs of cloud infrastructure have become a significant concern for business leaders. Seventy-three percent of finance and business professionals report that spending on cloud infrastructure was a C-suite or board-level issue, while 49% say their cloud spend is higher than it should be.

There are multiple reasons for the growing costs of the cloud, from supply chain disruptions and energy prices to the lack of competition in the cloud technology market. A strong financial operations (FinOps) framework will help you control cloud costs as you build and manage your data lake.

5. Performance

When data users run queries in a large data lake, they may run into performance issues that slow down analysis. For example, numerous small files in your data lake can create bottlenecks due to the limits on the number of information units that can be processed at one time. Deleted files also create a bottleneck if they remain stored for a period of time before being permanently removed.

6. Ingestion

Although data lakes are intended to store unprocessed data for later analysis, improper data ingestion leads to a data swamp where it is difficult to access, manage, and analyze data. Best practices for data ingestion start with a plan – while you don’t need to know the exact use case for the data, you do need to have a rough idea of the data’s purpose. It’s also important to compress the data and limit the number of small files your lake is ingesting due to the performance issues discussed above.

Types of data lake solutions

Companies host data lakes on different types of solutions – cloud, on-premise, hybrid, and multi-cloud.

• Cloud: Most organizations choose to store their data lakes in the cloud, a solution where a third party (such as Google Cloud) provides all the necessary infrastructure for a monthly fee.

• Multi-cloud: Multi-cloud data lakes are a combination of several cloud solutions, such as Amazon Web Services and Google Cloud.

• On-premise: The company sets up an on-premise data lake – hardware and software – using in-house resources. It requires a higher upfront investment compared to the cloud.

• Hybrid: The company uses cloud and on-premise infrastructure for their data lake. A hybrid setup is usually temporary while the company moves the data from on-premise to the cloud.

What to look for in a data lake solution?

When you are evaluating data lake solutions, keep the following criteria in mind.

• Integration with your existing data architecture: When your data lake has the ability to integrate with your other data systems, you will avoid silos and ensure you’re collecting all the data in one location.

• Strong cybersecurity standards. If you’re looking for a third-party data lake provider, run a detailed comparison of the security features and certifications of your top vendors. It’s important that security and privacy controls don’t just address outside threats but also control access on the inside.

• Costs. The ballooning costs of cloud infrastructure are an important consideration when selecting a vendor, especially if you expect your data volume to grow.

Why Segment Data Lakes is better than a traditional data lake

Traditional data lakes, like Hadoop, require engineers to build and maintain the data lake and its pipelines and can take anywhere from three months to a year to deploy. But the demand for relevant and personalized customer experiences, which require well-governed data, won’t wait. Companies need a data lake solution that can be implemented right now to attain deeper insights into their customers with their historical data.

Segment Data Lakes is a turnkey customer data lake solution built on top of AWS services that provides companies with a data-engineering foundation for data science and advanced analytics use cases. It automatically fills your data lake with all of your customer data without additional engineering effort on your part. It’s optimized for speed, performance, and efficiency. Unlike traditional data lakes, with Segment Data Lakes, companies can unlock scaled analytics, machine learning, and AI insights with a well-architected data lake that can be deployed in just minutes.

Additionally, Segment Data Lakes makes data discovery easy. Data scientists and analysts can use engines, like Amazon Athena, or load it directly into their Jupyter notebook with no additional setup for easy data querying. And Segment Data Lakes converts raw data from JSON into compressed Apache Parquet for quicker and cheaper queries.

When Rokfin implemented Segment Data Lakes, the company was able to decrease data storage costs by 60%. Furthermore, Rokfin unlocked richer customer insights by leveraging the complete dataset without extra engineering effort. These richer insights provided content creators at Rokfin with valuable information about the factors that led to higher acquisition and retention rates and helped them increase dashboard engagement by 20%.

Segment Data Lakes provides foundational data architecture to enable companies to create cutting-edge customer experiences using raw customer data.

Discover the untapped power of your data lake with a customer data platform

While data lakes are essential for storing archival data, you also need to be able to put that data to use. By pairing your data lake with a customer data platform (CDP), like Segment’s, you can combine your historical data with real-time data to power and optimize your marketing and product teams with actionable customer insights based on a complete customer profile.

Segment’s CDP improves data accessibility across the business. Segment’s CDP automatically cleans and standardizes your data before sending it on to third-party systems, such as your analytics, marketing customer service tools, customer engagement platforms, and more. So IT and engineering teams can use the data for broader data insights to form a long-term strategy. At the same time, nontechnical users, such as marketing and product teams, will be able to draw actionable insights and supercharge personalized engagement strategies with historical and real-time data.

With a customer data platform, you can make even more informed decisions with a comprehensive, single customer view. Through identity resolution, Segment’s CDP gathers data points from your data lake and other data sources and merges each customer's history into a single profile. With identity resolution, you can glean actionable insights, power your customer interactions, and create relevant, personalized experiences with data.

Segment Data Lakes and Segment’s CDP activate all of the historical data you have on a customer, with new data collected more recently for accurate insights and meaningful customer interactions.

Segment Data Lakes is available to all Segment business-tier customers as part of the current plan. Get started today by checking out our technical documentation and setup guide.

New to Segment? Sign up for a demo to learn how Segment can help you better understand your customers and engage with them effectively.