Netflix Improves Customer Experience, Increases Efficiency, and Reduces Costs with Amazon Kinesis Data Streams

AWS processes and enriches multiple terabytes per day, representing billions of events, with response times under 1 second for analytics queries.

Netflix is employing Amazon Web Services (AWS) for virtually all of your compute and storage needs, including databases, analytics, recommendation engines, video transcoding… hundreds of features that together use more than 100,000 server instances on AWS.

This results in a dynamic and extremely complex network environment where applications constantly communicate within AWS and across the Internet. The monitoring and optimizing your network It is essential for Netflix to continue improving its customer experience, increasing efficiency and reducing costs.

In particular, Netflix needed a solution to ingest, augment and analyze the multiple terabytes of data your network generates daily as Virtual Private Cloud (VPC) flow logs. This solution would allow Netflix to identify opportunities for performance improvement, for example, by determining which applications communicate between regions and placing them. The company could also increase uptime by quickly detecting and mitigating application downtime.

Each record contains information about communications between two IP addresses. However, in a dynamic environment like Netflix, where an IP address can switch between apps from day to day, or even minute to minute, IP addresses don't have much meaning. “The sources of the data that we had before starting this initiative were unilateral,” he indicates. John Bennett, executive software engineer at Netflix.

“We knew that one application connected to others, but we did not know both sides of the conversation and we did not know how to optimize those communications or the location of the applications on the network,” he highlights.

Netflix set out to establish a new data source that would provide more information about communication between apps and regions by combining VPC flow logs with app metadata.

From the beginning, AWS allowed Netflix to experiment with different approaches to analyzing its network data. “At the beginning of the design process, the flexibility to try different ways of processing the data was important,” Bennett says. “We tried several designs and used a lot of AWS products to get here.”

The solution that Netflix finally implemented, known internally as Dredge, centralizes flow records with Amazon Kinesis Data Streams. The application reads data from Amazon Kinesis Data Streams in real time and enriches IP addresses with application metadata to provide a complete picture of the network environment.

Bennett says, "The data is typically fed into a database to create an index that enables quick queries. Dredge merges flow logs with application metadata by streaming and indexing them without using a database, which eliminates a lot of complexity."

The rich data reaches an open source analytics application called Druid. Netflix uses Druid's OLAP query functionality to quickly divide data into regions, availability zones, and time ranges. This makes it easier to view and provides insight into how the network behaves and functions.

AWS was a logical choice for Dredge, in part because the data already resides in the AWS cloud. “It would have been difficult to publish, transmit and consume that amount of information from an external system like Kafka,” says Bennett. "It took just a few API calls to centralize multiple terabytes of stream logs into Amazon Kinesis Data Streams. Now we can focus on getting more information out of the data instead of just accessing it."

The scalability Amazon Kinesis Data Streams is well suited for the Dredge application because of the elastic and cyclical nature of network usage on Netflix. “When it comes to our network data, it's more cost-effective to be able to scale, which isn't as easy with other alternatives to Amazon Kinesis Data Streams,” Bennett says.

Real-time network monitoring

Netflix's Amazon Kinesis Streams-based solution proved highly scalable, processing billions of traffic flows per day. Typically, around 1,000 Amazon Kinesis shards work in parallel to process streaming data.

“Amazon Kinesis Data Streams processes multiple terabytes of log data per day, yet events show up in our analytics within seconds,” says Bennett. “We can discover and respond to issues in real time, ensuring high availability and a great customer experience.”

Now, Netflix can identify new ways to optimize its apps, for example, moving an app from one region to another, or switching to a network protocol better suited for a specific type of traffic. “Our solution built on Amazon Kinesis allows us to identify ways to increase efficiency, reduce costs, and improve resiliency to enhance the customer experience,” says Bennett.

Although a data transmission solution is not new to the IT sector, it is an innovation in the networking space. “Netflix invests heavily in AWS in part because it synthesizes the underlying network, so we don't have to deal with switches and routers,” Bennett says. "We can now monitor, analyze and optimize at a higher level of the stack. This would never have been possible using our data centers."