| Tip |
|---|
The Framework proposed in this space (Alex Xu) is applied to propose a design : Getting started - a framework to propose... |
Step 2 - High-level design
From the design above, 2 flows could be considered:
Video uploading flow
Transcoding servers : video transcoding is called video encoding. It’s the process of converting a video format to other formats (MPEG, HLS, etc.) which provide the best video streams and bandwidth capabilities.
Original and transcoded storage are BLOB storage.
Metadata cache is for better performance. Once the video has been uploaded, the user sends a request to update the video metadata (size, format, file name, …).
Video streaming flow
Streaming protocol like Microsoft Smooth Streaming, Apple HLS, etc. Video are streamed from CDN directly and there is very little latency.
Step 3 - Design deep dive
The DAG Model (Directed Acyclic Graph)
DAG Model help to achieve flexibility and parallelism.
Video transcoding architecture (leveraging cloud services)
Preprocessor has 4 responsibilities:
video splitting into smaller GOP (Group of pictures).
Splits videos by GOP (Group of pictures) alignment for old clients.
Generates DAG based on config files client programmers writes.
Cache data.
DAG Scheduler splits a DAG graph into stages of tasks.
Resource manager : It contains 3 queues and a task scheduler.
1 Task queue : priority queue containing tasks to be executed.
1 Worker queue : priority queue containing worker utilization info.
1 Running queue : contains info about currently running tasks and workers running tasks.
Task scheduler picks optimal task/work and instructs the chosen task worker to execute the job.
System Optimization
Speed Optimization
→ Parallelize video uploading from client
→ Parallelize video encoding : Introduction of Message Queue
Safety Optimization (Pre-signed upload URL & protect videos)
Pre-signed URL vs signed URL
A pre-signed URL gives you access to the object identified in the URL, provided that the creator of the pre-signed URL has permissions to access that object. That is, if you receive a pre-signed URL to upload an object, you can upload the object only if the creator of the pre-signed URL has the necessary permissions to upload that object (See page References & Glossary for design YouTube.
A signed URL includes additional information, for example, an expiration date and time, that gives you more control over access to your content. This additional information appears in a policy statement, which is based on either a canned policy or a custom policy.
Protect videos
Digital Rights Management Systems (DRM) Level Security: The metadata from the video file is received by the Content Decryption Module. The CDM creates a license request using the header metadata and is sent to the remote license server. Once the request is received, it returns a detailed license with the content keys. Then the CDM decrypts the content using the content keys. Once the decryption is done the video content is available to the user for playback. Encryption Media Extensions API securely handles these license request and the license information which are not accessible to the user. (PlayReady - Microsoft; Widevine - Google; FairPlay - Apple).
AES Encryption : AES-128 is the only publicly available encryption algorithm that is recommended by the NSA (National Security Agency) uses a known, external piece of information, called a key, to uniquely change the source data. This algorithm supports on-demand, live or DVR streaming. The encryption key which is needed to encrypt the videos is created using OpenSSL.
Visual watermarking : Image overlay on top of video that contains identifying info for the video (can be logo,…).
Cost-Saving Optimization
CDN ensures fast video delivery on a global scale.
Most popular videos are sent to CDN & other videos to video servers.
Short videos can be encoded on-demand only.
Build your CDN with an ISP (Internet Service Provider, ex. AT&T,…)
Error handling (build a highly fault tolerance system)
Retry Pattern
Replicas
Master vs Slave Servers
Metadata Cache
API severs are stateless so requests will be directed to a different API server.