The Framework proposed in this space (Alex Xu) is applied to propose a design : Getting started - a framework to propose...
Step 2 - High-level design
From the design above, 2 flows could be considered:
Video uploading flow
Transcoding servers : video transcoding is called video encoding. It’s the process of converting a video format to other formats (MPEG, HLS, etc.) which provide the best video streams and bandwidth capabilities.
Original and transcoded storage are BLOB storage.
Metadata cache is for better performance. Once the video has been uploaded, the user sends a request to update the video metadata (size, format, file name, …).
Video streaming flow
Streaming protocol like Microsoft Smooth Streaming, Apple HLS, etc. Video are streamed from CDN directly and there is very little latency.
Step 3 - Design deep dive
The DAG Model (Directed Acyclic Graph)
DAG Model help to achieve flexibility and parallelism.
Video transcoding architecture (leveraging cloud services)
Preprocessor has 4 responsibilities:
video splitting into smaller GOP (Group of pictures).
Splits videos by GOP (Group of pictures) alignment for old clients.
Generates DAG based on config files client programmers writes.
Cache data.
DAG Scheduler splits a DAG graph into stages of tasks.
Resource manager : It contains 3 queues and a task scheduler.
1 Task queue : priority queue containing tasks to be executed.
1 Worker queue : priority queue containing worker utilization info.
1 Running queue : contains info about currently running tasks and workers running tasks.
Task scheduler picks optimal task/work and instructs the chosen task worker to execute the job.
System Optimization
Speed Optimization
→ Parallelize video uploading from client
→ Parallelize video encoding : Introduction of Message Queue
