| Tip |
|---|
The Framework proposed in this space (Alex Xu) is applied to propose a design : Getting started - a framework to propose... |
STEP 2 - High-level design
Glossary explaining the boxes
See the page : References & Glossary.
Web Crawler Workflow
STEP 3 - Design deep dive
Starting point : Choose the right search algorithm
DFS (Depth-first search) or BFS (Breadth-first search) : BFS is commonly used by Web Crawlers and use a FIFO technic (First-in-First-out).
Handle too many requests (Url Frontier)
Purpose is to handle or to regulate the rate of server visits within a short period.
Queue Router (QR) ensures that each queue (Q1, Q2, .., Qn) only contains URLs from the same host.
QR uses a mapping table (see it here).
Each queue contains URLs from the same host (FIFO pattern implemented).
Queue Selector (QS) : each worker is mapped to a FIFO queue and the thread downloads web pages 1 by 1 from the same host.
Scalability to handle billions of web pages (Url Frontier)
Purpose is to prioritize and to be extremely efficient using parallelization.
Front queue manages prioritization;
Back queue manages quantity of requests.
Performance optimization (HTML Downloader & DNS Resolver)
Purpose is to provide a system robustness and to be able to handle the basic side (crash, malicious links, etc.).
