Back to Questions

Design a Web Crawler

51.5K views•Asked at Google•
Medium

Question Explain

Design a large-scale web crawler (like Googlebot). Discuss seed URLs, DNS resolution, politeness policies (robots.txt), and dealing with duplicate content.

Start Trying This Question Simulation

Practice answering orally or with text to get instant AI feedback.

Practice with This Question

Other interview questions like this

Design a URL Shortener

76.7K views

Design a URL Shortening Service (TinyURL)

63.1K views

Design a Chat System (WhatsApp/WeChat)

126K views