We're sorry but this app doesn't work properly without JavaScript enabled. Please enable it to continue.

Build a Web Scraper in Python

Build an SEO-flavored crawler in Python that maps internal links and teaches you how scraping really works.

Also available in:

gotypescript

What you'll learn in this Python course

Use Python to build a web crawler that scrapes pages. It's a tool that any SEO expert would be happy to have. You'll make HTTP requests and parse HTML to generate reports that can easily marshaled to standard output or a file. If you're interested in getting a job doing data analysis, this project will teach you how Python can be used to build a command-line application.

Chapter List

1
Setup
Get started with web scraping by setting up your Python environment and building essential functions to normalize URLs and extract links from HTML content.
2
Crawling
Build the core crawling engine by implementing the main crawler logic, HTTP request handling, and recursive page discovery to systematically traverse websites.
3
Concurrency
Learn to speed up your web crawler using asynchronous programming and concurrency, crawling multiple pages simultaneously while respecting configurable limits.
4
Reporting
Learn to analyze and export your web scraping results with custom reporting tools that generate both human-readable summaries and JSON datasets for further analysis.

Join 1,292 students in the Build a Web Scraper in Python course

Read reviews of their learning experiences

Great, got stuck for a while with concurrency, still need time to internalize it. But that's how to get a little better I guess.

(5/5)
Chen  profile image

Chen

Chongqing, China

This was a fun project. Great job, team boot.dev!

(5/5)
Shant Dashjian profile image

Shant Dashjian

United States

Really good, deep dive into this project. Occasionally a bit hard, but that's what Boots is for.

(4/5)
Dee Wolf profile image

Dee Wolf

United States

A fun mini-project of sorts to learn about web scrapers, associated libraries, and even async. A good balance of easy steps and challenging steps!

(5/5)
Kyle Rodgers profile image

Kyle Rodgers

Frederick, MD

Excellent course that helps us solidify all of the HTTP Client topics, as well as asynchronous programming

(5/5)
Francisco Zavala profile image

Francisco Zavala

Mexico City, Mexico

good but challenging

(5/5)
Peter  profile image

Peter

Netherlands

Very good introduction to concurrency and asynchronous operations in python. Not too difficult, not too easy. Requires reading a bit of documentation also.

(5/5)
Pavlos Pappas profile image

Pavlos Pappas

Cyprus

Nice simple web scraper.

(5/5)
Cody Milliron profile image

Cody Milliron

Lexington, NC - United States

Good course. You get to put into practice what you've learnt so far. Might have to check out the Go one as well.

(5/5)
Aduneer  profile image

Aduneer

Germany

Showing 1-9 of 30

or view more reviews

Mediocrity doesn't cut it anymore

The only way to become a great developer is to write a lot of code

Avoid tutorial hell

by writing a ton of code

Stay motivated with

a game-like curriculum

Build portfolio projects

to prove your skills

Delve deeper

into foundational concepts

Learn flexibly online

without interrupting your life

For 1% the price of college

to minimize your financial risk

Frequently asked Questions

Got questions? We've got answers

Yes! It's free to create an account and start learning. You'll get all the immersive and interactive features for free for a few chapters. After that, if you still haven't paid for a membership, you'll be in read-only (content only) mode.