Build a Web Scraper in Python
Build an SEO-flavored crawler in Python that maps internal links and teaches you how scraping really works. Also available in:
Also available in:
Author
Boot.dev Team
Our expert in-house course authors are tenured software developers whose love for coding is only outstripped by a love for crafting addicting lessons.
6 Hours
of content
14 Addicting
lessons
Earn a certificate
of completion
Learn online
at your pace
What you'll learn in this Python course
Use Python to build a web crawler that scrapes pages. It's a tool that any SEO expert would be happy to have. You'll make HTTP requests and parse HTML to generate reports that can easily marshaled to standard output or a file. If you're interested in getting a job doing data analysis, this project will teach you how Python can be used to build a command-line application.
Chapter List
Join 1,229 students in the Build a Web Scraper in Python course
Read reviews of their learning experiences
Excellent course that helps us solidify all of the HTTP Client topics, as well as asynchronous programming
Francisco Zavala
Mexico City, Mexico
good but challenging
Peter
Netherlands
Very good introduction to concurrency and asynchronous operations in python. Not too difficult, not too easy. Requires reading a bit of documentation also.
Pavlos Pappas
Cyprus
Nice simple web scraper.
Cody Milliron
Lexington, NC - United States
Good course. You get to put into practice what you've learnt so far. Might have to check out the Go one as well.
Aduneer
Germany
Build a rudimentary web scraper! Use your favorite coding language in the whole, wide world: Python! Make it super fast using coroutines and an async event loop. This is pretty fundamental stuff that you should be able to deal with. Real world examples are much more complex, but this course gets you thinking the right way.
Jason Harrington
Marquette, MI
It is a difficult project for me personally, but that's where learning happens.
Lin Fre
United States
Not going to lie, for me this one was hard. A relatively small project PACKED with interesting challenges. It can be quite challenging if you’re still getting comfortable with these concepts, but it’s definitely doable if you take your time and work through the problems step by step. I also found it really helpful to talk through the project with Boots.
Gregory Yevgraphov
Serbia
The web crawler project is solid, but the async version is simultaneously too hand-holdy and hard to debug—the Data Analyst path doesn't cover the subtleties of async and recursive concurrency beforehand.
nokorikage
The Void
Showing 1-9 of 26
Mediocrity doesn't cut it anymore
The only way to become a great developer is to write a lot of code
Avoid tutorial hell
by writing a ton of code
Stay motivated with
a game-like curriculum
Build portfolio projects
to prove your skills
Delve deeper
into foundational concepts
Learn flexibly online
without interrupting your life
For 1% the price of college
to minimize your financial risk
Frequently asked Questions
Got questions? We've got answers
Yes! It's free to create an account and start learning. You'll get all the immersive and interactive features for free for a few chapters. After that, if you still haven't paid for a membership, you'll be in read-only (content only) mode.
Platform
Languages
Subjects
Support
Pricing
Community
