Max Pages
As it stands, our crawler is only useful on fairly small sites. Sites with thousands or hundreds of thousands of pages take too long to crawl.
Google's crawlers run concurrently on fleets of servers to crawl the majority of the internet. You might not have that kind of budget for this project.
Let's add a maxPages setting so that we can crawl even gigantic websites and have our tool automatically stop when it's done a reasonable amount of work.
Assignment
go build -o crawler
# usage: ./crawler URL maxConcurrency maxPages
./crawler "https://example.com" 3 10
go build and ./crawler can also be replaced with go run . from your main package directory.
Make sure that your program prints a line to the console each time you crawl a page, as well as once per page in the report. This will help you see what your crawler is doing, and ensure you can kill it with ctrl+c if it's stuck in a loop or spamming requests.
Make sure that you're printing a line to the console each time you crawl a page, as well as once per page in the report.
Run and submit the CLI tests.
The Boot.dev CLI requires you to be signed in to submit your solution!
Copy/paste one of the following commands into your terminal:
Run
bootdev run 2084cf1d-4fd9-4abc-a1c5-932c60452373
Submit
bootdev run 2084cf1d-4fd9-4abc-a1c5-932c60452373 -s
Run the CLI commands to test your solution.
- go run . "https://crawler-test.com/" 3 25
- Expecting exit code: 0
- Expecting to have more than 49 lines
Using the Bootdev CLI
The Bootdev CLI is the only way to submit your solution for this type of lesson. We need to be able to run commands in your environment to verify your solution.
You can install it here. It's a Go program hosted on GitHub, so you'll need Go installed as well. Instructions are on the GitHub page.