Adding Recursion

This is going to be the largest step so far, and will require the most "figuring it out on your own"... you got this.

Assignment

Create a crawlPage function that uses recursion to traverse a website. Recursion is useful because we want to continue to crawl each URL we find on a page until we've crawled every page on the site. Here's my function signature:

func crawlPage(rawBaseURL, rawCurrentURL string, pages map[string]int)

rawCurrentURL is the current URL we're crawling
rawBaseURL is the root URL of the website we're crawling

In the first call to crawlPage() rawCurrentURL is a copy of rawBaseURL, but as we make further HTTP requests to all the URLs we find on the rawBaseURL, the rawCurrentURL value will change while the base stays the same.

The pages map keeps track of the number of times we've seen each internal link. This function should continue to pass the same map to itself.

Here's my pseudocode:

Make sure the rawCurrentURL is on the same domain as the rawBaseURL. If it's not, just return. We don't want to crawl the entire internet, just the domain in question.
Get a normalized version of the rawCurrentURL.
If the pages map already has an entry for the normalized version of the current URL, just increment the count and be done, we've already crawled this page.
Otherwise, add an entry to the pages map for the normalized version of the current URL, and set the count to 1.
Get the HTML from the current URL, and add a print statement so you can watch your crawler in real-time.
Assuming all went well with the request, get all the URLs from the response body HTML
Recursively crawl each URL on the page

Be careful testing this! Be sure to add print statements so you can see what your crawler is doing, and kill it with ctrl+c if it's stuck in a loop. If you make too many spammy requests to a website (including my blog) you could get your IP address blocked.

Call crawlPage in the main function instead of getHTML. When it's complete, print the keys and values of the pages map to the console.
Test your program by running it against a small site (10–50 pages, like my blog).

go run . https://wagslane.dev

When you're satisfied that everything is working, you can move on.