Adding Recursion

1. Create a crawlPage function that uses recursion to traverse a website. Recursion is useful because we want to continue to crawl each URL we find on a page until we've crawled every page on the site. Here's my function signature:

func crawlPage(rawBaseURL, rawCurrentURL string, pages map[string]int)

• rawCurrentURL is the current URL we're crawling
• rawBaseURL is the root URL of the website we're crawling

In the first call to crawlPage() rawCurrentURL is a copy of rawBaseURL, but as we make further HTTP requests to all the URLs we find on the rawBaseURL, the rawCurrentURL value will change while the base stays the same.

The pages map keeps track of the number of times we've seen each internal link. This function should continue to pass the same map to itself.

Here's my pseudocode:

• Make sure the rawCurrentURL is on the same domain as the rawBaseURL. If it's not, just return. We don't want to crawl the entire internet, just the domain in question.
• Get a normalized version of the rawCurrentURL.
• If the pages map already has an entry for the normalized version of the current URL, just increment the count and be done, we've already crawled this page.
• Otherwise, add an entry to the pages map for the normalized version of the current URL, and set the count to 1.
• Get the HTML from the current URL, and add a print statement so you can watch your crawler in real-time.
• Assuming all went well with the request, get all the URLs from the response body HTML
• Recursively crawl each URL on the page

2. Call crawlPage in the main function instead of getHTML. When it's complete, print the keys and values of the pages map to the console.
3. Test your program by running it against https://learnwebscraping.dev/practice/ecommerce/.

go run . https://learnwebscraping.dev/practice/ecommerce/

4. When you're satisfied that everything is working, you can move on.