This lesson's interactive features are locked, please to keep using them

Get HTML

Let's start crawling real web pages! For these remaining steps, use https://learnwebscraping.dev/practice/ecommerce/. A single product page like https://learnwebscraping.dev/practice/ecommerce/products/ashenfang-longsword-fan-1001/ is a good place to start.

Assignment

Create a getHTML function that takes a URL string and returns an HTML string and error.

func getHTML(rawURL string) (string, error)

For now, your function should:

Use http.NewRequest with an http.Client to fetch the webpage of the rawURL. Set a User-Agent header (e.g. BootCrawler/1.0) to avoid being blocked by servers.
Return an error if the HTTP status code is an error-level code (400+)
Return an error if the response content-type header is not text/html
Return any other possible errors
Return the webpage's HTML if successful

You may find io.ReadAll helpful in reading the response.

I'd argue that it's not necessary to create unit tests for a function like getHTML. It's primarily just side effects (internet access), making it not a pure function like normalizeURL and getURLsFromHTML. Most of the "logic" in getHTML is just a couple of standard library function calls: and there's not much reason to test the standard library.

Call getHTML from your main function and print the result. It should print some HTML that it fetched from the internet!

Run and submit the CLI tests.

Notice that they're grabbing some HTML from the main page of Wikipedia.