This lesson's interactive features are locked, please to keep using them

Get HTML

Let's start crawling real web pages! For these remaining steps, use https://learnwebscraping.dev/practice/ecommerce/. A single product page like https://learnwebscraping.dev/practice/ecommerce/products/ashenfang-longsword-fan-1001/ is a good place to start.

Assignment

Create an async getHTML function in crawl.ts. For now, it will just take a URL (in this case, the root of the site we're going to crawl).

async function getHTML(url: string);

For now, your function should:

Use fetch to fetch the webpage of the url
- Set a User-Agent header (e.g. BootCrawler/1.0) to avoid being blocked by servers.
If the HTTP status code is an error-level code (400+), print an error and return
If the response content-type header is not text/html print an error and return
Otherwise, just print the HTML body as a string and be done

Remember to use try/catch as appropriate for anything that could result in an error!

Import getHTML into your main function, and call it with the base_url passed in. Give your program a shot! It should print some HTML that it fetched from the internet!

Run and submit the CLI tests from the root of the project.