Aggregate

Feeds are essentially just lists of posts. A post represents a single web page. The entire point of the gator program is to fetch the actual posts from the feed URLs and store them in our database. That way we can display them nicely in our CLI.

Assignment

Enhance the agg command to actually fetch the RSS feeds, parse them, and print the posts to the console--all in a long-running loop.

Create a new migration that adds a last_fetched_at column to the feeds table. It should be nullable.
Add a markFeedFetched function. It should simply set the last_fetched_at and updated_at columns to the current time for a given feed (probably by ID is simplest).
Add a getNextFeedToFetch function. It should return the next feed we should fetch posts from. We want to scrape all the feeds in a continuous loop. A simple approach is to keep track of when a feed was last fetched, and always fetch the oldest one first (or any that haven't ever been fetched). SQL has a NULLS FIRST clause that can help with this.
- Drizzle has a sql operator that lets you write raw SQL queries when the ORM syntax isn't enough.
Write an aggregation function, I called mine scrapeFeeds. It should:
- Get the next feed to fetch from the DB.
- Mark it as fetched.
- Fetch the feed using the URL (we already wrote this function)
- Iterate over the items in the feed and print their titles to the console.
Update the agg command to now take a single argument: time_between_reqs.

time_between_reqs is a duration string, like 1s, 1m, 1h, etc. I created a parseDuration(durationStr: string): number function to parse the input into milliseconds using (fully-licensed) RegEx:

const regex = /^(\d+)(ms|s|m|h)$/;
const match = durationStr.match(regex);

It should print a message like Collecting feeds every 1m0s when it starts.
Use setInterval to run your scrapeFeeds function once every time_between_reqs. I called scrapeFeeds once before setInterval to start the loop immediately.

scrapeFeeds().catch(handleError);

const interval = setInterval(() => {
  scrapeFeeds().catch(handleError);
}, timeBetweenRequests);

I used the returned interval ID to stop the loop when the program is killed.
I added a SIGINT listener to stop the loop and print a message when the program is killed. This has to be wrapped in a Promise to stop the program from exiting immediately.

await new Promise<void>((resolve) => {
  process.on("SIGINT", () => {
    console.log("Shutting down feed aggregator...");
    clearInterval(interval);
    resolve();
  });
});

Do NOT DOS the servers you're fetching feeds from. Anytime you write code that makes a request to a third party server you should be sure that you are not making too many requests too quickly. That's why I recommend printing to the console for each request, and being ready with a quick Ctrl+C to stop the program if you see something going wrong.

The agg command should now be a never-ending loop that fetches feeds and prints posts to the console. The intended use case is to leave the agg command running in the background while you interact with the program in another terminal.

You should be able to kill the program with Ctrl+C.

There are no CLI tests for this lesson, test your own program and make sure everything behaves as expected. Here are a few RSS feeds to get you started:

TechCrunch: https://techcrunch.com/feed/
Hacker News: https://news.ycombinator.com/rss
Boot.dev Blog: https://blog.boot.dev/index.xml