puppeteer/chromium: handle crashing memory-heavy pages?

Question

I have the following code:

puppeteer.launch().then(async browser => {
    for (let id of ids) {
        try {
            const page = await browser.newPage();
            //const url = 'chrome://crash';
            await page.goto(url + id)
            await page.waitFor(5000);
            await page.screenshot({
                path: path.join(__dirname, "../public/images/screenshots/" + id + ".png"),
                clip: { x: 10, y: 70, width: 780, height: 470}
            });
        } catch (error) {
            console.log('Exception', id, error.message);
            page.close();
        }
    };
    browser.close();
});

It generally works OK but I'm having problems with a particular page (with a URL that unfortunately I can't share).

This page tries to load a GB of data and causes Chrome to crash, so I guess it's causing Chromium to crash too.

The error I see from this page is: Exception 6766 Navigation Timeout Exceeded: 30000ms exceeded. This is fine, but it doesn't seem to stop at this point - it causes my whole server to hang, I guess because it's trying to use too much memory at the OS level.

How can I stop my server from hanging and handle this gracefully? Are there flags I can provide to Chromium to limit the memory used and give up gracefully? I'm also not sure my error handling as a whole is correct, so any tips would be appreciated.

Answer 1

a few ideas:

First, this may not be related to your initial problem but it will save some memory. You're opening many tabs (pages) without closing them, that can use a lot of memory if your list of Ids is long.

Try to iterate over the IDs by loading a new url instead of creating new pages.

Also you can extend the timeout to check if more time will eventually do the trick.

There are chromium flags that you can pass on when you launch puppeteer. Here is the complete Chromium switch list

You could try with: --unlimited-storage or --force-gpu-mem-available-mb.

Also this may come handy to debug: --full-memory-crash-report

You could search for all the other memory related flags there.

Here is how you can pass on the flags to puppeteer, and the other suggestions that I mentioned:

puppeteer.launch({args: ['--unlimited-storage', '--full-memory-crash-report']}).then(async browser => {
  const page = await browser.newPage();

  for (let id of ids) {
      try {
          //const url = 'chrome://crash';
          // Timeout will be 2 min and will wait till network is idle before taking the screenshot
          await page.goto(url + id, {timeout: 120000, waitUntil: 'networkidle0'}); 
          await page.screenshot();
      } catch (error) {
          console.log('Exception', id, error.message);
          // If you catch an error you should throw it and handle it on the parent function
          // or set up an event listener so you can now what's the state of your app
          throw new Err (error);

          // Since we are only opening one page and browsing from here
          // no need to close when you encounter an error
          // just goto the next iteration
          // page.close();
      }
  };
  browser.close();
});