使用Puppeteer AWS Lambda遍历多个有效负载并拍摄多个屏幕截图 修改后的代码示例播放负载 S3中的示例输出

如何解决使用Puppeteer AWS Lambda遍历多个有效负载并拍摄多个屏幕截图 修改后的代码示例播放负载 S3中的示例输出

我当前正在使用以下Puppeteer AWS Lambda Layer抓取30个URL，并在S3中创建和保存屏幕截图。目前，我发送了30个单独的有效负载，因此运行了30个AWS Lambda函数。 https://github.com/shelfio/chrome-aws-lambda-layer

每个JSON有效负载包含一个URL和一个图像文件名，这些URL和图像文件名每2-3秒通过POST请求发送到API Gateway。列表中的前6或9个Lambda函数似乎运行良好，然后由于AWS Cloudwatch中的报告Navigation failed because browser has disconnected!而失败。

因此，我正在寻找替代解决方案，如何通过处理单个JSON负载数组来编辑下面的代码以批量截取一组30个URL的屏幕快照？ （例如，For循环等）

这是我当前用于生成单个AWS Lambda屏幕截图并将其发送到S3的代码：

// src/capture.js

// this module will be provided by the layer
const chromeLambda = require("chrome-aws-lambda");

// aws-sdk is always preinstalled in AWS Lambda in all Node.js runtimes
const S3Client = require("aws-sdk/clients/s3");

process.setMaxListeners(0) // <== Important line - Fix MaxListerners Error

// create an S3 client
const s3 = new S3Client({ region: process.env.S3_REGION });

// default browser viewport size
const defaultViewport = {
  width: 1920,height: 1080
};

// here starts our function!
exports.handler = async event => {

  // launch a headless browser
  const browser = await chromeLambda.puppeteer.launch({
    args: chromeLambda.args,executablePath: await chromeLambda.executablePath,defaultViewport
  });
  console.log("Event URL string is ",event.url)

  const url = event.url;
  const domain = (new URL(url)).hostname.replace('www.','');

  // open a new tab
  const page = await browser.newPage();

  // navigate to the page
  await page.goto(event.url);

  // take a screenshot
  const buffer = await page.screenshot()

  // upload the image using the current timestamp as filename
  const result = await s3
    .upload({
      Bucket: process.env.S3_BUCKET,Key: domain + `.png`,Body: buffer,ContentType: "image/png",ACL: "public-read"
    })
    .promise();

  // return the uploaded image url
  return { url: result.Location };
};

当前个人JSON有效负载

{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/gavurin.com.png","url":"https://gavurin.com"}

解决方法

我试图复制该问题并将代码修改为使用循环。

在处理此问题时，我发现了一些值得指出的地方：

• lambda需要大量RAM （在我的测试中至少为1GB，但更好）。使用少量的RAM会导致故障。
• lambda超时必须很大，才能处理许多要截屏的URL。
• 您从JSON有效负载中获取的img 完全没有使用。我没有修改此行为，因为我不知道这是否是设计使然。
在运行异步for循环和/或不关闭打开的页面时，发现与您类似的错误。
• 修改后的返回值以输出 s3数组的网址。
• 未定义网址

修改后的代码

这是使用nodejs12.x运行时在我的测试中工作的修改后的代码：

// src/capture.js

var URL = require('url').URL;

// this module will be provided by the layer
const chromeLambda = require("chrome-aws-lambda");

// aws-sdk is always preinstalled in AWS Lambda in all Node.js runtimes
const S3Client = require("aws-sdk/clients/s3");

process.setMaxListeners(0) // <== Important line - Fix MaxListerners Error

// create an S3 client
const s3 = new S3Client({ region: process.env.S3_REGION });

// default browser viewport size
const defaultViewport = {
  width: 1920,height: 1080
};

// here starts our function!
exports.handler = async event => {

  // launch a headless browser
  const browser = await chromeLambda.puppeteer.launch({
    args: chromeLambda.args,executablePath: await chromeLambda.executablePath,defaultViewport
  });
  
  const s3_urls = [];

  for (const e of event) {
    console.log(e);

    console.log("Event URL string is ",e.url)

    const url = e.url;
    const domain = (new URL(url)).hostname.replace('www.','');

    // open a new tab
    const page = await browser.newPage();

    // navigate to the page
    await page.goto(e.url);

    // take a screenshot
    const buffer = await page.screenshot()

    // upload the image using the current timestamp as filename
    const result = await s3
      .upload({
        Bucket: process.env.S3_BUCKET,Key: domain + `.png`,Body: buffer,ContentType: "image/png",ACL: "public-read"
      })
      .promise();
      
      await page.close();
      
      s3_urls.push({ url: result.Location });
      
  }
  
  await browser.close();

  // return the uploaded image url
  return s3_urls;
};         

示例播放负载

[
    {"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/gavurin.com.png","url":"https://gavurin.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/google.com.png","url":"https://google.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/amazon.com","url":"https://www.amazon.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/stackoverflow.com","url":"https://stackoverflow.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/duckduckgo.com","url":"https://duckduckgo.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/docs.aws.amazon.com","url":"https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-features.html"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/github.com","url":"https://github.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/github.com/shelfio/chrome-aws-lambda-layer","url":"https://github.com/shelfio/chrome-aws-lambda-layer"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/gwww.youtube.com","url":"https://www.youtube.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/w3docs.com","url":"https://www.w3docs.com"}       
]

S3中的示例输出