Mongoose / Mongodb，更新每个文档查询，非常慢

如何解决Mongoose / Mongodb，更新每个文档查询，非常慢

我在猫鼬中有此更新查询。这是1600个帖子，大约需要5分钟才能运行。

瓶颈是什么？我使用了错误的方法吗？

export const getAndStoreLatestKPI = async () => {
  console.log("start kpi");
  try {
    const marketCaps = await getKPI();
    const stocks = await mongoose.model("stock").find().exec();

    for (const stock of stocks) {
      const marketCap = marketCaps.find(
        (marketCap) => marketCap.i === stock.insId
      );

      if (marketCap != null) {
        const marketCapAdjustedVal =
          stock.country === "Finland" ? marketCap.n * 10 : marketCap.n;

        const update = {
          marketCap: marketCapAdjustedVal,};
        console.log(marketCapAdjustedVal);
        await mongoose
          .model("stock")
          .findOneAndUpdate({ insId: stock.insId },{ update });
      }
    }
    console.log("done");
    return Promise.resolve();
  } catch (err) {
    return Promise.reject(err);
  }
};

export const getKPI = async (kpiId: number) => {
  try {
    const kpiFetch = await Axios.get(someurl);
    return Promise.resolve(kpiFetch.data.values);
  } catch (err) {
    return Promise.reject(err);
  }
};

解决方法

因此，主要瓶颈是您的for循环。对于每个库存项目，您执行几个“昂贵”的操作，例如从外部API获取数据+进行一次更新，然后按1的比例进行操作。

我建议您做的是一次循环几个项目。类似于想法multithreading。 在nodejs中有几种不同的解决方案，例如nodejs worker threads

不过，我个人使用并建议使用bluebird，它可以为您提供此功能以及许多其他功能。

一些示例代码：

import Bluebird = require('bluebird');
const stocks = await mongoose.model("stock").find().exec();

await Bluebird.map(stocks,async (stock) => {
     const marketCap = marketCaps.find(
        (marketCap) => marketCap.i === stock.insId
      );

      if (marketCap != null) {
        const marketCapAdjustedVal =
          stock.country === "Finland" ? marketCap.n * 10 : marketCap.n;

        const update = {
          marketCap: marketCapAdjustedVal,};
        console.log(marketCapAdjustedVal);
        await mongoose
          .model("stock")
          .findOneAndUpdate({ insId: stock.insId },{ update });
      }
},{concurrency: 25})
// concurrency details how many concurrent process run parallel. the heavier they are the less you want concurrent for obvious reasons.