如何解决批量并行HTTP请求,每个请求带有异步循环
我正在尝试使用数组中的一堆关键字向API批量运行并行请求。 Article by Denis Fatkhudinov。
我遇到的问题是,对于每个关键字,我需要使用不同的page
参数再次运行请求,其运行次数与pages
变量中的数字相同。
我不断得到Cannot read property 'then' of undefined
来返回chainNext
函数。
没有for循环的批量处理并行请求很好,我正在努力将for循环合并到进程中。
// Parallel requests in batches
async function runBatches() {
// The keywords to request with
const keywords = ['many keyword strings here...'];
// Set max concurrent requests
const concurrent = 5;
// Clone keywords array
const keywordsClone = keywords.slice()
// Array for future resolved promises for each batch
const promises = new Array(concurrent).fill(Promise.resolve());
// Async for loop
const asyncForEach = async (pages,callback) => {
for (let page = 1; page <= pages; page++) {
await callback(page);
}
};
// Number of pages to loop for
const pages = 2;
// Recursively run batches
const chainNext = (pro) => {
// Runs itself as long as there are entries left on the array
if (keywordsClone.length) {
// Store the first entry and conviently also remove it from the array
const keyword = keywordsClone.shift();
// Run 'the promise to be' request
return pro.then(async () => {
// ---> Here was my problem,I am declaring the constant before running the for loop
const promiseOperation = await asyncForEach(pages,async (page) => {
await request(keyword,page)
});
// ---> The recursive invocation should also be inside the for loop
return chainNext(promiseOperation);
});
}
return pro;
}
return await Promise.all(promises.map(chainNext));
}
// HTTP request
async function request(keyword,page) {
try {
// request API
const res = await apiservice(keyword,page);
// Send data to an outer async function to process the data
await append(res.data);
} catch (error) {
throw new Error(error)
}
}
runBatches()
解决方法
问题仅仅是因为pro尚未定义,因为您尚未对其进行初始化。
您基本上执行以下代码:
Promise.all(new Array(concurrent).fill(Promise.resolve().map(pro => {
// pro is undefined here because the Promise.resolve had no parameter
return pro.then(async () => {})
}));
我不确定您背后的想法,但这是您在更简洁的版本中遇到的问题。
,我通过将实际请求promiseOperation
移到for循环中并在其中返回递归函数来使其工作
// Recursively run batches
const chainNext = async (pro) => {
if (keywordsClone.length) {
const keyword = keywordsClone.shift()
return pro.then(async () => {
await asyncForEach(pages,(page) => {
const promiseOperation = request(keyword,page)
return chainNext(promiseOperation)
})
})
}
return pro
}
批量处理并行请求的信用额为https://itnext.io/node-js-handling-asynchronous-operations-in-parallel-69679dfae3fc
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。