尝试在迭代中运行100个并行任务不起作用

如何解决尝试在迭代中运行100个并行任务不起作用

好吧，我试图每次运行100次任务（具有并行性），但是我无法使其正常工作。

我正在尝试强行使用API​​，为此，该API允许我连接尽可能多的IDS（不超过超时时间）

// consts:
// idsNum = 1000
// maxTasks = 100

// We prepare the ids that we will process (in that case 100,000)
var ids = PrepareIds(i * idsNum * maxTasks,idsNum,maxTasks);

// This was my old approach (didn't work)
//var result = await Task.WhenAll(ids.AsParallel().Select(async x => (await client.GetItems(x)).ToArray()));

// This is my new approach (also this didn't worked...)
var items = new List<item[]>();
ids.AsParallel().Select(x => client.GetItems(x).GetAwaiter().GetResult()).ForAll(item =>
{
    //Console.WriteLine("processed!");
    items.Add(item.ToArray());
});

var result = items.ToArray();

如您所见，我发表了Console.WriteLine("processed!");陈述，以检查是否有任何工作...但是我无法使其正常工作。

这些是我的其他方法：

    private static IEnumerable<ulong[]> PrepareIds(int startingId,int idsNum = 1000,int maxTasks = 100)
    {
        for (int i = 0; i < maxTasks; i++)
            yield return Range((ulong)startingId,(ulong)(startingId + idsNum)).ToArray();
    }

然后...

    public static async Task<IEnumerable<item>> GetItems(this HttpClient client,ulong[] ids,Action notFoundCallback = null)
    {
        var keys = PrepareDataInKeys(type,ids); // This prepares the data to be sent to the API server
        var content = new FormUrlEncodedContent(keys);

        content.Headers.ContentType =
            new MediaTypeHeaderValue("application/x-www-form-urlencoded") { CharSet = "UTF-8" };
        client.DefaultRequestHeaders.ExpectContinue = false;

        // We create a post request
        var response = await client.PostAsync(new Uri(FILE_URL),content);
        string contents = null;
        JObject jObject;
        try
        {
            contents = await response.Content.ReadAsStringAsync();
            jObject = JsonConvert.DeserializeObject<JObject>(contents);
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex);
            Console.WriteLine(contents);
            return null;
        }

        // Then we read the items from the parsed JObject
        JArray items;
        try
        {
            items = jObject
               .GetValue("...")
               .ToObject<JObject>()
               .GetValue("...").ToObject<JArray>();
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex);
            return null;
        }

        int notFoundItems = 0;
        int nonxxxItems = 0;
        int xxxItems = 0;

        var all = items.BuildEnumerable(notFoundCallback,item =>
        {
            if (item.Result != 1)
                ++notFoundItems;
            else if (item.id != 5)
                ++nonxxxItems;
            else
                ++xxxItems;
        });

        CrawledItems += ids.Length;

        NotFoundItems += notFoundItems;
        NonXXXItems += nonxxxItems;
        XXXItems += xxxItems;

        return all;
    }

    private static IEnumerable<item> BuildEnumerable(this JArray items,Action notFoundCallback,Action<item> callback = null)
    {
        foreach (var item in items)
        {
            item _item;

            try
            {
                _item = new item(item.ToObject<JObject>());
                callback?.Invoke(_item);
            }
            catch (Exception ex)
            {
                if (notFoundCallback == null)
                    Console.WriteLine(ex,Color.Red);
                else
                    notFoundCallback();

                continue;
            }

            yield return _item;
        }
    }

如您所见，我使用HttpClient创建了100个并行发布请求。但是我无法管理它。

所以我要实现的事情是检索尽可能多的项目，因为我需要抓取+2,000,000个项目。

但是会触发任何断点，也不会在控制台上更新任何标题（我正在使用Konsole project以便在控制台上的固定位置打印值），那么在那里可以提供任何建议吗？