Skip to content

func ${Entity}.GetAll() does not return all Entities #807

@cybercom684

Description

@cybercom684

Summary

wc.Product.GetAll() and similar entity methods only return the first result page from the WooCommerce REST API.
For large stores, this causes incomplete data results and inconsistent sync behavior.

Expected behavior

GetAll() should return all entities (iterate all pages), or an extended helper should be provided.

Actual behavior

Only the first per_page batch is returned.
Developers have to implement their own pagination and concurrency handling manually.


Proposed Solution

A static helper to fetch all entities in parallel (with configurable concurrency and batch size):

using WooCommerceNET.WooCommerce.v3;

namespace LibraryTestTool.WooCommerce;

using System.Collections.Concurrent;
using WooCommerceNET.WooCommerce.v3;

public static class WCHelper
{
    /// <summary>
    /// Get all entities per batch as List&lt;T&gt; (thread-safe and paginated)
    /// </summary>
    /// <typeparam name="T">Entity type (e.g. Product, Order)</typeparam>
    /// <param name="fetchPage">Function to fetch one page of data given the page number</param>
    /// <param name="perPage">Items per page, e.g. 20</param>
    /// <param name="maxConcurrency">Maximum concurrent requests, e.g. 2</param>
    /// <param name="startPage">Start page index, default 1</param>
    /// <returns>All fetched entities</returns>
    public static async Task<List<T>> GetAllAsync<T>(
        Func<int, Task<List<T>>> fetchPage,
        int perPage = 20,
        int maxConcurrency = 2,
        int startPage = 1)
    {
        var allEntities = new ConcurrentBag<T>();
        var semaphore = new SemaphoreSlim(maxConcurrency);
        var runningTasks = new List<Task<List<T>>>();

        int currentPage = startPage;
        bool morePages = true;

        while (morePages || runningTasks.Count > 0)
        {
            while (morePages && runningTasks.Count < maxConcurrency)
            {
                await semaphore.WaitAsync();

                int pageToLoad = currentPage++;
                var task = fetchPage(pageToLoad)
                    .ContinueWith(t =>
                    {
                        semaphore.Release();
                        return t.Result ?? new List<T>();
                    });

                runningTasks.Add(task);
            }

            var finishedTask = await Task.WhenAny(runningTasks);
            runningTasks.Remove(finishedTask);

            var resultPage = await finishedTask;
            if (resultPage.Count == 0)
            {
                morePages = false;
            }
            else
            {
                foreach (var item in resultPage)
                    allEntities.Add(item);

                if (resultPage.Count < perPage)
                    morePages = false;
            }
        }

        return allEntities.ToList();
    }
}
// UseCase for Products
var wc = new WCObject(rest, CultureInfo.GetCultureInfo("de-DE"));

var allProducts = await WCHelper.GetAllAsync(
    async (page) =>
    {
        return await wc.Product.GetAll(new Dictionary<string, string>
        {
            { "per_page", "20" },
            { "page", page.ToString() }
        });
    },
    perPage: 20,
    maxConcurrency: 2
);

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions