PartialLoader

на русском

The library is designed to load a large number of objects from the source in batches. The chunk size is limited by a timeout and/or a fixed size.

How to use it

Let's create an instance: 

            PartialLoader<Cat> partialLoader = new();

Set the data source (mandatory): 

            public async IAsyncEnumerable<Cat> GenerateManyCats()
            {
                ...
            }

            partialLoader.SetDataProvider(GenerateManyCats());

Set the timeout value (default - no timeout): 

            partialLoader.SetTimeout(TimeSpan.FromMilliseconds(200));

Set the portion size (default - unlimited): 

            partialLoader.SetPaging(5000);

Before each request for the next piece of data, install one or more "utilizers" - Action<Cat>, which will somehow process each received object: 

            partialLoader
                .AddUtilizer(item => ...)
                .AddUtilizer(item => ...)
                ;

The need to re-install "utilizers" each time is due to the fact that this class should normally be used on an ASP.NET server. The instance is needed save between requests, but the context will not be saved, and "utilizers" will most likely depend on the context.

Let's ask for another batch: 

            await partialLoader.LoadAsync();

Let's check if all the data has been received: 

            if(partialLoader.State is PartialLoaderState.Partial)
            {
                ... // Not all
            }
            else
            {
                ... // All
            }

If not all, it will install the "utilizers" again and request the next portion.

Descendants of PartialLoader<T> with special behavior and helper classes

ChunkPartialLoader<T> - saves the next chunk in a list accessible via the Chunk property. 

            do {
                await partialLoader.LoadAsync();
                List<Cat> moreCats = partialLoader.Chunk;
            }
            while(partialLoader.State is PartialLoaderState.Partial);

ResultPartialLoader<T> - adds another portion to the list available through the Result property. 

            do {
                await partialLoader.LoadAsync();
            }
            while(partialLoader.State is PartialLoaderState.Partial);
            List<Cat> allCats = partialLoader.Result;

ChunkResultPartialLoader<T> is a combination of the previous ones. 

            do {
                await partialLoader.LoadAsync();
                List<Cat> moreCats = partialLoader.Chunk;
            }
            while(partialLoader.State is PartialLoaderState.Partial);
            List<Cat> allCats = partialLoader.Result;

Example:

Processing a request for a huge number of cats in several portions. 

            app.MapGet("/manyManyCatsByPortions",  async (HttpContext context) =>
            {
                PartialLoader<Cat> partialLoader;
                string key = null!;

                // Get storage via dependency injection.
                CatsLoaderStorage loaderStorage = context.RequestServices.GetRequiredService<CatsLoaderStorage>();

                if (!context.Request.Headers.ContainsKey(Constants.PartialLoaderSessionKey))
                {
                    // If this is the first request, then create a PartialLoader and start generating.
                    partialLoader = context.RequestServices.GetRequiredService<PartialLoader<Cat>>()
                        .SetTimeout(TimeSpan.FromMilliseconds(timeout))
                        .SetPaging(paging)
                        .SetDataProvider(GenerateManyCats(count, delay))
                        .SetIsNullEnding(true)
                    ;
                    key = Guid.NewGuid().ToString();
                    loaderStorage.Data[key] = partialLoader;
                }
                else
                {
                    // If this is a subsequent request, then take the PartialLoader from the store and continue generating.
                    key = context.Request.Headers[Constants.PartialLoaderSessionKey];
                    partialLoader = loaderStorage.Data[key];
                }

                JsonSerializerOptions jsonOptions = new JsonSerializerOptions { PropertyNameCaseInsensitive = false };
                jsonOptions.Converters.Add(new TransferJsonConverterFactory(context.RequestServices)
                    .AddTransient<ICat>());

                // Add a response header with a request series ID.
                context.Response.Headers.Add(Constants.PartialLoaderSessionKey, key);

                // We get a portion of data, simultaneously writing them to the stream
                await context.Response.WriteAsJsonAsync(partialLoader, jsonOptions).ConfigureAwait(false);

                if (partialLoader.State is PartialLoaderState.Full)
                {
                    if (key is { })
                    {
                        loaderStorage.Data.Remove(key);
                    }
                }

            });