How do you process an uploaded file to a Web API?

[mcpine]

Could you provide an example of ETL from an uploaded file to a Web API? Let's say the file was a .csv file. I do not see how to do this within your documentation.
Thank you very much!

[paillave]

Do you mean "how to process a file that is downloaded from an URL"?
Or do you mean "how to produce and upload a file to an URL"?

[paillave]

To process a file that comes from the web, here is what you can do:

using Paillave.Etl.Core;

var res = await StreamProcessRunner.CreateAndExecuteAsync(
    new Uri("https://www.thecodingguys.net/resources/cs-cheat-sheet.pdf"),
    t => t.Select("DownloadFile", i =>
{
    using HttpClient client = new HttpClient();
    return FileValue.Create(client.GetStreamAsync(i).Result, i.Segments.Last(), "From the web");
}));
Console.WriteLine(!res.Failed);

After the select operator, you can use any regular operator to process your file.

[mcpine]

Let's say I created a web form on my web site and I have an input that takes a file. When I submit my form it calls my web api to upload the file to this web api. Can you provide an example of how the web api using your ETL package can take the file from the form post request and parse the file? In this case a .csv file?

[paillave]

ok, U R talking about processing a file posted to a webapi where the action processes it.

So here's the way:

using Paillave.Etl.Core;
using Paillave.Etl.TextFile;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapPost("/upload", async (IFormFile file) =>
{
    var res = await StreamProcessRunner.CreateAndExecuteAsync(file,
        t => t
            .Select("Use uploaded file", i => FileValue.Create(i.OpenReadStream(), i.FileName, "Posted to API"))
            .CrossApplyTextFile("parse my text file", i => i.UseMap(m => new { Code = m.ToColumn("Code"), Label = m.ToColumn("Label") }))
            // And so on...
        );
    if (res.Failed)
        return Results.Problem($"[{res.ErrorTraceEvent.NodeName}]: {res.ErrorTraceEvent.Content.Message}");
    else
        return Results.Ok();
});
app.Run();

Note: not tested, maybe U'll have to make some small changes. Let me know, the community would love it.

[paillave]

Note for my self:
Make an method that permits to have MinimalApi that makes this in a snap of a finger. This would be a new nuget package: Paillave.EtlNet.WebApi

[mcpine]

Thank you very much! This is exactly what I needed. Here's my test example inside a MVC web api:

[HttpPost("uploadmassmerchantfile")]
[Produces("application/json")]
[AllowedExtensions(new[]{".csv", ".xslx"})]
[RequestSizeLimit(20 * 1024 * 1024)]
public async Task<IActionResult> UploadMassMerchantFile([FromForm] IFormFile file)
{
	string result = String.Empty;	
	if (file == null) {
		return BadRequest();
	}
	if (string.IsNullOrEmpty(Request.GetMultipartBoundary())) {
		return BadRequest();
	}
	var res = await StreamProcessRunner.CreateAndExecuteAsync(file,
		t => t
			.Select("Use uploaded file", i => FileValue.Create(i.OpenReadStream(), i.FileName, "Posted to API"))
			.CrossApplyTextFile("parse file", FlatFileDefinition.Create(i => new
			{
				SavingsType = i.ToColumn("SavingsType"),
			}).IsColumnSeparated(','))
			.Do("write to output", i => result = i.SavingsType)
	);
	if (res.Failed)
		return BadRequest($"[{res.ErrorTraceEvent.NodeName}]: {res.ErrorTraceEvent.Content.Message}");
	else
		return Ok();
}