Helper methods and convenience wrappers

[mfagerlund]

I'm currently working on as set of helper methods and convenience wrappers, stuff to cut down on boilerplate code. I'm talking about stuff like building and training networks. Helper methods to convert c# data (float[,], float[,,]) to tensors. Managing samples with features and labels. Providing a way to shuffle and split a large block of samples for training.

If that's something what would be interesting to the community, let me know!

Here's the classic Irish Mushroom Classification problem solved using my wrappers. And TorchSharp, of course.

[NiklasGustafsson]

I think this is very interesting stuff. I think it belongs in its own repository, at least until it stabilizes.

Working on datasets and dataset iteration is another big gaping hole right now, so if you have any interest in helping with that, it would be great. What's in the Examples in terms of data sets is just a toy.

[mfagerlund]

Not quite sure what you mean by "datasets and dataset iteration", but I think you mean easily shuffling data between iterations.

I currently load all my data into one huge tensor for features and one for labels and then have a simple method for creating batches from these.

        public static IEnumerable<Samples> CreateBatches(torch.Tensor features, torch.Tensor labels, int batchSize = 100, bool randomize = true)
        {
            var indices = Enumerable.Range(0, (int)features.shape[0]);
            foreach (var batchIndices in indices.Batched(batchSize, randomize))  // The Batched method is a helper method, doesn't exist in your c#
            {
                using var indicesTensor = torch.tensor(batchIndices, device: features.device);
                var batchFeatures = features.index_select(0, indicesTensor);
                var batchLabels = labels.index_select(0, indicesTensor);
                yield return new Samples(batchFeatures, batchLabels);
            }
        }

    public class Samples
    {
        public Samples(torch.Tensor features, torch.Tensor labels)
        {
            Features = features;
            Labels = labels;
        }

        public torch.Tensor Features { get; set; }
        public torch.Tensor Labels { get; set; }
        public float Count => Features.shape[0];
        public long InputCount => Features.shape[^1];
        public long OutputCount => Labels.shape[^1];

        public IEnumerable<Samples> CreateBatches(int batchSize = 100, bool randomize = true) => torchu.CreateBatches(Features, Labels, batchSize, randomize);
    }

One "Samples" object (I hate naming a class in plural, but what can you do) can contain all your data, but it can also represent a single batch of data.

For very large tasks, I think several separate sample collections would have to be used.

[GeorgeS2019]

@mfagerlund I do not speak for Microsoft, We appreciate your passion and enthusiasm. TorchSharp at this stage needs people like yourself to contribute to make others able to use and learn TorchSharp faster.

One guy who shared at the recent hackathon that it took him 2 weeks of hard work to learn TorchSharp and after that it is productive.

Please keep posting your ideas at discussion.

[mfagerlund]

I started two days ago. But I have written 3 (!) autodiffrentiation/NN packages for fun. Why 3? Just stupid I guess ;)

[NiklasGustafsson]

Not quite sure what you mean by "datasets and dataset iteration",

torchvision and torchtext both come with 'datasets' modules, which are super-useful for getting started. They also come with a common set of iteration constructs, etc.

One of the challenges here is that some data sets are small enough to fit in memory, while others do not. Some fit in CPU memory, but not in GPU. That means that specialized iteration constructs need to take that into account and sometimes read every mini-batch from disk, sometimes cache it in CPU memory, sometimes it can stay resident in GPU memory. That requires a little bit of thought and design work.

All while taking randomization and sampling into account...

[mfagerlund]

Yeah, that sounds like a bit of work. I'll see what I can come up with once I start looking at some larger datasets.

The dream scenario would be that the user didn't have to know about what method will be used in the background, the user would specify a RAM budget and the system would deal with the rest in the background.

[GeorgeS2019]

@mfagerlund If you have time, please put some thinking into how to bring models saved in PyTorch to TorchSharp. There in an ongoing discussion. If with luck you have any idea to share, please start another discussion topic.

I believe the .NET community needs feedback how to shuffle models trained in PyTorch to TorchSharp and ML.NET (direct or indirectly through integrated TorchSharp).

Some ideas, see the link above, is to keep everything to ONNX. We are very curious what you think of that, when you have time, after a couple of beers :-)

[mfagerlund]

Ah, I wasn't even aware that that was a difficult thing to do. I'll have a peek later on. Sounds like it could be complicated, though - perhaps we'd have to export a portable version from pytorch that could be imported easier?

[GeorgeS2019]

@mfagerlund It requires simple yet addressing complex issues involving PyTorch, TorchSharp and ML.NET. I am with the position that, the exiting ONNX format needs further improvements to accommodate the need for e.g. TorchSharp to save and import all models ONLY in ONNX. There is no need for proprietary TorchSharp saved model.

One FORMAT e.g ONNX "to rule all Machine learning platforms" <= borrowing from Lord of the Ring, One Ring to rule all

When exporting PyTorch Models to ONNX, there is always the challenge that sometimes ONNX max not have the operators. Hence, we now have ONNX extensions.

Instead of making PyTorch, TorchSharp to address these issues separately on their own, WE NEED these platforms to conform with the operators available in ONNX. If not there, find ways to re-write PyTorch code so it is consistent with the existing ONNX. If this is not possible, feedback to ONNX team for change.

[NiklasGustafsson]

The way serialization works in PyTorch is very closely tied to Python's pickling system, or by using TorchScript.

With the former, the model is not serialized, it's just the weights. You have to have the source code of the model to load weights. That's true in Python as well as .NET. We have a special format that does not depend on Python pickling that allows you to exchange weights between PyTorch and TorchSharp, but you do have to have the model represented in source code in order to load.

https://github.com/dotnet/TorchSharp/blob/main/docfx/articles/saveload.md