As of April 2023, TorchSharp has addressed many of the pain points of ~900 ML.NET Apr2021 survey responses

[GeorgeS2019]

Apr 2021 survey outcomes requested

• Deep Learning support
• Address Deep AI scopes not supported by ML.NET especially deep NLP

Mission of TorchSharp

Original discussion title (Jul 9, 2021): How TorchSharp can address the pain points of ~900 ML.NET Apr2021 survey responses

WIP April 2023 Update

• Adopting PyTorch Syntax to ease the learning curve of Deep AI for .NET community

This controversial decision back in July 2021 turns out to be the RIGHT one and is key to growing adoption by the .NET community, especially for Deep AI scopes not available in the existing ML.NET

• One of the Apr 2021 survey outcomes was more support for Deep NLP in ML.NET

ML.NET now supports TorchSharp and Deep NLP to a certain scope. TorchSharp is the option when the Deep NLP scope needed is not available in the existing ML.NET

Previous April 2021 Discussions

The Apr 2021 ML.NET survey and the result discussions
It is clear that NLP is high on priority
This means **more deep learning NLP use cases ** e.g. using ML.NET to load pretrained Hugging Face transformer models using OnnxRuntime

TorchSharp is on track! (especially after the recent renaming effort to make the ML.NET import (more straight forwards) the transformer pretrained models in onnx)

Writing an issue to feedback that TorchText is the next step in development after TorchVision
Good Job!

[GeorgeS2019]

An illustration (after feedback changes from @NiklasGustafsson) of how similar is TorchSharp codes now to PyTorch!

[dsyme]

That comparison is really very cool. Would be good to see F# side-by-side too - have you got a repo hosting that code that some F# folk could help contribute an F# version to?

(The code would be basically identical , var --> let etc.)

[GeorgeS2019]

@dsyme @NiklasGustafsson

I have both codes properly format using Azure DevOps Server wiki markdown

Please send me back after you convert it to Fsharp. I will provide here the PyTorch vs TorchSharp (F#)

using TorchSharp;

// N is batch size; D_in is input dimension;
// H is hidden dimension; D_out is output dimension.
int N = 64; int D_in = 1000; int H = 100; int D_out = 10;

// Create random Tensors to hold inputs and outputs
var x = torch.randn(N, D_in);
var y = torch.randn(N, D_out);

// Use the nn package to define our model as a sequence of layers. nn.Sequential
// is a Module which contains other Modules, and applies them in sequence to
// produce its output. Each Linear Module computes output from input using a
// linear function, and holds internal Tensors for its weight and bias.
var model = torch.nn.Sequential(
            ("lin1", Linear(D_in, H)),
            ("relui",ReLU()),
            ("lin2",Linear(H, D_out))
           );

// The nn package also contains definitions of popular loss functions; in this
// case we will use Mean Squared Error (MSE) as our loss function.
var loss_fn = mse_loss(Reduction.Sum);

var learning_rate = 1e-4f;
for(int t = 0; t < 500; t++)
{   // Forward pass: compute predicted y by passing x to the model. Module objects
    // override the __call__ operator so you can call them like functions. When
    // doing so you pass a Tensor of input data to the Module and it produces
    // a Tensor of output data.
    var y_pred = model.forward(x);

    // Compute and print loss. We pass Tensors containing the predicted and true
    // values of y, and the loss function returns a Tensor containing the loss.
    var loss = loss_fn(y_pred, y);
    if( t % 100 == 99){
        Console.WriteLine(string.Format("step: {0} loss: {1}",t+1, loss.ToSingle()));
    }
    // Zero the gradients before running the backward pass.
    model.zero_grad();

    // Backward pass: compute gradient of the loss with respect to all the learnable
    // parameters of the model. Internally, the parameters of each Module are stored
    // in Tensors with requires_grad=True, so this call will compute gradients for
    // all learnable parameters in the model.
    loss.backward();

    // Update the weights using gradient descent. Each parameter is a Tensor, so
    // we can access its gradients like we did before.
    using (var noGrad = new AutoGradMode(false)) { 
        foreach (var param in model.parameters()) { 
            param.sub_(param.grad()*learning_rate);
        }
    }
}

[saint4eva]

Both looks like python to me. I think we should respect .NET idiosyncrasies and naming convention. These libraries are to be used by millions of .NET developers - so should lean towards .NET culture and sentiments. And not to please a few python developers who will be expected to use ML.NET.

[GeorgeS2019]

@saint4eva
The SciSharp community (e.g. Tensorflow.NET, NumpSharp etc. ) promotes "python like" naming to encourage more .NET developers to have access to Deep Learning => one of the pain points of ML.NET users (based on April 2021 survey)

There has been months of discussion on this naming topics for TorchSharp.

Therefore, the primary naming goal (for both Tensorflow.NET and TorchSharp ) is to empower NET developers have access to the deep learning development still missing in ML.NET (according to the survey) - instead of pleasing a few python developers.

==> Most important!
It is hope that ONCE more and more .NET developers are doing the type of needed deep learning for .NET community, they will contribute to address the missing deep learning examples (according to the survey) in ML.NET.

[dsyme]

@saint4eva All deep learning architectures are originally implemented in Python. Pretty much all deep learning is done in Python. The important thing to optimise is the efficiency of moving deep learning architectures, model implementations, optimizers, data-loading, training loops etc. to .NET, so you can get on with training. It's not about "pleasing a few Python devs" but rather the massive collection of assets that are available in python. For example look at all the Huggingface transformer implementations. There are ~100 there. Those are the things that we need to optimise bringing over.

This is just not like other .NET APIs.

[lostmsu]

@dsyme I only used TorchSharp for a little bit, but as a mainly C# developer I wholeheartedly agree with @saint4eva . The tooling around C# is made according to .NET class library design guidelines, and deviating from it too far makes powerful things useless. For example, discoverability is really bad for NN layers, because you need to use factory methods to create them. And then you need to also know where to find them.

This issue is caused by copying Python API verbatim: Python has module-level functions, and torch.nn is a module that has both classes and functions. In .NET torch.nn is an OK class to host functions (maybe .NET-style PascalCase would be better). But nesting classes into torch.nn is extremely counter-intuitive. C# tooling does global class search, but I think nested classes are excluded, and create unnecessary torch.nn. prefixes if found with most tools.

[NiklasGustafsson]

I have been very torn about this for several months. I love .NET and its naming conventions, and I agree that the aesthetics of a language are important to the community that uses it.

The driving reason behind our thinking is that 99% of all deep learning texts are relying on Python. TorchSharp is not so much about winning Python developers over to .NET, I don't think that is realistic. The purpose is to make the learning curve faster for .NET developers who are getting into deep learning.

The SciSharp community has already pioneered the idea of staying true to the Python naming conventions, in order to allow users to more readily take advantage of existing texts as guidance. It's not quite copy-and-paste at the moment, and it can never really be, but it's darned close.

We are planning to integrate TorchSharp into ML.NET, just as TF.NET is already integrated. When we do this, the higher-level APIs will (as per current thinking) follow .NET naming conventions and ML.NET patterns. We believe that the vast majority of .NET developers will want to rely on the higher-level APIs rather than the 'hard-core' TorchSharp APIs.

[GeorgeS2019]

Both Tensorflow.NET and TorchSharp face the challenge of keeping up with the rapid progress in latest AI developed using Tensorflow and PyTorch respectively. The goal is to enable and empower .NET developers to have access to the latest AI development while embracing the .NET6 enterprise/mobile devops rapid development cycle.

As @dsyme said, this requires efficient migration of the latest AI concepts, libraries/frameworks developed in python to .NET environment.

As @NiklasGustafsson said, this will grow and engage the .NET community in the latest AI concepts. Once we are ON TRACK to have a .NET ML community in the latest AI as active as that of the python community, the focus will be to make the latest AI available in ML.NET in a higher-level APIs with the primary focus => to make the difficult latest AI simpler to implement through ML.NET!. This is HOW I see Microsoft should democratize AI/ML! => make very hard AI/ML simpler for .NET developers.

[lostmsu]

@NiklasGustafsson I think the specific problem I mentioned can be solved without contradictions. I don't see a good reason to have classes nested (my biggest pet peeve). Can't we have torch.nn.Sequential factory and TorchSharp.NN.Sequential class instead of

Took a look at the new naming scheme, looks like the nested class issue is not present. But, for instance, Sequential class is missing a public constructor.

The problem that I see with "majority developers will want to rely on C#-style high-level APIs" is that the minority that would want to build on low-level TorchSharp itself would be appalled by its internals and coding style, which would stagnate the project.

As for the people reading deep learning books, it should not be too hard to also read and memorize 3-5 simple rules of how to find corresponding TorchSharp members. The argument seems moot here.

Besides, why not follow C++ API? C++ is much closer to C# and the PyTorch team itself admits it is more polished than Python version.

[dsyme]

which would stagnate the project.

This project will succeed if it's seen as making .NET viable in the PyTorch ecosystem (and that brings enough value to enough deep learning practitioners, or allows enough .NET people to play in the ecosystem). This project is not trying to create an independent .NET machine learning ecosystem.

by its internals...coding style

I presume you mean API style, so I'll answer that - if there are specific problems with the internals or coding style please let us know.

Regarding API style - I don't think so, and to be honest we've made the decision and we're moving on with the project. One day perhaps we'll revisit it, or perhaps someone will wrap this library. Here are some further observations for you:

1. The C# developers who are WeddedToPascalCase are not people making deep learning models (or if they are, they are doing it in Python).

2. Almost everyone interested in this project will be following PyTorch documentation and examples at some point. That means knowing PyTorch naming (whether C++ or Python)

3. I don't know anyone who seriously thinks PascalCaseNaming brings advantages to mathematical tensor programming. No one except .NET people will ever do it. There are some good libraries that have taken a lot of time to map names across into the .NET world, and I don't mean to disparage them (e.g. Math.NET Numerics, ExtremeOptimization). But those aren't shallow wrapper libraries like this one, and, crucially, those aren't trying to bring .NET into an existing ecosystem.

   Take just one example mvlgamma. What are we seriously going to use to follow C# naming conventions? The pointless capitalization MvlGamma? Or the impossible to remember MultivariateLogGamma? Neither are an improvement - people abbreviate this stuff for a reason. To be honest the Pytorch naming conventions including C++ follow the conventions of the mathematical programming universe and use lowercase, with many abbreviations.

4. Microsoft itself looked at TensorFlow.NET and said "yes, we'll rely on that for ML.NET". Despite the API design. Because it brings canonical value, and because the decisions made sense.

As for the people reading deep learning books, it should not be too hard to also read 3-5 and memorize simple rules of how to find corresponding TorchSharp members. The argument seems moot here.

Everytime I've tried to use a .NET math library I spend hours finding what I need to translate samples, all of it entirely unnecessary. There are literally hundreds or thousands of extra names, words and namespaces you need to know. mvlgamma is a good example, but looking through Tensor.cs there are many other examples.

In any case, this is open source, and you're welcome to fork - or better start a project that sits on top of this one and wraps the API with .NET names?

Besides, why not follow C++ API? C++ is much closer to C# and the PyTorch team itself admits it is more polished than Python version.

We do wrap the C++ API, but in terms of naming and API I can't particularly say it's better. e.g. see sparse here: https://github.com/xamarin/TorchSharp/blob/master/src/Native/LibTorchSharp/THSTensor.cpp#L1109

The C++ API is, however, getting better and better and I can see it will get a lot of use for model delivery (though not for original model design/experimentation/authoring). However for model delivery the operational differences are much more important - notably the C++ API has the huge advantage that more optimizations can be performed (I assume), and linking can remove everything that's not needed (where here we are wrapping the massive LibTorch binaries).

[lostmsu]

@dsyme I have no specific preference for case (although, MultivariateLogGamma is much easier to read). But here's a couple of screenshots to illustrate what I believe is an issue (in terms of nested classes and factory methods):

This one is from VS 2022 Preview. The autocomplete suggests Sequential, but that is the class, which, as I mentioned above does not have a public constructor. torch.nn.Sequential is completely missing in the dropdown. I must know to add using static TorchSharp.NN.Modules; or using static torch.nn; after the recent PR to find Sequential the factory method.

The same would happen if I tried ReLU or relu (casing does not matter). I would see the class, but not the factory function.

The same situation happens with VS 2019 and ReSharper installed:

Another case in point, re: porting existing PyTorch code from Python. Let's take OpenAI spinning up repository. I picked a random file there, that I never saw before: https://github.com/openai/spinningup/blob/master/spinup/algos/pytorch/ppo/core.py

A few screenshots:

As you can see, the naming is all over the place: torch.nn module is imported as just nn, which in C# would be the most inconvenient, as its technical equivalent is using nn == torch.nn;, that one has to always type in manually - no tool does it unless there's a name conflict.

If you try to port the Actor class as-is you will notice, that regardless of having or not having using nn = torch.nn; this will never compile:

class Actor: nn.Module
{
}

Because Module is not a member of torch.nn. But even if it were, you'd need to know to do using static torch or using nn = torch.nn from above for this to work, neither of which would be offered by VS or ReSharper.

[lostmsu]

Admittedly, F# is better in this regard:

But even it does not show any kind of information pertaining to which of the Sequential things you want.

[saint4eva]

@saint4eva
The SciSharp community (e.g. Tensorflow.NET, NumpSharp etc. ) promotes "python like" naming to encourage more .NET developers to have access to Deep Learning => one of the pain points of ML.NET users (based on April 2021 survey)

Don't you think that promoting python naming culture to encourage .NET developers would be counter-productive and counter-intuitive?

All I am saying is that whatever we are doing, we should always have it at the back of our mind that we are serving the .NET community/ developers. And mind you that .NET community is more principled than any other community - which naming convention is one of them.

Notwithstanding, thank you for your efforts. I appreciate.

[saint4eva]

which would stagnate the project.

This project will succeed if it's seen as making .NET viable in the PyTorch ecosystem (and that brings enough value to enough deep learning practitioners, or allows enough .NET people to play in the ecosystem). This project is not trying to create an independent .NET machine learning ecosystem.

by its internals...coding style

I presume you mean API style, so I'll answer that - if there are specific problems with the internals or coding style please let us know.

Regarding API style - I don't think so, and to be honest we've made the decision and we're moving on with the project. One day perhaps we'll revisit it, or perhaps someone will wrap this library. Here are some further observations for you:

1. The C# developers who are WeddedToPascalCase are not people making deep learning models (or if they are, they are doing it in Python).
2. Almost everyone interested in this project will be following PyTorch documentation and examples at some point. That means knowing PyTorch naming (whether C++ or Python)
3. I don't know anyone who seriously thinks PascalCaseNaming brings advantages to mathematical tensor programming. No one except .NET people will ever do it. There are some good libraries that have taken a lot of time to map names across into the .NET world, and I don't mean to disparage them (e.g. Math.NET Numerics, ExtremeOptimization). But those aren't shallow wrapper libraries like this one, and, crucially, those aren't trying to bring .NET into an existing ecosystem.
   Take just one example mvlgamma. What are we seriously going to use to follow C# naming conventions? The pointless capitalization MvlGamma? Or the impossible to remember MultivariateLogGamma? Neither are an improvement - people abbreviate this stuff for a reason. To be honest the Pytorch naming conventions including C++ follow the conventions of the mathematical programming universe and use lowercase, with many abbreviations.
4. Microsoft itself looked at TensorFlow.NET and said "yes, we'll rely on that for ML.NET". Despite the API design. Because it brings canonical value, and because the decisions made sense.

As for the people reading deep learning books, it should not be too hard to also read 3-5 and memorize simple rules of how to find corresponding TorchSharp members. The argument seems moot here.

Everytime I've tried to use a .NET math library I spend hours finding what I need to translate samples, all of it entirely unnecessary. There are literally hundreds or thousands of extra names, words and namespaces you need to know. mvlgamma is a good example, but looking through Tensor.cs there are many other examples.

In any case, this is open source, and you're welcome to fork - or better start a project that sits on top of this one and wraps the API with .NET names?

Besides, why not follow C++ API? C++ is much closer to C# and the PyTorch team itself admits it is more polished than Python version.

We do wrap the C++ API, but in terms of naming and API I can't particularly say it's better. e.g. see sparse here: https://github.com/xamarin/TorchSharp/blob/master/src/Native/LibTorchSharp/THSTensor.cpp#L1109

The C++ API is, however, getting better and better and I can see it will get a lot of use for model delivery (though not for original model design/experimentation/authoring). However for model delivery the operational differences are much more important - notably the C++ API has the huge advantage that more optimizations can be performed (I assume), and linking can remove everything that's not needed (where here we are wrapping the massive LibTorch binaries).

@dsyme I think being an f# developer, you would not understand the implication and cognitive overload of not sticking to C# naming. Maybe f# looks like python, I can see where your sentiments stem from.

Looking at a codebase and immediately understanding the patterns and API style is an optimisation. Taking decision to promote python to force .NET developers to play in the pytorch ecosystem does not sound well.

Have you asked yourself why Asp.Net Core is quite successful, and community members contributed to it? One can borrow ideas from other ecosystems but subsuming yourself in that ecosystem is not good.

Many ecosystems borrowed ideas from C# or .NET, but they did not copy the culture and idiosyncrasies verbatim - they adapted the ideas to fit their ecosystem.

Anyways, if you already have made your decision I wish you all the best,

[GeorgeS2019]

Imagine, a few lecturers at Universities from different corners of the world start to teach students on deep learning using TorchSharp (likewise for Tensorflow.NET). The naming change adopted here enable many thousands of students to simply access the abundant PyTorch/Tensorflow education materials (written for python) but start to write deep learning in either c# or f# AND eventually supply the work force companies need to adopt deep learning in .NET environment.

Once we achieve this critical mass of .NET deep learning developers (who will actively contribute to many deep learning .NET repositories in GitHub), we can always revisit this discussion of staying truth to .NET convention later.

FYI => a few of the blogs I read about TorchSharp before the naming change were about the lack of documentation. Both Tensorflow.NET and TorchSharp suffer the lack of resources to make their documentation keeping up with the rapid development in tensorflow and pytorch. The naming change adopted here remove this obstacle.

We are all passionate of .NET AI/ML!

Change requires often some scarify among our belief systems along the way.

[NiklasGustafsson]

@GeorgeS2019 -- can you either distill a distinct set of asks from this issue, and then close it?

[GeorgeS2019]

@NiklasGustafsson there has been a follow up. If necessary, I will further feedback a set of separate issues in coming months.

[michieal]

As someone trying to use this API... jumping in (even though I have experience with AI Models in C/CPP) is a huge learning curve. Everything that I can find, use the old naming conventions, and yeah -- I'm bloody lost here.

Since most of the tutorials are only available if you are in windows, using the windows native IDE, VS... And not online in any other format... Even though 99% of AI development and deployment is done on LINUX... This is BEYOND frustrating, and that's a ****ing understatement.
(You'll grant me the courtesy of understanding my frustrations here, and the fact that I am doing my best to contain it.)

I have to ask -- is there a bloody conversion chart somewhere? I mean, what should be a simple task of loading up a pt/pth file, and appending it to another that is already loaded, has become a nightmare. I have to go out and learn python to try to use it in C#?! really?!

Did ya'll decide "oh, they won't mind..." when you made this needlessly obtuse and occulted? I wanna know!!!

[GeorgeS2019]

@michieal

I am doing my best to contain it

We are a group of users here with some of us very experience in handling new users who want to express frustration :-)
Go ahead, and we want your feedback!! We want to improve. That is our attitude!!

I have experience with AI Models in C/CPP

Do you work with e.g. PyTorch before? Do you only work with LibTorch using e.g. C/CPP?
This background information will help us.

That is it from my side Today :-) Welcome :-)

[NiklasGustafsson]

The decision to align the TorchSharp APIs with PyTorch was not one we took lightly -- we agonized over that one for months. We knew then, and still do, that there were going to be people who liked it, there were going to be people who disliked it, but tolerated it, and there were going to be people who were so turned off by it that they would not use the library. All of these have come true.

We made the decision because we knew we didn't have the resources, and never would, to replicate the vast amounts of documentation, examples, and tutorials that exist for PyTorch, so aligning with that API.

Since I agree with the idea that each language has its own aesthetics, which should be followed, especially in public APIs, it pained me to help make that decision, but I still believe that we would simply have ended up not maintaining TorchSharp for this long had we not been able to build on the PyTorch "soft" assets.

[michieal]

Well then, after two years... with no map on how to use this... I really have to wonder if you are going to reach your "critical mass" in this lifetime. And, if this decision "pained" you... why didn't you do things to make it easier on others? So far, in showing this api to others, the unanimous consensus has been a resounding "WTF?!"

Personally, I want to do DEEP Learning AIs in .Net. However, with the extreme lack of anything intelligible as to how to use this API... I am going to say that .Net isn't going to fade away -- this API however, not so sure about.

Take it for what you will, but, I think that you have seriously alienated most .Net programmers...

[GeorgeS2019]

Again we welcome strong feedback.

2 years ago, I totally agree with many of your points.
NOW..deep AI for .NET is here to stay.

I thanks many who volunteer to help to keep this vision going for last 2 years.

The next phase of Deep AI for .NET through TorchSharp will boldly go beyond no pythonic deep AI has ever gone.

Let us revisit this in 2 years..😃

The closer to irreversible success, the more strong feedback we receive...

This is a very good sign.

Please keep going 😄

[michieal]

Well, take the bloody feedback, and make a flippin' map!!! lmfao.

You say that this is a good sign, but from where I am sitting, I'm facepalming. So, do realize that continually frustrating people is not how you achieve success. Helping them to get somewhere with your project -- that's how you achieve success. (Never thought I would have to spell that out to anyone, but yet... HERE WE ARE!)

[GeorgeS2019]

Please share the codes and let the community here help u

[NiklasGustafsson]

Last time I checked, both the tutorials and examples at dotnet/TorchSharpExamples built and ran on Linux and MacOS, as well as Windows. If that has regressed, please file a bug in that repo.

TorchSharp is a thin library on top of libtorch, and the API design was done to make it straightforward to build on the plethora of Python-based examples and tutorials that are out there, since we do not ahve the resources to create our own.

I can (most of the time) copy-and-paste tensor and module expressions from Python into C#, but there are inherent differences that cannot be overcome without programmer involvement:

1. Python and .NET memory management are different.
2. Python's syntax for passing arguments by name is different from C#'s.
3. We can't mimic the module invocation syntax module(x) without resorting to using dynamic, so (with the latest release) it's module.call(x) in C#.
4. Python's and C#'s statement/class/method, etc. syntax are obviously very different.

I'm not sure what the old API you are referring to is -- we switched to Python-like naming conventions, following the SciSharp community in that regard, and the PyTorch scope hierarchy (which forced us to use a lot of static classes everywhere) a very long time ago. The examples and tutorials online at dotnet/TorchSharpExamples do not use the old version of the APIs.

Here are some other resources:

https://github.com/dotnet/TorchSharp/wiki
https://github.com/dotnet/TorchSharpExamples/tree/main/src
https://github.com/dotnet/TorchSharpExamples/tree/main/tutorials/CSharp

[GeorgeS2019]

We are flexible with understanding
if u need help, let us know...😃

[michieal]

Well, that's encouraging. Thank you, @GeorgeS2019

[GeorgeS2019]

I get lot of help here...I speak from good experience

[NiklasGustafsson]

However, if you ask ChatGPT for a code example, you're going to get "pre-name change" code. And most google results give you the same too.

ChatGPT is utterly unreliable when it comes to generating code for TorchSharp, and I don't think it's because it's seen a lot of 'pre-name-change' code (there wasn't much). It's not seen a lot of TorchSharp code, period, and so is generally hallucinating what it thinks the API should look like, following C# naming, etc. It's decent at explaining TS code, but useless at generating it.

[michieal]

I was trying any avenue I could find, to get help, as I downloaded TS via Nuget. (ChatGPT actually suggested TS to me.)
So, it hallucinates, but at least it's a "spokesperson" for you. lol

[NiklasGustafsson]

But, I guess that's the past. I'll go read the resources that you gave me and see what I can figure out.

I am more than happy to help you (and anyone else) get over the learning curve, which may lead to better tutorials, examples, and Wiki articles. It helps if you have specific questions that you can raise to get the ball rolling.

I'll be out of the office for a week starting Monday, just FYI.

[michieal]

Well, here's the smallest code snippet from the source code. this is in the Model.py file. (I figure that it's small enough to work with here, to get an understanding)

class FeedForward(nn.Module):
    def __init__(
        self,
        dim: int,
        hidden_dim: int,
        multiple_of: int,
    ):
        super().__init__()
        hidden_dim = int(2 * hidden_dim / 3)
        hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) // multiple_of)

        self.w1 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )
        self.w2 = RowParallelLinear(
            hidden_dim, dim, bias=False, input_is_parallel=True, init_method=lambda x: x
        )
        self.w3 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )

    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

I'm guessing, that I would use TorchScript to construct this? or, am I off there?

Also, I am today years old learning that python has lambda declarations. lol.

[NiklasGustafsson]

No, you would translate to C# manually. It would look something like (I didn't try to compile it):

public class FeedForward : torch.nn.Module<Tensor,Tensor>
{
    private ColumnParallelLinear w1;
    private RowParallelLinear w2;
    private ColumnParallelLinear w3;

    public FeedForward(int dim, int hidden_dim, int multiple_of) : base(nameof(FeedForward))
    {
        var hidden_dim = (int) 2 * hidden_dim / 3.0;
        hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) / multiple_of); // The last division must be integer division.

        w1 = new ColumnParallelLinear(dim, hidden_dim, bias: false, gather_output: false, init_method: x => x);
        w2 = new RowParallelLinear(hidden_dim, dim, bias: false, gather_output: false, init_method: x => x);
        w3 = new ColumnParallelLinear(dim, hidden_dim, bias: false, gather_output: false, init_method: x => x);
        RegisterComponents();
    }

    public override Tensor forward(Tensor x)
    {
        using _ = torch.NewDisposeScope();
        return w2.forward(functional.silu(w1.forward(x)) * w3.foward(x)).MoveToOuterDisposeScope();
    }
}

[NiklasGustafsson]

You can definitely use TorchScript, too, if PyTorch is able to export the model. However, it will then be a black box and you won't be able to modify it or use it to learn the details of how to use TorchSharp. The one benefit is you don't have to mess with translating the code.

[michieal]

Well, I would rather learn. I'm not a fan of black boxes, especially in regards to code. And, my design goals means that I would need to use LLaMA in conjunction with a pre-filtering AI module to convert the user input into a viable input for the llama section. (So trying to not toss out the word "module" all over, and confuse the subject. lol.)

For that part, I was thinking that a BERT model would work well, as I am trying to (ultimately) make an AI assistant that you can ask questions and get a creative / helpful / mostly factual response.

[NiklasGustafsson]

Just to give you an idea about the difference between call() and forward(), here's the Module<T,TResult> implementation of call:

public TResult call(T input)
{
    // Call pre-hooks, if available.

    foreach (var hook in pre_hooks.Values) {
        var modified = hook(this, input);
        if (modified is not null)
            input = modified;
    }

    var result = forward(input);

    // Call post-hooks, if available.

    foreach (var hook in post_hooks.Values) {
        var modified = hook(this, input, result);
        if (modified is not null)
            result = modified;
    }

    return result;
}

You should only implement (i.e. override) forward in your custom module.

[NiklasGustafsson]

Ah, yes -- something really hard to start out! :-)

PyTorch relies on Python pickling for saving model and optimizer state. That is a magical serialization format, but it is tightly coupled to the Python object model and runtime. There are libraries like https://github.com/irmen/pickle that can handle unpickling in C#, with limitations, but it doesn't unpickle classes as classes, it restores them as Dictionary<string,object>, which is not sufficient for our needs -- modules need to have their logic restored, too, including all the calls to native code.

So, we have had to rely on two separate solutions for sharing module state (weights + buffers) between Python and .NET:

1. TorchScript -- this has no dependence on the Python runtime and has good performance. TorchSharp supports loading and saving ScriptModules first created in Python (traced as well as scripted), but not creating them from scratch. Not all models are supported by TorchScript (this is not a TorchSharp limitation).

2. A custom format for saving the state_dict in Python, then loading in .NET. For this, you have to recreate the exact model definition, and have an instance of it to load the state into. This is a tedious process, but works. We have a Python script to export the state from Python in the source repo (it can also export optimizer state now).

There are two articles under the 'Wiki' header that covers these topics. Hopefully, those are sufficient to get you started. If not, please let me know where the information gaps are.

https://github.com/dotnet/TorchSharp/wiki/Sharing-Model-Data-between-PyTorch-and-TorchSharp
https://github.com/dotnet/TorchSharp/wiki/TorchScript

There's also a discussion of serialization in one of the tutorials:

https://github.com/dotnet/TorchSharpExamples/blob/main/tutorials/CSharp/tutorial6.ipynb

[GeorgeS2019]

@michieal
https://github.com/wang1ang/SentencePieceWrapper

[NiklasGustafsson]

BlingFire may also be useful, and it's on NuGet:

https://github.com/microsoft/BlingFire

[michieal]

Thank you, both of you! This is definitely helpful!

[michieal]

Okay, so... I didn't die... but wow, the migraine. lol.

The github versions of SentencePiece (wrappers) are seriously incomplete, especially for something that wants to know information about the vocab & tokenizer model. So, I had to make my own wrapper, based on the original SentencePiece c++ code. I will probably go through later and expose the training features, but for now I don't think? I need it.

also, I am going to copypasta this to a new issue, so that it's not buried.

[michieal]

So, I moved my pieces of this thread, sans the wth?! part to #980

[GeorgeS2019]

@michieal

These are in F#, would be great is there is a HERO among us to port that to c#

• .Net interactive notebook to show how to create a BERT model in TorchSharp
• BERT model defined in TorchSharp
• GPT style language model in TorchSharp

[michieal]

I've never programmed in F#.

[michieal]

...And, ILSpy failed to translate it.

[FlatlinerDOA]

I've started a port of Andrej Karpathy's NanoGPT in C# - https://github.com/FlatlinerDOA/NanoGPTSharp
I had his basic Shakespeare generator working, now moving up the stack to support loading GPT2 pre-trained weights.

[GeorgeS2019]

@FlatlinerDOA

I have been monitoring your project. This is a treasure to the TorchSharp community.

I have been lobbying the Torchsharp approach that combines deep NLP ( e.g. GPT) and Deep RL.