How is genAI compatible with Onnxruntime API?

[elephantpanda]

The GenAI is very nice to get things up and running, and I installed the c# DirectML version from nuget. Unfortunately it is not a very low-level API so it's hard to do things like manipulate the input/output token vectors etc. Is it compatible at all with the Onnx managed runtime for c# or is it entirely it's own thing? I feel like I may have to rewrite most of it using the lower level Onnx Runtime c# api.

For example there is the function processor.ProcessImages which creates a namedTensors object. But there's not much I can do with that except feed it into the SetInputs function. Ideally I would like to inspect it, or use the tokenizer to decode it or things like that.

[lilhoser]

I have a similar question regarding configuring Cuda EP. As discussed here https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements, there are some aspects of CUDA EP that need tweaking for different models, but the current onnxruntime-genai does not provide an interface to these APIs or a way to pass session options-

var cudaProviderOptions = new OrtCUDAProviderOptions(); // Dispose this finally

var providerOptionsDict = new Dictionary<string, string>();
providerOptionsDict["device_id"] = "0";
providerOptionsDict["gpu_mem_limit"] = "2147483648";
providerOptionsDict["arena_extend_strategy"] = "kSameAsRequested";
providerOptionsDict["cudnn_conv_algo_search"] = "DEFAULT";
providerOptionsDict["do_copy_in_default_stream"] = "1";
providerOptionsDict["cudnn_conv_use_max_workspace"] = "1";
providerOptionsDict["cudnn_conv1d_pad_to_nc1d"] = "1";

cudaProviderOptions.UpdateOptions(providerOptionsDict);

SessionOptions options = SessionOptions.MakeSessionOptionWithCudaProvider(cudaProviderOptions);