Benchmark TorchSharp GPT NEO performance with respect to those of PyTorch and C++ (ONNX)

[GeorgeS2019]

GPT NEO: better performance of python GPT NEO than its onnx runtime version in C++

In this issue, it seems PyTorch performs better than the c++ (ONNX runtime).

This is related to TorchSharp Deep NLP or NLG (Natural Language Generation) discussions

The codes are provided.

I hope community members here could work together to benchmark Torchsharp using this hot GPT topic given the benchmark codes and use case provided by the community member from ONNX Runtime.

FYI
@fwaris
@ChengYen-Tang

import torch

from transformers import GPTNeoForCausalLM, GPT2Tokenizer
model = GPTNeoForCausalLM.from_pretrained("EleutherAI/gpt-neo-1.3B")
tokenizer = GPT2Tokenizer.from_pretrained("EleutherAI/gpt-neo-1.3B")
device = torch. Device('cpu')

• GPT2Tokenizer for .NET is available here
• GPT-NEO and WIP instruction/discussion on how to convert PyTorch weight file from HuggingFace to be usable for TorchSharp

[NiklasGustafsson]

@GeorgeS2019 -- is this an issue or a discussion topic?

[GeorgeS2019]

@NiklasGustafsson

This issue has been added to ONNX Runtime backlog

This is a request (most likely long-term) for the benefit for new users interested to push for TorchSharp GPT to consider a contribution to add an additional selling point for TorchSharp's readiness in production through the "official" benchmarking with respect to PyTorch and ONNX c++

[GeorgeS2019]

Similar class of TorchSharp benchmarking is provided here

[NiklasGustafsson]

Okay, I'm going to convert this to a discussion. This is not a feature request or a bug, or a report of missing features or documentation. If you think benchmarks are important to the future of TorchSharp, please make the case for it in more detail in the discussion thread and build the community support for it there. I would suggest broadening the benchmarking case to include other scenarios than GPT.