How FP16 embeddings are converted to FP32 tensors in GPT-2 example ?

[astariul]

In the main.cpp of GPT-2 example, when converting the token IDs to a tensor through the embeddings, we use the ggml_get_rows operation :

https://github.com/ggerganov/ggml/blob/6b846cbde81ae02cd3e363311180ae706091933e/examples/gpt-2/main.cpp#L596

But isn't it wrong if we have weights in FP16 ?

---

wte is defined here :

https://github.com/ggerganov/ggml/blob/6b846cbde81ae02cd3e363311180ae706091933e/examples/gpt-2/main.cpp#L253

And the type of this tensor is checkpoint dependent, so it can be FP16.

But the problem is that the ggml_get_rows operation always return FP32 tensors :

https://github.com/ggerganov/ggml/blob/6b846cbde81ae02cd3e363311180ae706091933e/src/ggml.c#L4763-L4765

So how the conversion from FP16 to FP32 happens ? I can't see it either in ggml_get_rows or the main.cpp...

---

I'm asking this because I'm trying to debug my own model, and I found that the output of wte is different in ggml than in python !

[YavorGIvanov]

Here is where it happens https://github.com/ggerganov/ggml/blob/6b846cbde81ae02cd3e363311180ae706091933e/src/ggml.c#L10447