Meta-Llama-3.1-70B-Instruct and HuggingFaceTGIGenerator?

[JulienGuyLI]

Hello,

I am calling Meta-Llama-3.1-70B-Instruct using Haystack v2.0's HuggingFaceTGIGenerator in the context of a RAG application:

Although this model is advertised as having a 128k context window size, as opposed to Meta-Llama-3.1-70B-Instruct's 8k context window size, I get the following error:

I am puzzled as to whether this model's context window is purposefully limited to 8k tokens by HuggingFace
or if this is an issue of compatibility with Haystack 2.0? Any hints would be appreciated here.

[Amnah199]

Hi @JulienGuyLI . Seems like this is an issue with Meta-Llama-3.1-70B-Instruct model. Here is a discussion related to this:
https://huggingface.co/spaces/huggingchat/chat-ui/discussions/430

[JulienGuyLI]

Thank you for bringing this up Amnah199, I will post on this thread and keep an eye on it.