GPU processing aborted after some time : GGML_ASSERT : ggml-cuda.cu:3572: src0->type == GGML_TYPE_F16

[sroleden]

I am running on the llama2 llama-2-7b-chat-codeCherryPop.ggmlv3.q2_K.bin model for embedding using LlamaCppEmbeddings documents and store them in FAISS vector store. I am using the Ubuntu OS and I am using the latest llama-cpp-python and other libraries.

embedding = LlamaCppEmbeddings(model_path=model_path, n_gpu_layers=50, n_batch=256, n_threads=96, n_ctx=4096 )

vectordb = FAISS.from_documents(docs,embedding=embedding)

when I start the program it start consuming the GPU power, however after 10 to 15 minutes, it aborts the process.
please find the last log line.

llama_print_timings: load time = 1404.06 ms
llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: prompt eval time = 3880.49 ms / 118 tokens ( 32.89 ms per token, 30.41 tokens per second)
llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: total time = 3882.83 ms
GGML_ASSERT: /tmp/pip-install-ograoe9x/llama-cpp-python_7a7e3c28c4a442aaa85d68b46a7ae4ca/vendor/llama.cpp/ggml-cuda.cu:3572: src0->type == GGML_TYPE_F16
run.sh: line 9: 304489 Aborted (core dumped) python process.py

does anyone has an idea bout this issue ?

[hengzi52125]

I'm facing the same issue, and I was using the ggml-model-f16.bin model, which is indeed in FP16 format. I'm not sure where the problem is coming from.