Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have no response when using the custom model client #407

Open
YuhaoCheng opened this issue Dec 17, 2024 · 1 comment
Open

Have no response when using the custom model client #407

YuhaoCheng opened this issue Dec 17, 2024 · 1 comment

Comments

@YuhaoCheng
Copy link

Thanks for your excellent work. However, I found some weird performance of the tool when working with the in-house model client.

Firstly, I define the client:

from openai import OpenAI
client = OpenAI(base_url="IN-HOUSE IP address", api_key="IN-HOUSE KEY")

And I register the model by using the config:

ell.config.register_model("CUSTOM", client)

Then I use the ell.simple:

ell.init(store="./ell_logdir", verbose=True)
@ell.simple(model="CUSTOM", client=client)
def my_lmp(prompt: str):
    """You are a helpful assistant."""
    return f"Respond to: {prompt}"

Finally, I use this simple function:

temp = my_lmp("Sam")
logger.info(temp)
logger.info("Finish")

But I got nothing:
image

By the way, if I do not use the ell to help me record the prompt, I can get the correct response.

@Metasymbols
Copy link

我在本地调ollama是可以成功的
`
import ell
from openai import OpenAI

client = OpenAI(
base_url='http://127.0.0.1:11434/v1',
api_key='ollama', # required, but unused
)

ell.init(verbose=True)

class PromptComponent:
def init(self, template: str):
self.template = template

@ell.simple(model="qwen2.5:7b-instruct-q4_K_M", client=client)
def generate_prompt(self, **kwargs) -> str:
    prompt = self.template.format(**kwargs)
    return prompt

def set_template(self, template: str):
    self.template = template

def get_template(self) -> str:
    return self.template

def main():
template = "你好,请用python给我实现:{name} 组件"
prompt_component = PromptComponent(template)
prompt_component.generate_prompt(name="提示工程")

if name == "main":
main()

`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants