#278: Optimise the LLM Client

While we now have a script to access a local LLM, we have a usability problem: We have to wait until the LLM has found an answer before we see that something is going on. Until then our script looks dead, and we may think it failed while it just waits on the LLM. A word-by-word output as we get with the web interfaces or at least a paragraph-by-paragraph update may soften this problem. Let us see how we can shorten the feedback time.

Allow our own questions

So far, our client had the hard-coded questions about the Zen of Python. To increase user-friendliness, we need to give the user the opportunity to ask the question in which they are interested. For that we can use the input() function and ask the user for the question that we then send to the API:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:1234/v1", api_key="not_needed")

question = input("Please ask your question or 'end' to quit: ")

chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a Python expert. Provide accurate and concise responses."
        },
        {
            "role": "user",
            "content": question,
        },        
    ],
    model="gpt-4o",
)
print(chat_completion.choices[0].message.content)

That makes our client more useful. But the output is still slow, and we need to restart the script to ask another question. Let us shorten the time until we get the output.

Switch to asynchronous output

We can massively improve the perceived performance of our script when we do not wait until the whole answer is ready and instead, start the output as soon as we got the first part back. To get that optimised response we need to switch to the streaming client. All we must do is to add the parameter stream=True to our request and then print the different chunks we get back:

...
chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a Python expert. Provide accurate and concise responses."
        },
        {
            "role": "user",
            "content": question,
        },        
    ],
    model="gpt-4o",
    stream=True,
)

for chunk in stream:
        print(chunk.choices[0].delta.content or "", end="")

With this little change we now get feedback nearly instantly – even when it is only the starting tag of <think>. We still must wait for the whole answer, but now at least we see that something is going on.

Loop through the questions

The final change that we can do is to put the call to the OpenAI client in a loop. That way we can ask multiple questions in one go without restarting the script. To still reach an end we check for the input end, otherwise we keep looping:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:1234/v1", api_key="not_needed")
prompt = "Please ask your question or 'end' to quit: "

question = input(prompt)
print("You entered: " + question)

while question.strip() != "end":
    stream = client.chat.completions.create(
        messages=[
            {
                "role": "system",
                "content": "You are a Python expert. Provide accurate and concise responses."
            },
            {
                "role": "user",
                "content": question,
            },        
        ],
        model="gpt-4o",
        stream=True,
    )
    for chunk in stream:
        print(chunk.choices[0].delta.content or "", end="")

    question = input(f"\n\n{prompt}")

With that we turned our basic client into something useful, that offers a much faster feedback cycle when we experiment with our LLM.

More room for improvements

The paragraph-by-paragraph output is fast enough for me. But if you may want to go a step further and try something like word-by-word updates, you should have a look at the text components from Rich. I covered Rich in post #132 and it is still the go-to library if you want to have a fancy console output.

Another angle for improvements could be to cache the answers. Should you try to send the same question to the LLM, you could cut it short and repeat the answer you got the last time – out of a cache and not newly calculated through LM Studio. For that case we must make sure that we handle the switch in models, then this can happen in LM Studio without our Python script noticing it.

I leave these optimisations to you. Let me know what you came up with.

#278: Optimise the LLM Client

Allow our own questions

Switch to asynchronous output

Loop through the questions

More room for improvements

Next