Auto Suggestion Improvement

[lei4519]

Background

Just now, I realized that avante's auto suggestion is not copilot, but cursor ++(cursor tab), and this video demonstrates its capabilities

My.Movie.mp4

The video has been speeded up, and it's actually very slow

You can see that it provides document-wide multi-line suggestions (cursor tabs) instead of text-only completion at the cursor (copilot).

For some relatively simple requirement scenarios, we don't want to "describe the requirement - wait for the response - check the result" in the chat box, we enter the accurate and complete code is much faster than it.

So cursor tab is a killer feature for many people, and the ai automatically guesses the requirements based on what we type and gives complete completion suggestions

It's funny that I was still looking for a replacement for cursor tab, not realizing that Avante has already implemented it. Here are some community posts

• https://www.reddit.com/r/neovim/comments/1gzo0zt/neovim_cursor_tab_alternative/
• https://www.reddit.com/r/ChatGPTCoding/comments/1i2bz6m/cursor_tab_is_amazing_are_there_any_emerging_open/
• https://www.reddit.com/r/neovim/comments/1i79ssy/cursor_with_vim_mode_vs_avante/

Problem

Now there are two problem:

1. Very slow

• This doesn't fit the auto suggestion usage scenario: If the AI doesn't respond immediately, we continue typing characters instead of waiting for it

2. The token consumption is huge
   In a normal project, there are usually hundreds or thousands of lines of code. I just tested it for one afternoon, and I burned through a few million tokens

Solution ideas

For an open source project, it is difficult to train a large model specifically for this function like cursor

We can only rely on the power of the big models themselves, and the good news is that some big models (deepseek/claude) provide caching capabilities

• https://api-docs.deepseek.com/news/news0802
• https://www.anthropic.com/news/prompt-caching

When the cache is hit, it can not only reduce the consumption of tokens, but also improve the response speed

• deepseek

• claude

This means that if we can take advantage of the caching capability, we can solve both problems at the same time

Concrete implementation

If you want to use the power of caching, you need to transfer the conversation message incrementally

The current Suggestion implementation sends the entire contents of the file in full with each session

  local code_content =
    Utils.prepend_line_number(table.concat(api.nvim_buf_get_lines(bufnr, 0, -1, false), "\n") .. "\n\n")

  Llm.stream({
    selected_files = { { content = code_content, file_type = filetype, path = "" } },
    mode = "suggesting",
})

We may need the following modifications:

• Cache the history message of each buffer in memory
• Each buffer fires suggestion for the first time, passing the entire file content
• Subsequent trigger suggestion, append the part that sends changes to the history message (such as git diff/vimdiff)
• Guide large models to make suggested responses based on incremental changes

I have a simple demo here (ignoring the transmitted message format, this is just for my own testing)

Demo

rules.md

user:

1. export const Test = () => {
2. return <div>hello</div>
3. }

ai:
ok

user:
[{
action: 'add',
line: 2,
content: ' const [input]'
}]

ai:
[{
action: 'edit',
line: 2,
content: ' const [input, setInput] = useState("")'
},{
action: 'edit',
line: 3,
content: 'return <input value={input} onChange={e => setInput(e.target.value)} />'
}]

chat history:

- Datetime: 2025-01-23 23:07:31

- Model: copilot/claude-3.5-sonnet

- Selected files:
  - rules.md

> 1. export const Test = () => {
> 2. return <div>hello</div>
> 3. }

ok

---

- Datetime: 2025-01-23 23:07:59

- Model: copilot/claude-3.5-sonnet

- Selected files:
  - rules.md

> [{
> action: 'add',
> line: 2,
> content: ' const [input]'
> }]

[{
action: 'add',
line: 1,
content: 'import { useState } from "react"'
},{
action: 'add',
line: 3,
content: ' const [input, setInput] = useState("")'
},{
action: 'edit',
line: 4,
content: ' return <input value={input} onChange={e => setInput(e.target.value)} />'
}]

---

- Datetime: 2025-01-23 23:09:19

- Model: copilot/claude-3.5-sonnet

- Selected files:
  - rules.md

> [{
>
> > action: 'add',
> > line: 4,
> > content: ' const handleChan'
> > }]

[{
action: 'add',
line: 4,
content: ' const handleChange = (e) => setInput(e.target.value)'
},{
action: 'edit',
line: 5,
content: ' return <input value={input} onChange={handleChange} />'
}]

---

- Datetime: 2025-01-23 23:13:59

- Model: copilot/claude-3.5-sonnet

- Selected files:
  - rules.md

> [{
>
> > action: 'add',
> > line: 4,
> > content: ' // 处理中文输入法的问题'
> > }]

[{
action: 'add',
line: 3,
content: ' const [isComposing, setIsComposing] = useState(false)'
},{
action: 'edit',
line: 4,
content: ' // 处理中文输入法的问题'
},{
action: 'edit',
line: 5,
content: ' const handleChange = (e) => {'
},{
action: 'add',
line: 6,
content: ' if (!isComposing) {'
},{
action: 'add',
line: 7,
content: ' setInput(e.target.value)'
},{
action: 'add',
line: 8,
content: ' }'
},{
action: 'add',
line: 9,
content: ' }'
},{
action: 'edit',
line: 10,
content: ' return <input'
},{
action: 'add',
line: 11,
content: ' value={input}'
},{
action: 'add',
line: 12,
content: ' onChange={handleChange}'
},{
action: 'add',
line: 13,
content: ' onCompositionStart={() => setIsComposing(true)}'
},{
action: 'add',
line: 14,
content: ' onCompositionEnd={(e) => {'
},{
action: 'add',
line: 15,
content: ' setIsComposing(false)'
},{
action: 'add',
line: 16,
content: ' setInput(e.target.value)'
},{
action: 'add',
line: 17,
content: ' }}'
},{
action: 'add',
line: 18,
content: ' />'
}]


---

- Datetime: 2025-01-23 23:16:51

- Model: copilot/claude-3.5-sonnet

- Selected files:
  - rules.md

> [{
>
> > action: 'edit',
> > line: 3,
> > content: ' const isComposing = useRef(false)'
> > }]

[{
action: 'add',
line: 1,
content: 'import { useState, useRef } from "react"'
},{
action: 'delete',
line: 3,
content: ' const [isComposing, setIsComposing] = useState(false)'
},{
action: 'add',
line: 3,
content: ' const isComposing = useRef(false)'
},{
action: 'edit',
line: 7,
content: ' if (!isComposing.current) {'
},{
action: 'edit',
line: 14,
content: ' onCompositionStart={() => isComposing.current = true}'
},{
action: 'edit',
line: 15,
content: ' onCompositionEnd={(e) => {'
},{
action: 'edit',
line: 16,
content: ' isComposing.current = false'
}]

finally, Is anyone doing something similar? Or how do you feel about the feasibility of this