Leebra Aii
login register

42x Faster Prompt Lookup Drafting in llama.cpp

22 days ago by cm0002 to c/Aii

save

share

report

Four changes to the n-gram caches of llama.cpp make drafting up to 41.6x faster, load the static cache up to 23.5x faster, and lower peak memory up to 2.65x.
Aii
Aii

@programming.dev

login for more options
391
477
276

AI related news and articles.

Rules:

  • No Videos.
  • No self promotion: Don't post links to your articles.

go to feed...