As an addendum I forgot:

Your idea was actually already implemented:

github.com/ggml-org/llama.cpp/pull/9742

XTC is a novel sampler that turns truncation on its head: Instead of pruning the least likely tokens, under certain circumstances, it removes the most likely tokens from consideration.

It’s a very cool idea: it chops off the most likely token when the rest of the sampling indicates it probably can.

In my tests back then, the results were… mixed, but the idea is fascinating.

But you can’t do this with Claude, as its a closed model and their sampling is years behind cutting edge.

See: gist.github.com/…/71ba712f9f899adcb08b94bce20d539…