Edit history

Most recent⁨Jul⁩ ⁨27⁩, ⁨2026⁩, ⁨16:12⁩

You want this one:

Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more:

For faster inference at the cost of a little more VRAM usage:

And you run those in:

And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead.

Original⁨Jul⁩ ⁨27⁩, ⁨2026⁩, ⁨16:11⁩

You want this one:

Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more:

For faster inference at the cost of a little more VRAM usage: