[AIT] I know this isn’t everyone’s cup of tea, but I’m excited about Deepseek V4 flash. It’s (for me) the perfect size and architecture to self-host an LLM.
My box (and brain) are chugging through a queue:
-
Figure out why my swap is going crazy, and how to ban processes from it [Done].
-
Figure out why Code OSS is unhappy [Partially Done].
-
Make an ik_llama.cpp iMatrix for Deepseek V4 [Done].
-
Figure out why quantization isn’t working [Done].
-
Make a test IQ2_KL/MXFP4_R8 quant to see how it does squeezed onto my box [in progress].
-
Test. Tune. Inevitably troubleshoot the dozen other things that go wrong. Figure out how much spare RAM that leaves me.
-
Make a higher quality IQ3_KT quantization. This will take all night on my CPU.
-
KLD test both of them vs the full precision, to quantify quantization loss. Likely an overnight task, too.
-
Try merging the new model release with the base model, 50/50, for a less “deep fried” model. imatrix, quant, test.
The goal is to host it on a single RTX 3090, Ryzen 7000 with 128GB RAM, for anyone curious. Though I may try smaller models too, like Laguna S1.