I’m probably the only one here that looks at the stupid prices of hardware (RAM, GPUs, drives etc) as a fun challenge. As in “OK…shit’s expensive…what’s the MOST I can do with the LEAST?”.
I think that comes from spending time in the lowspecgamer culture, where folks (hello!) will literally recode the game engine, bridge motherboard pins with paperclips and drill holes into the PC before buying new hardware.
For example, I specced out a (very decent) “sneakernet” / internet mirror for myself, as an intellectual exercise…and every time I run the numbers, they go from 8TB on the low end to 16TB on the high.
That includes mirroring somewhere around 3-5yrs of YouTube videos (in glorious 540-720p).
I can (and do) host an obscene amount of stuff on a SBC with a 256GB M.2.
Given narrow jobs like DNS, VPN, media, sync, search and automation and you can do a stupid amount with pi4 4GB.
Same thing with LLMs. People try to solve the problem of LLMs with bigger models, bigger GPUs etc…and I’m over here thinking “actually, the LLM is really just the mouthpiece; everything else around it is just good old programming”.
Memory, search, STT and TTS, OCR, file tools…none of that needs to live in the model or chew up a GPU…and once normal software handles that stuff, the question becomes: how small can the LLM be and still be useful middleware?
Turns out…pretty fucking small.
Honesty, I don’t think prices are coming down any time soon (RAM, 8TB NVMe drives, Threadrippers, 25GbE gear…hell, I saw a Blackwell yesterday on “sale” for $15K USD) so we might need to get creative.
Personally, I’m building towards resilience and independence. My next DIY will be the solar system… and because I’m paring down my homelab significantly, I can get away with a micro-system for around $2K.
Do wish hard drives were cheaper tho. Might have to go back to my “DVD as back up server” idea.
TL;DR: the only way to win is not to play the game…or at least, play it by your own rules.