Shifu@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 21 days agoSpent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet.plus-squaremessage-squaremessage-square5linkfedilinkarrow-up119arrow-down13
arrow-up116arrow-down1message-squareSpent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet.plus-squareShifu@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 21 days agomessage-square5linkfedilink
Shifu@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 1 month agoBitNet b1.58-2B-4T at 36 tok/s on Xeon - pure C99, no Python/BLAS, single binary for ternary + GGUFplus-squaremessage-squaremessage-square0linkfedilinkarrow-up16arrow-down12
arrow-up14arrow-down1message-squareBitNet b1.58-2B-4T at 36 tok/s on Xeon - pure C99, no Python/BLAS, single binary for ternary + GGUFplus-squareShifu@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 1 month agomessage-square0linkfedilink
