Apologies for the basic question, but what’s the difference between GGML and GPTQ? Do these just refer to different compression methods? Which would you choose if you’re using a 3090ti GPU?
Apologies for the basic question, but what’s the difference between GGML and GPTQ? Do these just refer to different compression methods? Which would you choose if you’re using a 3090ti GPU?
As far as I know they are different types of quantization.
The main difference you have to keep in mind as an end user is that, currently, GPTQ needs the full model to load in VRAM (the memory of your GPU) while GGML can share layers between the system RAM and the VRAM.
Performance wise I think it depends on the foundational model used, I know some time ago someone (The_Bloke?) did some testing, but I read it on Reddit and I don’t feel like going to search for it.
There’s this interesting post on huggingface Link, but it’s pretty old and things could have changed (for example GGML has gone through different iterations).
I’m just going by memory, so take everything I wrote with a pinch of salt. I never personally used GPTQ.
Also llama.cpp offers very fast performance with the ggmls compared to using transformers, and sometimes faster than ExLlama.