Apologies for the basic question, but what’s the difference between GGML and GPTQ? Do these just refer to different compression methods? Which would you choose if you’re using a 3090ti GPU?

  • Mechanize
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    As far as I know they are different types of quantization.

    The main difference you have to keep in mind as an end user is that, currently, GPTQ needs the full model to load in VRAM (the memory of your GPU) while GGML can share layers between the system RAM and the VRAM.

    Performance wise I think it depends on the foundational model used, I know some time ago someone (The_Bloke?) did some testing, but I read it on Reddit and I don’t feel like going to search for it.
    There’s this interesting post on huggingface Link, but it’s pretty old and things could have changed (for example GGML has gone through different iterations).

    I’m just going by memory, so take everything I wrote with a pinch of salt. I never personally used GPTQ.

    • markon@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Also llama.cpp offers very fast performance with the ggmls compared to using transformers, and sometimes faster than ExLlama.