Télécharger l’appObtenir l’appParcourir
On this page

How do I run open LLMs on a consumer GPU with GGUF quantization?

Choose a GGUF file supported by your app, then check its size and your available GPU memory. Quantization and file size alone do not establish quality, speed or fit at your chosen context length.

Updated .

Answer records

Check model files and sizes

License terms and download availability are separate. Read the exact upstream license before using a model.

Answer response assembled as of . This dates the returned rows, not an upstream model update or a successful model run. Check each linked file record for current availability.

Models or files
11
Citation links
6
Data links
3

  1. legraphista/glm-4-9b-chat-IMat-GGUF

    Model from Hugging Face for text generation: legraphista/glm-4-9b-chat-IMat-GGUF · gguf

    License
    unknown. No license stated License details
    Hosting
    Availability is not confirmed. Inspect the current file record before treating it as a hosted download. 30 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    Matches this answer's catalog filters. Check the model page for current files, license terms and availability.

  2. LiquidAI/LFM2.5-2.6B-GGUF

    Model from Hugging Face for text generation: LiquidAI/LFM2.5-2.6B-GGUF · gguf

    License
    unknown. No license stated License details
    Hosting
    Availability is not confirmed. Inspect the current file record before treating it as a hosted download. 15 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    Matches this answer's catalog filters. Check the model page for current files, license terms and availability.

  3. rippertnt/HyperCLOVAX-SEED-Text-Instruct-1.5B-Q4_K_M-GGUF

    Model from Hugging Face for llm: rippertnt/HyperCLOVAX-SEED-Text-Instruct-1.5B-Q4_K_M-GGUF

    License
    unknown. No license stated License details
    Hosting
    Availability is not confirmed. Inspect the current file record before treating it as a hosted download. 7 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    Matches this answer's catalog filters. Check the model page for current files, license terms and availability.

  4. ornith-ai/Ornith-1.0-35B-GGUF

    Model from Hugging Face for text generation: ornith-ai/Ornith-1.0-35B-GGUF · transformers

    License
    mit. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    Availability is not confirmed. Inspect the current file record before treating it as a hosted download. 9 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    Matches this answer's catalog filters. Check the model page for current files, license terms and availability.

  5. ornith-ai/Ornith-1.0-9B-GGUF

    Model from Hugging Face for text generation: ornith-ai/Ornith-1.0-9B-GGUF · transformers

    License
    mit. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    Availability is not confirmed. Inspect the current file record before treating it as a hosted download. 9 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    Matches this answer's catalog filters. Check the model page for current files, license terms and availability.

  6. prism-ml/Bonsai-27B-gguf

    Model from Hugging Face for text generation: prism-ml/Bonsai-27B-gguf · llama.cpp

    License
    apache-2.0. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    Availability is not confirmed. Inspect the current file record before treating it as a hosted download. 14 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    Matches this answer's catalog filters. Check the model page for current files, license terms and availability.

  7. ornith-ai/Ornith-1.5-397B-GGUF

    Model from Hugging Face for text generation: ornith-ai/Ornith-1.5-397B-GGUF · transformers

    License
    mit. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    Availability is not confirmed. Inspect the current file record before treating it as a hosted download. 9 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    Matches this answer's catalog filters. Check the model page for current files, license terms and availability.

  8. prism-ml/Ternary-Bonsai-27B-gguf

    Model from Hugging Face for text generation: prism-ml/Ternary-Bonsai-27B-gguf · llama.cpp

    License
    apache-2.0. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    Availability is not confirmed. Inspect the current file record before treating it as a hosted download. 16 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    Matches this answer's catalog filters. Check the model page for current files, license terms and availability.

  9. lmstudio-community/gemma-4-E4B-it-GGUF

    Model from Hugging Face for llm: lmstudio-community/gemma-4-E4B-it-GGUF

    License
    apache-2.0. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    Availability is not confirmed. Inspect the current file record before treating it as a hosted download. 6 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    Matches this answer's catalog filters. Check the model page for current files, license terms and availability.

  10. ggml-org/gemma-4-E4B-it-GGUF

    Model from Hugging Face for any to any: ggml-org/gemma-4-E4B-it-GGUF

    License
    apache-2.0. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    Availability is not confirmed. Inspect the current file record before treating it as a hosted download. 12 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    Matches this answer's catalog filters. Check the model page for current files, license terms and availability.

  11. CMSManhattan/JiRackUltra_1b

    Model from Hugging Face for text generation: CMSManhattan/JiRackUltra_1b

    License
    mit. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    Availability is not confirmed. Inspect the current file record before treating it as a hosted download. 32 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    Matches this answer's catalog filters. Check the model page for current files, license terms and availability.

Page generated 2026-11-13. Catalog records may be cached or incomplete.

Cite this page

https://huggingbay.xyz/answers/run-llms-on-consumer-gpu

How do I run open LLMs on a consumer GPU with GGUF quantization?. Hugging Bay. https://huggingbay.xyz/answers/run-llms-on-consumer-gpu

Share this page

Share: Reddit Hacker News 微博

机器可读引用包 (machine-readable citations): citations.json · 中文指南: llms-zh.txt

Quick answer: compatibility and sources

How do I run open LLMs on a consumer GPU with GGUF quantization?

It depends on your task, available memory, file format and license. The records below match this guide’s filters. Check the linked file evidence before choosing: a catalog match does not prove runtime support, safety or answer quality.

An answer pack is a question-specific filter over catalog records. It is not a benchmark, safety review or guarantee that a model will run.

As of .

The machine-readable answer record and its citation list link to the source data behind this guide. Catalog matches are not a benchmark or model test.

Updated .

Page sources

Page-copy source revision 8871bab84e1cf80aea1686a944ec335078a6c036 · Source files: apps/web/server.mjs, apps/web/lib/answers-copy.mjs, apps/web/lib/answers-landing-html.mjs, apps/web/lib/answer-pack-html.mjs, apps/web/src/App.tsx, apps/web/lib/aeo-page.mjs · Full revision record

Key facts

Answer pack
Filters catalog rows for one reader question; order is not a quality ranking.
Citation API
Machine-readable citations link each row to source endpoints.
Hosting line
Describes delivery status separately from license permission.
Unmeasured runs
No row implies a successful model test unless run evidence says so.

Evidence you can inspect (plain text)

  • GET /api/answers indexes public answer packs.
  • GET /api/answers/run-llms-on-consumer-gpu exposes machine-readable rows when the slug exists.
  • Open /artifact/hf-model-qwen-qwen3-8b/files for exact filenames, availability and any recorded SHA-256 for the model.
  • Open /evidence/artifacts/hf-model-qwen-qwen3-8b for run-readiness evidence tied to that record; absent receipts stay unmeasured.
  • Open /api/artifacts/hf-model-qwen-qwen3-8b/card.md for machine-readable source facts (not a runtime test).
  • Guides do not assert model tests unless linked run evidence includes a receipt.

Unmeasured states are stated explicitly; this page does not invent run scores or compatibility guarantees.

Which date applies to which fact

  • Answer API rows assembled:
  • Committed page-copy sources:
  • Upstream model metadata: Dates on artifact records when shown; not implied by page-copy revision
  • Run receipts: Only on /evidence pages when recorded; otherwise unmeasured

Page-copy revision (Updated line above) identifies committed text sources, not upstream model edits or test outcomes.

Nous utilisons un seul cookie facultatif pour compter les visites.

Confidentialité