Skip to main content
`Ctrl`+`K`

# GGUF Exporting

Currently, only support asymmetric int4 per_group weight-only quantization, and the group_size must be 32.The models supported include Llama2-7b, Llama2-13b, Llama2-70b, and Llama3-8b.

## Example of GGUF Exporting (for already quantized model)

    from quark.torch import export_gguf

    model_dir = "meta-llama/Llama-2-7b-chat-hf"
    export_gguf(quantized_model, output_dir="./output_dir", model_type="llama", tokenizer_path=model_dir)

After running the code above successfully, there will be a `.gguf` file under output_dir, `./output_dir/llama.gguf` for example.