Skip to main content
`Ctrl`+`K`

# Hugging Face format (safetensors format)

Hugging Face format (safetensors format) is an optional exporting format for Quark, and the file list of this exporting format is the same as the file list of the original Hugging Face model, with quantization information added to these files. Taking the llama2-7b model as an example, the exported file list and added information are as below:

File name | Additional Quantization Information
--- | ---
config.json | Original configuration, with quantization configuration added in a `"quantization_config"` key
generation_config.json | -
model*.safetensors | Quantized checkpoint (weights, scaling factors, zero points)
model.safetensors.index.json | Mapping of weights names to safetensors files, in case the model weights are sharded into multiple files (optional)
special_tokens_map.json | -
tokenizer_config.json | -
tokenizer.json | -

## Exporting to Hugging Face format (safetensors format)

Here is an example of how to export to Hugging Face format (safetensors format) a Quark model using `export_safetensors()`:

    from quark.torch import ModelQuantizer, export_safetensors
    from quark.torch.quantization.config.config import Int8PerTensorSpec, QLayerConfig, QConfig

    from transformers import AutoModelForCausalLM

    quant_spec = Int8PerTensorSpec(
       observer_method="min_max",
       symmetric=True,
       scale_type="float",
       round_method="half_even",
       is_dynamic=False
    ).to_quantization_spec()

    global_quant_config = QLayerConfig(weight=quant_spec)
    quant_config = QConfig(global_quant_config=global_quant_config)

    model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m", torch_dtype="auto")

    quantizer = ModelQuantizer(quant_config)
    quantized_model = quantizer.quantize_model(model, dataloader=None)
    quantized_model = quantizer.freeze(quantized_model)

    export_safetensors(
       model=quantized_model,
       output_dir="./opt-125m-quantized"
    )

By default, `export_safetensors()` exports models with `model.save_pretrained()` using a Quark-specific format for the checkpoint and `"quantization_config"` key in the `config.json` file. This format may not directly be usable by some downstream libraries (AutoAWQ, vLLM).

Until downstream libraries support Quark quantized models, one may export models so that the weight checkpoint and `config.json` file targets a specific downstream libraries, using `custom_mode="awq"` or `custom_mode="fp8"`. Example:

    export_safetensors(
       model=quantized_model,
       output_dir="./opt-125m-quantized",
       custom_mode="awq"
    )

    export_safetensors(
       model=quantized_model,
       output_dir="./opt-125m-quantized",
       custom_mode="fp8"
    )

For example, `custom_mode="awq"` would e.g. use `qzeros` instead of `weight_zero_point`, `qweight` instead of `weight` in the checkpoint. Moreover, the `quantization_config` in the `config.json` file is custom, and the full Quark `quark.torch.quantization.config.config.Config` is not serialized.

In the `config.json`, such an export results in using `"quant_method": "awq"`, that can e.g. be loaded through AutoAWQ in Transformers library.

## Loading quantized models saved in Hugging Face format (safetensors format)

Quark provides the importing function for HF format export files. In other words, these files can be reloaded into Quark. After reloading, the weights of the quantized operators in the model are stored in the real_quantized format.

Currently, this importing function supports weight-only, static, and dynamic quantization for FP8, INT8/UINT8, FP4, INT4/UINT, AWQ, GPTQ and Qronos.

Here is an example of how to load a serialized quantized model from a folder containing the model (as `*.safetensors`) and its artifacts (`config.json`, etc.), using `import_model_from_safetensors()`:

    from quark.torch import import_model_from_safetensors
    from transformers import AutoConfig, AutoModelForCausalLM
    import torch

    # We only need the backbone/architecture of the original model,
    # not its weights, as weights are loaded from the quantized checkpoint.
    config = AutoConfig.from_pretrained("facebook/opt-125m")
    with torch.device("meta"):
       original_model = AutoModelForCausalLM.from_config(config)

    quantized_model = import_model_from_safetensors(original_model, model_dir="./opt-125m-quantized")