Skip to main content
`Ctrl`+`K`

# Quantization configuration

Quark Quantization Config API for PyTorch

class quark.torch.quantization.config.config.QConfig(global_quant_config: BaseQLayerConfig, exclude: list[str] = [], algo_config: list[BaseAlgoConfig] | None = None, log_severity_level: int | None = 1, version: str | None = '0.11', layer_type_quant_config: dict[type[Module], QLayerConfig] = {}, layer_quant_config: dict[str, QLayerConfig] = {}, kv_cache_quant_config: dict[str, QLayerConfig] = {}, kv_cache_group: list[str] = [], min_kv_scale: float = 0.0, kv_cache_post_rope: bool = False, softmax_quant_spec: QTensorConfig | None = None, quant_mode: QuantizationMode = QuantizationMode.eager_mode)

  A class that encapsulates comprehensive quantization configurations for a machine learning model, allowing for detailed and hierarchical control over quantization parameters across different model components.

  Parameters:

    - global_quant_config (QLayerConfig) – Global quantization configuration applied to the entire model unless overridden at the layer level.

    - layer_type_quant_config (Dict[torch.nn.Module, QLayerConfig]) – A dictionary mapping from layer types (e.g., nn.Conv2d, nn.Linear) to their quantization configurations.

    - layer_quant_config (Dict[str, QLayerConfig]) – A dictionary mapping from layer names to their quantization configurations, allowing for per-layer customization. Default is `{}`.

    - kv_cache_quant_config (Dict[str, QLayerConfig]) – A dictionary mapping from layer names to kv_cache quantization configurations. Default is `{}`.

    - softmax_quant_spec (Optional[QTensorConfig]) – A quantization specifications of nn.functional.softmax output. Default is `None`.

    - exclude (List[str]) – A list of layer names to be excluded from quantization, enabling selective quantization of the model. Default is `[]`.

    - algo_config (Optional[AlgoConfig]) – Optional configuration for the quantization algorithm, such as GPTQ, AWQ and Qronos. After this process, the datatype/fake_datatype of weights will be changed with quantization scales. Default is `None`.

    - quant_mode (QuantizationMode) – The quantization mode to be used (`eager_mode` or `fx_graph_mode`). Default is `QuantizationMode.eager_mode`.

class quark.torch.quantization.config.config.QLayerConfig(input_tensors: BaseQTensorConfig | list[BaseQTensorConfig] | None = None, output_tensors: BaseQTensorConfig | list[BaseQTensorConfig] | None = None, weight: BaseQTensorConfig | list[BaseQTensorConfig] | None = None, bias: BaseQTensorConfig | list[BaseQTensorConfig] | None = None, target_device: DeviceType | None = None)

  A data class that specifies quantization configurations for different components of a module, allowing hierarchical control over how each tensor type is quantized.

  Parameters:

    - input_tensors (Optional[Union[QTensorConfig, List[QTensorConfig]]]) – Input tensors quantization specification. If None, following the hierarchical quantization setup. e.g. If the `input_tensors` in `layer_type_quant_config` is `None`, the configuration from `global_quant_config` will be used instead. Defaults to `None`. If None in `global_quant_config`, `input_tensors` are not quantized.

    - output_tensors (Optional[Union[QTensorConfig, List[QTensorConfig]]]) – Output tensors quantization specification. Defaults to `None`. If `None`, the same as above.

    - weight (Optional[Union[QTensorConfig, List[QTensorConfig]]]) – The weights tensors quantization specification. Defaults to `None`. If `None`, the same as above.

    - bias (Optional[Union[QTensorConfig, List[QTensorConfig]]]) – The bias tensors quantization specification. Defaults to `None`. If `None`, the same as above.

    - target_device (Optional[DeviceType]) – Configuration specifying the target device (e.g., CPU, GPU, IPU) for the quantized model.

  to_dict() → dict[str, Any]

    Serialize layer configuration to dictionary.

  classmethod from_dict(quantization_config: dict[str, Any]) → QLT

    Deserialize layer configuration from dictionary.

class quark.torch.quantization.config.config.TwoStageSpec(first_stage: DataTypeSpec | QTensorConfig, second_stage: DataTypeSpec | QTensorConfig)

  A data class that specifies two-stage quantization configurations for different components of a module, allowing hierarchical control over how each tensor type is quantized.

class quark.torch.quantization.config.config.ProgressiveSpec(first_stage: DataTypeSpec | QTensorConfig, second_stage: DataTypeSpec | QTensorConfig)

  A data class that specifies a progressive quantization specification for a tensor. The first stage quantizes the input tensor, while the second stage quantizes the output from the first stage.

  For example, to progressively quantize a float16 tensor:

  1. First quantize it to fp8_e4m3 using fp8_e4m3 per-tensor quantization, get a fp8_e4m3 tensor.

  2. Then quantize the fp8_e4m3 tensor to int4 using int4 per-channel quantization, get a int4 tensor.

  The configuration for this example would be:

      quant_spec = ProgressiveSpec(
          first_stage=FP8E4M3PerTensorSpec(observer_method="min_max",
                                           is_dynamic=False),
          second_stage=Int4PerChannelSpec(symmetric=False,
                                          scale_type="float",
                                          round_method="half_even",
                                          ch_axis=0,
                                          is_dynamic=False)
      ).to_quantization_spec()

class quark.torch.quantization.config.config.ScaleQuantSpec(first_stage: DataTypeSpec | QTensorConfig, second_stage: DataTypeSpec | QTensorConfig)

  A data class that specifies a two-stage quantization process for scale quantization.

  The quantization happens in two stages:

  1. First stage quantizes the input tensor itself.

  2. Second stage quantizes the scale values from the first stage quantization.

  For example, given a float16 tensor:

  1. First quantize the tensor to fp4_e2m1 using fp4_e2m1 per-group quantization, producing a fp4_e2m1 tensor with float16 scale values.

  2. Then quantize those float16 scale values to fp8_e4m3 using fp8_e4m3 per-tensor quantization.

  The configuration for this example would be:

      quant_spec = ScaleQuantSpec(
          first_stage=FP4PerGroupSpec(group_size=16, is_dynamic=False),
          second_stage=FP8E4M3PerTensorSpec(observer_method="min_max", is_dynamic=False)
      ).to_quantization_spec()

class quark.torch.quantization.config.config.DataTypeSpec

class quark.torch.quantization.config.config.Uint4PerTensorSpec(observer_method: str | None = 'min_max', symmetric: bool | None = False, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using uint4 per tensor quantization.

  Example:

      quantization_spec = Uint4PerTensorSpec(is_dynamic=True, symmetric=False).to_quantization_spec()

class quark.torch.quantization.config.config.Uint4PerChannelSpec(ch_axis: int, symmetric: bool | None = False, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = None, zero_point_type: str | None = 'int32')

  Helper class to define a `QTensorConfig` using uint4 per channel quantization.

  Example:

      quantization_spec = Uint4PerChannelSpec(
          symmetric=True,
          scale_type="float",
          round_method="half_even",
          ch_axis=0,
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Uint4PerGroupSpec(ch_axis: int, group_size: int, symmetric: bool = False, is_dynamic: bool | None = False, scale_type: str | None = 'float', round_method: str | None = 'half_even')

  Helper class to define a `QTensorConfig` using uint4 per group quantization.

  Example:

      quantization_spec = Uint4PerGroupSpec(
          symmetric=False,
          scale_type="float",
          round_method="half_even",
          ch_axis=1,
          is_dynamic=False,
          group_size=128
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Int3PerGroupSpec(ch_axis: int, group_size: int, symmetric: bool | None = True, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using int3 per group quantization.

  Example:

      quantization_spec = Int3PerGroupSpec(
          symmetric=True,
          scale_type="float",
          round_method="half_even",
          is_dynamic=False,
          group_size=32,
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Int3PerChannelSpec(ch_axis: int, symmetric: bool | None = False, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using int3 per channel quantization.

  Example:

      quantization_spec = Int3PerChannelSpec(
          symmetric=False,
          scale_type="float",
          round_method="half_even",
          ch_axis=0,
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Int2PerGroupSpec(ch_axis: int, group_size: int, symmetric: bool | None = True, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using int2 per group quantization.

  Example:

      quantization_spec = Int2PerGroupSpec(
          symmetric=True,
          scale_type="float",
          round_method="half_even",
          is_dynamic=False,
          group_size=32,
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Int4PerTensorSpec(observer_method: str | None = 'min_max', symmetric: bool | None = True, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using int4 per tensor quantization.

  Example:

      quantization_spec = Int4PerTensorSpec(
          observer_method="min_max",
          symmetric=True,
          scale_type="float",
          round_method="half_even",
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Int4PerChannelSpec(ch_axis: int, symmetric: bool | None = True, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using int4 per channel quantization.

  Example:

      quantization_spec = Int4PerChannelSpec(
          symmetric=False,
          scale_type="float",
          round_method="half_even",
          ch_axis=0,
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Int4PerGroupSpec(ch_axis: int, group_size: int, symmetric: bool = True, is_dynamic: bool | None = False, scale_type: str | None = 'float', round_method: str | None = 'half_even')

  Helper class to define a `QTensorConfig` using int4 per group quantization.

  Example:

      quantization_spec = Int4PerGroupSpec(
          symmetric=True,
          scale_type="float",
          round_method="half_even",
          ch_axis=1,
          is_dynamic=False,
          group_size=128
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Uint8PerTensorSpec(observer_method: str | None = 'min_max', symmetric: bool | None = False, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using uint8 per tensor quantization.

  Example:

      quantization_spec = Uint8PerTensorSpec(
          observer_method="percentile",
          symmetric=True,
          scale_type="float",
          round_method="half_even",
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Uint8PerChannelSpec(ch_axis: int, symmetric: bool | None = True, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using uint8 per channel quantization.

  Example:

      quantization_spec = Uint8PerChannelSpec(
          symmetric=True,
          scale_type="float",
          round_method="half_even",
          ch_axis=0,
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Uint8PerGroupSpec(ch_axis: int, group_size: int, symmetric: bool | None = False, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using uint8 per group quantization.

  Example:

      quantization_spec = Uint8PerGroupSpec(
          symmetric=False,
          scale_type="float",
          round_method="half_even",
          ch_axis=1,
          is_dynamic=False,
          group_size=128
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Int8PerTensorSpec(observer_method: str | None = 'min_max', symmetric: bool | None = True, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using int8 per tensor quantization.

  Example:

      quantization_spec = Int8PerTensorSpec(
          observer_method="min_max",
          symmetric=True,
          scale_type="float",
          round_method="half_even",
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Int8PerChannelSpec(ch_axis: int, symmetric: bool | None = False, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using int8 per channel quantization.

  Example:

      quantization_spec = Int8PerChannelSpec(
          symmetric=False,
          scale_type="float",
          round_method="half_even",
          ch_axis=0,
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.Int8PerGroupSpec(ch_axis: int, group_size: int, symmetric: bool | None = True, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using int8 per group quantization.

  Example:

      quantization_spec = Int8PerGroupSpec(
          symmetric=True,
          scale_type="float",
          round_method="half_even",
          ch_axis=1,
          is_dynamic=False,
          group_size=128
      ).to_quantization_spec()

class quark.torch.quantization.config.config.FP8E4M3PerTensorSpec(observer_method: str | None = 'min_max', scale_type: str | None = None, is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using FP8E4M3 per tensor quantization.

  Example:

      quantization_spec = FP8E4M3PerTensorSpec(
          observer_method="min_max",
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.FP8E4M3PerChannelSpec(ch_axis: int, symmetric: bool | None = True, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using FP8E4M3 per channel quantization.

  Example:

      quantization_spec = FP8E4M3PerChannelSpec(is_dynamic=False, ch_axis=0).to_quantization_spec()

class quark.torch.quantization.config.config.FP8E4M3PerGroupSpec(ch_axis: int, group_size: int, scale_format: str | None = 'float32', scale_calculation_mode: str | None = None, is_dynamic: bool | None = True)

  Helper class to define a `QTensorConfig` using FP8E4M3 per group quantization.

  Example:

      quantization_spec = FP8E4M3PerGroupSpec(
          ch_axis=-1,
          group_size=group_size,
          is_dynamic=True
      ).to_quantization_spec()

class quark.torch.quantization.config.config.FP8E5M2PerTensorSpec(observer_method: str | None = 'min_max', symmetric: bool | None = True, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using FP8E5M2 per tensor quantization.

  Example:

      quantization_spec = FP8E5M2PerTensorSpec(
          observer_method="min_max",
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.FP8E5M2PerChannelSpec(ch_axis: int, symmetric: bool | None = True, scale_type: str | None = 'float', round_method: str | None = 'half_even', is_dynamic: bool | None = False)

  Helper class to define a `QTensorConfig` using FP8E5M2 per channel quantization.

  Example:

      quantization_spec = FP8E5M2PerChannelSpec(is_dynamic=False, ch_axis=0).to_quantization_spec()

class quark.torch.quantization.config.config.FP8E5M2PerGroupSpec(ch_axis: int, group_size: int, scale_format: str | None = 'float32', scale_calculation_mode: str | None = None, is_dynamic: bool | None = True)

  Helper class to define a `QTensorConfig` using FP8E5M2 per group quantization.

  Example:

      quantization_spec = FP8E5M2PerGroupSpec(
          ch_axis=-1,
          group_size=group_size,
          is_dynamic=True
      ).to_quantization_spec()

class quark.torch.quantization.config.config.FP4PerGroupSpec(ch_axis: int, group_size: int, scale_format: str | None = 'float32', scale_calculation_mode: str | None = None, is_dynamic: bool | None = True)

  Helper class to define a `QTensorConfig` using FP4 per group quantization.

  Example:

      quantization_spec = FP4PerGroupSpec(
          ch_axis=-1,
          group_size=group_size,
          is_dynamic=True
      ).to_quantization_spec()

class quark.torch.quantization.config.config.FP6E2M3PerGroupSpec(ch_axis: 'int', group_size: 'int', scale_format: 'str | None' = 'float32', scale_calculation_mode: 'str | None' = None, is_dynamic: 'bool | None' = True)

class quark.torch.quantization.config.config.FP6E3M2PerGroupSpec(ch_axis: 'int', group_size: 'int', scale_format: 'str | None' = 'float32', scale_calculation_mode: 'str | None' = None, is_dynamic: 'bool | None' = True)

class quark.torch.quantization.config.config.Float16Spec

  Helper class to define a `QTensorConfig` using float16 data type. The resulting QTensorConfig does not quantize the tensor..

  Example:

      quantization_spec = Float16Spec().to_quantization_spec()

class quark.torch.quantization.config.config.Bfloat16Spec

  Helper class to define a `QTensorConfig` using bfloat16 data type. The resulting QTensorConfig does not quantize the tensor..

  Example:

      quantization_spec = Bfloat16Spec().to_quantization_spec()

class quark.torch.quantization.config.config.OCP_MXSpec

class quark.torch.quantization.config.config.OCP_MXFP8E4M3Spec(ch_axis: int, is_dynamic: bool = True, scale_calculation_mode: str = 'even')

  Helper class to define a `QTensorConfig` using MX OCP data type using FP8E4M3.

  Example:

      quantization_spec = OCP_MXFP8E4M3Spec(
          ch_axis=-1,
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.OCP_MXFP8E5M2Spec(ch_axis: int, is_dynamic: bool = True, scale_calculation_mode: str = 'even')

  Helper class to define a `QTensorConfig` using MX OCP data type using FP8E5M2.

  Example:

      quantization_spec = OCP_MXFP8E5M2Spec(
          ch_axis=-1,
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.OCP_MXFP6E3M2Spec(ch_axis: int, is_dynamic: bool = True, scale_calculation_mode: str = 'even')

  Helper class to define a `QTensorConfig` using MX OCP data type using FP6E3M2.

  Example:

      quantization_spec = OCP_MXFP6E3M2Spec(
          ch_axis=-1,
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.OCP_MXFP6E2M3Spec(ch_axis: int, is_dynamic: bool = True, scale_calculation_mode: str = 'even')

  Helper class to define a `QTensorConfig` using MX OCP data type using FP6E2M3.

  Example:

      quantization_spec = OCP_MXFP6E2M3Spec(
          ch_axis=-1,
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.OCP_MXFP4Spec(ch_axis: int, is_dynamic: bool = True, scale_calculation_mode: str = 'even')

  Helper class to define a `QTensorConfig` using MX OCP data type using FP4.

  Example:

      quantization_spec = OCP_MXFP4Spec(
          ch_axis=-1,
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.OCP_MXINT8Spec(ch_axis: int, is_dynamic: bool = True, scale_calculation_mode: str = 'even')

  Helper class to define a `QTensorConfig` using MX OCP data type using INT8.

  Example:

      quantization_spec = OCP_MXINT8Spec(
          ch_axis=-1,
          is_dynamic=False
      ).to_quantization_spec()

class quark.torch.quantization.config.config.OCP_MXFP4DiffsSpec(ch_axis: 'int', is_dynamic: 'bool | None' = False, scale_calculation_mode: 'str | None' = 'even')

class quark.torch.quantization.config.config.MX6Spec(ch_axis: int, block_size: int, scale_calculation_mode: str | None = 'even')

  Helper class to define a `QTensorConfig` using MX6 data type as defined in https://arxiv.org/pdf/2302.08007. More details are available in the Two Level Quantization Formats documentation.

  Example:

      quantization_spec = MX6Spec(is_dynamic=False).to_quantization_spec()

class quark.torch.quantization.config.config.MX9Spec(ch_axis: int, block_size: int, scale_calculation_mode: str | None = 'even')

  Helper class to define a `QTensorConfig` using MX9 data type as defined in https://arxiv.org/pdf/2302.08007. More details are available in the Two Level Quantization Formats documentation.

  Example:

      quantization_spec = MX9Spec(is_dynamic=False).to_quantization_spec()

class quark.torch.quantization.config.config.BFP16Spec(ch_axis: int, scale_calculation_mode: str | None = 'even')

  Helper class to define a `QTensorConfig` using bfp16 data type.

  Example:

      quantization_spec = BFP16Spec(is_dynamic=False).to_quantization_spec()

class quark.torch.quantization.config.config.QTensorConfig(dtype: Dtype, observer_cls: type[ObserverBase] | None = None, is_dynamic: bool | None = None, qscheme: QSchemeType | None = None, ch_axis: int | None = None, group_size: int | None = None, symmetric: bool | None = None, round_method: BaseRoundType | None = None, scale_type: ScaleType | None = None, scale_format: str | None = None, scale_calculation_mode: str | None = None, qat_spec: Any | None = None, mx_element_dtype: Dtype | None = None, zero_point_type: ZeroPointType | None = ZeroPointType.int32, is_scale_quant: bool = False)

  A data class that defines the specifications for quantizing tensors within a model.

  Parameters:

    - dtype (Dtype) – The data type for quantization (e.g., int8, int4).

    - is_dynamic (Optional[bool]) – Specifies whether dynamic or static quantization should be used. Default is `None`, which indicates no specification.

    - observer_cls (Optional[Type[ObserverBase]]) – The class of observer to be used for determining quantization parameters like min/max values. Default is `None`.

    - qscheme (Optional[QSchemeType]) – The quantization scheme to use, such as per_tensor, per_channel or per_group. Default is `None`.

    - ch_axis (Optional[int]) – The channel axis for per-channel quantization. Default is `None`.

    - group_size (Optional[int]) – The size of the group for per-group quantization, also the block size for MX datatypes. Default is `None`.

    - symmetric (Optional[bool]) – Indicates if the quantization should be symmetric around zero. If True, quantization is symmetric. If `None`, it defers to a higher-level or global setting. Default is `None`.

    - round_method (Optional[RoundType]) – The rounding method during quantization, such as half_even. If None, it defers to a higher-level or default method. Default is `None`.

    - scale_type (Optional[ScaleType]) – Defines the scale type to be used for quantization, like power of two or float. If `None`, it defers to a higher-level setting or uses a default method. Default is `None`.

    - mx_element_dtype (Optional[Dtype]) – Defines the data type to be used for the element type when using mx datatypes, the shared scale effectively uses FP8 E8M0.

    - is_scale_quant (Optional[bool]) – Indicates whether this spec is for quantizing scales rather than tensors. Default is `False`.

  Example:

      from quark.torch.quantization.config.type import Dtype, ScaleType, RoundType, QSchemeType
      from quark.torch.quantization.config.config import QTensorConfig
      from quark.torch.quantization.observer.observer import PerChannelMinMaxObserver

      quantization_spec = QTensorConfig(
          dtype=Dtype.int8,
          qscheme=QSchemeType.per_channel,
          observer_cls=PerChannelMinMaxObserver,
          symmetric=True,
          scale_type=ScaleType.float,
          round_method=RoundType.half_even,
          is_dynamic=False,
          ch_axis=1,
      )

  to_dict() → dict[str, Any]

    Serialize tensor configuration to dictionary.

  classmethod from_dict(config_dict: dict[str, Any]) → QTT

    Deserialize tensor configuration from dictionary.

class quark.torch.quantization.config.config.QATSpec

class quark.torch.quantization.config.config.TQTSpec(threshold_init_meth: TQTThresholdInitMeth | None = None)

  Configuration for the Trained Quantization Thresholds (TQT) post-training quantization method, implementing https://arxiv.org/abs/1903.08066.

quark.torch.quantization.config.config.load_pre_optimization_config_from_file(file_path: str) → PreQuantOptConfig

  Load pre-optimization configuration from a JSON file.

  Parameters:

    file_path (str) – The path to the JSON file containing the pre-optimization configuration.

  Returns:

    The pre-optimization configuration.

  Return type:

    PreQuantOptConfig

quark.torch.quantization.config.config.load_quant_algo_config_from_file(file_path: str) → AlgoConfig

  Load quantization algorithm configuration from a JSON file.

  Parameters:

    file_path (str) – The path to the JSON file containing the quantization algorithm configuration.

  Returns:

    The quantization algorithm configuration.

  Return type:

    AlgoConfig

class quark.torch.quantization.config.config.PreQuantOptConfig

class quark.torch.quantization.config.config.AlgoConfig

class quark.torch.quantization.config.config.SmoothQuantConfig(name: str = 'smooth', alpha: float = 1, scale_clamp_min: float = 0.001, scaling_layers: list[dict[str, Any]] = [], model_decoder_layers: str = '')

  A data class that defines the specifications for Smooth Quantization.

  Parameters:

    - name (str) – The name of the configuration, typically used to identify different quantization settings. Default is `"smooth"`.

    - alpha (int) – The factor of adjustment in the quantization formula, influencing how aggressively weights are quantized. Default is `1`.

    - scale_clamp_min (float) – The minimum scaling factor to be used during quantization, preventing the scale from becoming too small. Default is `1e-3`.

    - scaling_layers (List[Dict[str, Any]]) – Specific settings for scaling layers, allowing customization of quantization parameters for different layers within the model. Default is `None`.

    - model_decoder_layers (str) – Specifies any particular decoder layers in the model that might have unique quantization requirements. Default is `None`.

  The parameter `scaling_layers` can be left to an empty list (default), in which case they will be automatically detected.

  Example:

      from quark.torch.quantization.config.config import SmoothQuantConfig

      scaling_layers=[
          {
              "prev_op": "input_layernorm",
              "layers": ["self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj"],
              "inp": "self_attn.q_proj",
              "module2inspect": "self_attn"
          },
          {
              "prev_op": "post_attention_layernorm",
              "layers": ["mlp.gate_proj", "mlp.up_proj"],
              "inp": "mlp.gate_proj",
              "module2inspect": "mlp"
          }
      ]

      smoothquant_config = SmoothQuantConfig(
          scaling_layers=scaling_layers,
          model_decoder_layers="model.layers"
      )

class quark.torch.quantization.config.config.RotationConfig(scaling_layers: dict[str, list[dict[str, Any]]], name: str = 'rotation', r1: bool = True, r2: bool = False, r3: bool = False, r4: bool = False, rotation_size: int | None = None, random: bool | None = None, random_r1: bool = False, random_r2: bool = False, backbone: str = 'model', model_decoder_layers: str = 'model.layers', v_proj: str = 'self_attn.v_proj', o_proj: str = 'self_attn.o_proj', self_attn: str = 'self_attn', mlp: str = 'mlp', online_r1_rotation: bool | None = None, online_config: OnlineRotationConfig | None = None, trainable: bool = False, train_smooth: bool | None = None, smooth_positions: list[str] | None = None)

  A data class that defines the specifications for the rotation algorithms.

  Parameters:

    - name (str) – The name of the configuration, typically used to identify different rotation settings. Default is `"rotation"`.

    - r1 (bool) – Whether to apply `R1` rotation. See SpinQuant paper for details. Defaults to `True`.

    - r2 (bool) – Whether to apply `R2` rotation. See SpinQuant paper for details. Defaults to `True`.

    - r3 (bool) – Whether to apply `R3` rotation. It is only useful when using KV cache quantization. See SpinQuant paper for details. Defaults to `True`.

    - r4 (bool) – Whether to apply `R4` rotation. See SpinQuant paper for details. Defaults to `True`.

    - rotation_size (Optional[int]) – The size of rotations to apply on activations/weights. By default, the activation last dimension (e.g. `hidden_size`), or weight input/output channel dimension is used as rotation size. In case the parameter `rotation_size` is specified, smaller rotations of size `(rotation_size, rotation_size)` are applied per-block. Defaults to `None`.

    - random (Optional[bool]) – Deprecated. Use `random_r1` and `random_r2` instead. Defaults to `None`.

    - random_r1 (bool) – A boolean flag indicating whether `R1` should be a random Hadamard matrix. See SpinQuant paper for details. This can be useful for data augmentation purposes where random rotations may be required. Default is `False`.

    - random_r2 (bool) – A boolean flag indicating whether `R2` should be a random Hadamard matrix. See SpinQuant paper for details. This can be useful for data augmentation purposes where random rotations may be required. Default is `False`. `random_r1` and `random_r2` are only relevant if we are using Hadamard rotations for `R1` and `R2`.

    - scaling_layers (List[Dict[str, str]]) –

      Specific settings for scaling layers, specifying the layer names where `R1` rotations must be applied if chosen, or where smoothing scales must be trained, if `train_smooth=True`. It is a dictionary with keys `"first_layer"`, `"middle_layers"` and `"last_layer"`, which are dictionaries specifying:

      - `"prev_modules"`: The list of previous modules the activation rotation may be fused into. This is optional/unused for online R1.

      - `"norm_module"`: The list of normalization layer that is in between the modules in `"prev_modules"` and `"next_modules"`. The normalization weight will typically need to be merged first into the modules of `"next_modules"` in order to permute the activation rotation with the normalization, and fuse the activation rotation in the modules of `"prev_modules"`. This is optional/unused for online R1.

      - `"next_modules"`: The list of modules to fuse inverse rotation into, on the input features dimension. This corresponds to weight rotation on the input features dimension. This is optional/unused for online R1.

      - `"target_modules"`: If specified, the list of modules to apply online rotation on, this may be only on a subset of `"next_modules"`. This is for example useful for MOE models, in case the experts gate/router is not quantized (skipping online rotation for it). Useful only for online ``R1`` rotations, optional. If not provided, `"next_modules"` is used instead as the list of modules to apply rotation on.

    - backbone (str) – A string indicating the path to the model backbone.

    - model_decoder_layers (str) – A string indicating the path to the list of decoder layers.

    - v_proj (str) – A string indicating the path to the v projection layer, starting from the decoder layer it is in.

    - o_proj (str) – A string indicating the path to the o projection layer, starting from the decoder layer it is in.

    - self_attn (str) – A string indicating the path to the self attention block, starting from the decoder layer it is in.

    - mlp (str) – A string indicating the path to the multilayer perceptron layer, starting from the decoder layer it is in.

    - online_r1_rotation (Optional[bool]) – Whether the activation rotation `R1` should be kept online instead of being fused into the preceding layer weights. In case `online_r1_rotation=False`, a single `R1` rotation is shared across all layers, whereas specialized rotations can be used per linear layer in case the activation rotation is done online. Defaults to `False`, i.e. `R1` is fused offline. If R1 is not used, defaults to `None`.

    - trainable (bool) – Whether to insert trainable rotations.

    - online_config (Optional[OnlineRotationConfig]) – Configuration specific to online rotations. Refer to `OnlineRotationConfig`. Defaults to `OnlineRotationConfig(shared_parallel=False)` for online trainable `R1`, otherwise defaults to `None` (no effect).

    - train_smooth (Optional[bool]) – Whether to train SmoothQuant scales. The trainable transform is then `T = D @ R` (or `T = R @ D`) with the rotation `R` and the SmoothQuant scales `D` (seen as a 1D vector, or a diagonal matrix). This parameter is similar to the approach in OSTQuant paper. Defaults to `None`. In case `trainable=True`, defaults to `False`.

    - smooth_positions (Optional[List[str]]) – A list of rotation positions to train smoothing scales for. Acceptable list values are `"r1"`, `"r2"`, `"r4"`. Defaults to `None`. In case `trainable=True`, defaults to `[]`.

  Example for llama model, offline rotations:

      from quark.torch.quantization.config.config import RotationConfig

      scaling_layers = {
          "first_layer": [
              {
                  "prev_modules": ["model.embed_tokens"],
                  "norm_module": "model.layers.layer_id.input_layernorm",
                  "next_modules": [
                      "model.layers.layer_id.self_attn.q_proj",
                      "model.layers.layer_id.self_attn.k_proj",
                      "model.layers.layer_id.self_attn.v_proj",
                  ],
              },
              {
                  "prev_modules": ["model.layers.layer_id.self_attn.o_proj"],
                  "norm_module": "model.layers.layer_id.post_attention_layernorm",
                  "next_modules": ["model.layers.layer_id.mlp.up_proj", "model.layers.layer_id.mlp.gate_proj"],
              },
          ],
          "middle_layers": [
              {
                  "prev_modules": ["model.layers.pre_layer_id.mlp.down_proj"],
                  "norm_module": "model.layers.layer_id.input_layernorm",
                  "next_modules": [
                      "model.layers.layer_id.self_attn.q_proj",
                      "model.layers.layer_id.self_attn.k_proj",
                      "model.layers.layer_id.self_attn.v_proj",
                  ],
              },
              {
                  "prev_modules": ["model.layers.layer_id.self_attn.o_proj"],
                  "norm_module": "model.layers.layer_id.post_attention_layernorm",
                  "next_modules": ["model.layers.layer_id.mlp.up_proj", "model.layers.layer_id.mlp.gate_proj"],
              },
          ],
          "last_layer": [
              {
                  "prev_modules": ["model.layers.layer_id.mlp.down_proj"],
                  "norm_module": "model.norm",
                  "next_modules": ["lm_head"],
              }
          ],
      }

      rotation_config = RotationConfig(
          model_decoder_layers="model.layers",
          v_proj="self_attn.v_proj",
          o_proj="self_attn.o_proj",
          self_attn="self_attn",
          mlp="mlp",
          scaling_layers=scaling_layers,
          r1=True,
          r2=True,
      )

  Example for llama model, online rotations:

      from quark.torch.quantization.config.config import RotationConfig

      scaling_layers = {
          "first_layer": [
              {
                  "target_modules": [
                      "model.layers.layer_id.self_attn.q_proj",
                      "model.layers.layer_id.self_attn.k_proj",
                      "model.layers.layer_id.self_attn.v_proj",
                  ]
              },
              {
                  "target_modules": ["model.layers.layer_id.mlp.up_proj", "model.layers.layer_id.mlp.gate_proj"]
              },
          ],
          "middle_layers": [
              {
                  "target_modules": [
                      "model.layers.layer_id.self_attn.q_proj",
                      "model.layers.layer_id.self_attn.k_proj",
                      "model.layers.layer_id.self_attn.v_proj",
                  ]
              },
              {
                  "target_modules": ["model.layers.layer_id.mlp.up_proj", "model.layers.layer_id.mlp.gate_proj"]
              },
          ],
          "last_layer": [
              {
                  "target_modules": [],
              }
          ],
      }

      rotation_config = RotationConfig(
          model_decoder_layers="model.layers",
          v_proj="self_attn.v_proj",
          o_proj="self_attn.o_proj",
          self_attn="self_attn",
          mlp="mlp",
          scaling_layers=scaling_layers,
          r1=True,
          r2=True,
          online_r1_rotation=True,
          rotation_size=32,
      )

class quark.torch.quantization.config.config.OnlineRotationConfig(shared_parallel: 'bool')

class quark.torch.quantization.config.config.AutoSmoothQuantConfig(name: str = 'autosmoothquant', scaling_layers: list[dict[str, Any]] | None = None, model_decoder_layers: str | None = None, compute_scale_loss: str | None = 'MSE')

  A data class that defines the specifications for AutoSmoothQuant.

  Parameters:

    - name (str) – The name of the quantization configuration. Default is `"autosmoothquant"`.

    - scaling_layers (List[Dict[str, str]]) – Configuration details for scaling layers within the model, specifying custom scaling parameters per layer. Default is `None`.

    - compute_scale_loss (str) – Calculate the best scale loss, “MSE” or “MAE”. Default is `"MSE"`.

    - model_decoder_layers (str) – Specifies the layers involved in model decoding that may require different quantization parameters. Default is `None`.

  Example:

      from quark.torch.quantization.config.config import AutoSmoothQuantConfig

      scaling_layers = [
          {
              "prev_op": "input_layernorm",
              "layers": ["self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj"],
              "inp": "self_attn.q_proj",
              "module2inspect": "self_attn"
          },
          {
              "prev_op": "self_attn.v_proj",
              "layers": ["self_attn.o_proj"],
              "inp": "self_attn.o_proj"
          },
          {
              "prev_op": "post_attention_layernorm",
              "layers": ["mlp.gate_proj", "mlp.up_proj"],
              "inp": "mlp.gate_proj",
              "module2inspect": "mlp"
          },
          {
              "prev_op": "mlp.up_proj",
              "layers": ["mlp.down_proj"],
              "inp": "mlp.down_proj"
          }
      ]

      autosmoothquant_config = AutoSmoothQuantConfig(
          model_decoder_layers="model.layers",
          scaling_layers=scaling_layers
      )

class quark.torch.quantization.config.config.AWQConfig(name: str = 'awq', scaling_layers: list[dict[str, Any]] = [], model_decoder_layers: str = '')

  Configuration for Activation-aware Weight Quantization (AWQ).

  Parameters:

    - name (str) – The name of the quantization configuration. Default is `"awq"`.

    - scaling_layers (List[Dict[str, Any]]) – Configuration details for scaling layers within the model, specifying custom scaling parameters per layer. Default is `None`.

    - model_decoder_layers (str) – Specifies the layers involved in model decoding that may require different quantization parameters. Default is `None`.

  The parameter `scaling_layers` can be left to an empty list (default), in which case they will be automatically detected.

  Example:

      from quark.torch.quantization.config.config import AWQConfig

      scaling_layers = [
          {
              "prev_op": "input_layernorm",
              "layers": ["self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj"],
              "inp": "self_attn.q_proj",
              "module2inspect": "self_attn"
          },
          {
              "prev_op": "post_attention_layernorm",
              "layers": ["mlp.gate_proj", "mlp.up_proj"],
              "inp": "mlp.gate_proj",
              "module2inspect": "mlp"
          },
      ]

      awq_config = AWQConfig(
          model_decoder_layers="model.layers",
          scaling_layers=scaling_layers
      )

class quark.torch.quantization.config.config.GPTQConfig(name: str = 'gptq', block_size: int = 128, damp_percent: float = 0.01, desc_act: bool = True, static_groups: bool = True, inside_layer_modules: list[str] = [], model_decoder_layers: str = '')

  A data class that defines the specifications for Accurate Post-Training Quantization for Generative Pre-trained Transformers (GPTQ).

  Parameters:

    - name (str) – The configuration name. Default is `"gptq"`.

    - block_size (int) – GPTQ divides the columns into blocks of size block_size and quantizes each block separately. Default is `128`.

    - damp_percent (float) – The percentage used to dampen the quantization effect, aiding in the maintenance of accuracy post-quantization. Default is `0.01`.

    - desc_act (bool) – Indicates whether descending activation is used, typically to enhance model performance with quantization. Default is `True`.

    - static_groups (bool) – Specifies whether the order of groups for quantization are static or can be dynamically adjusted. Default is `True`. Quark export only support static_groups as True.

    - inside_layer_modules (List[str]) – Lists the names of internal layer modules within the model that require specific quantization handling. Default is `None`.

    - model_decoder_layers (str) – Specifies custom settings for quantization on specific decoder layers of the model. Default is `None`.

  Example:

      from quark.torch.quantization.config.config import GPTQConfig

      gptq_config = GPTQConfig(
          inside_layer_modules=[
              "self_attn.k_proj",
              "self_attn.v_proj",
              "self_attn.q_proj",
              "self_attn.o_proj",
              "mlp.up_proj",
              "mlp.gate_proj",
              "mlp.down_proj"
          ],
          model_decoder_layers="model.layers"
      )

class quark.torch.quantization.config.config.GPTAQConfig(name: str = 'gptaq', block_size: int = 128, damp_percent: float = 0.01, alpha: float = 0.25, desc_act: bool = True, static_groups: bool = True, inside_layer_modules: list[str] = [], model_decoder_layers: str = '')

  A data class that defines the specifications for Accurate Post-Training Quantization for Generative Pre-trained Transformers (GPTQ).

  Parameters:

    - name (str) – The configuration name. Default is `"gptaq"`.

    - block_size (int) – GPTAQ divides the columns into blocks of size block_size and quantizes each block separately. Default is `128`.

    - damp_percent (float) – The percentage used to dampen the quantization effect, aiding in the maintenance of accuracy post-quantization. Default is `0.01`.

    - desc_act (bool) – Indicates whether descending activation is used, typically to enhance model performance with quantization. Default is `True`.

    - static_groups (bool) – Specifies whether the order of groups for quantization are static or can be dynamically adjusted. Default is `True`. Quark export only support static_groups as True.

    - inside_layer_modules (List[str]) – Lists the names of internal layer modules within the model that require specific quantization handling. Default is `None`.

    - model_decoder_layers (str) – Specifies custom settings for quantization on specific decoder layers of the model. Default is `None`.

  Example:

      from quark.torch.quantization.config.config import GPTAQConfig

      gptq_config = GPTAQConfig(
          inside_layer_modules=[
              "self_attn.k_proj",
              "self_attn.v_proj",
              "self_attn.q_proj",
              "self_attn.o_proj",
              "mlp.up_proj",
              "mlp.gate_proj",
              "mlp.down_proj"
          ],
          model_decoder_layers="model.layers"
      )

class quark.torch.quantization.config.config.QronosConfig(inside_layer_modules: list[str], model_decoder_layers: str, name: str = 'qronos', block_size: int = 128, desc_act: bool = True, static_groups: bool = True, alpha: float = 0.001, beta: float = 10000.0)

  Configuration for Qronos, an advanced post-training quantization algorithm. Implemented as proposed in https://arxiv.org/pdf/2505.11695

  Parameters:

    - inside_layer_modules (List[str]) – Lists the names of internal layer modules within the model that require specific quantization handling.

    - model_decoder_layers (str) – Specifies custom settings for quantization on specific decoder layers of the model.

    - name (str) – The configuration name. Default is `"qronos"`.

    - block_size (int) – Qronos divides the columns into blocks of size block_size and quantizes each block separately. Default is `128`.

    - desc_act (bool) – Indicates whether descending activation is used, typically to enhance model performance with quantization. Default is `True`.

    - static_groups (bool) – Specifies whether the order of groups for quantization are static or can be dynamically adjusted. Default is `True`. Quark export only supports `static_groups=True`.

    - alpha (float) – Dampening factor for numerical stability during matrix inversions. Default is `1e-6`.

    - beta (float) – Stabilisation factor for Cholesky decomposition. Default is `1e4`.

  Example:

      from quark.torch.quantization.config.config import QronosConfig

      qronos_config = QronosConfig(
          inside_layer_modules=[
              "self_attn.k_proj",
              "self_attn.v_proj",
              "self_attn.q_proj",
              "self_attn.o_proj",
              "mlp.up_proj",
              "mlp.gate_proj",
              "mlp.down_proj"
          ],
          model_decoder_layers="model.layers"
      )