Quantization Strategies

Quantization Strategies#

class quark.onnx.quantization.config.spec.CalibMethod(*values)[source]#

Enumeration of calibration methods used for determining quantization parameters.

class quark.onnx.quantization.config.spec.ScaleType(*values)[source]#

Enumeration of scale types used in quantization.

class quark.onnx.quantization.config.spec.QuantGranularity(*values)[source]#

Enumeration of quantization granularity.

class quark.onnx.quantization.config.spec.QTensorConfig(symmetric: bool, scale_type: ScaleType, calibration_method: CalibMethod, quant_granularity: QuantGranularity, data_type: BaseDataType)[source]#

Configuration for a quantized tensor.

Parameters:

  • symmetric (bool) – Whether to use symmetric quantization.

  • scale_type (ScaleType) – Type of scaling to apply.

  • calibration_method (CalibMethod) – Method for calibration.

  • quant_granularity (QuantGranularity) – Level of quantization granularity.

  • data_type (BaseDataType) – Data type of quantization.

set_symmetric(symmetric: bool) → None[source]#

Set whether symmetric quantization is used.

set_scale_type(scale_type: ScaleType) → None[source]#

Set the scale type.

set_calibration_method(calibration_method: CalibMethod) → None[source]#

Set the calibration method.

set_quant_granularity(quant_granularity: QuantGranularity) → None[source]#

Set the quantization granularity.

set_data_type(data_type: BaseDataType) → None[source]#

Set the data type.

class quark.onnx.quantization.config.spec.Int8Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.Percentile, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.Int8'>)[source]#

Quantization specification for int8 tensors (default Float32 scaling and Percentile calibration).

class quark.onnx.quantization.config.spec.UInt8Spec(symmetric: bool = False, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.Percentile, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.UInt8'>)[source]#

Quantization specification for uint8 tensors.

class quark.onnx.quantization.config.spec.XInt8Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.PowerOf2, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.MinMSE, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.Int8'>)[source]#

Quantization specification for int8 tensors with power-of-2 scaling.

class quark.onnx.quantization.config.spec.Int16Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.Percentile, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.Int16'>)[source]#

Quantization specification for int16 tensors.

class quark.onnx.quantization.config.spec.UInt16Spec(symmetric: bool = False, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.Percentile, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.UInt16'>)[source]#

Quantization specification for uint16 tensors.

class quark.onnx.quantization.config.spec.Int32Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.Percentile, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.Int32'>)[source]#

Quantization specification for int32 tensors.

class quark.onnx.quantization.config.spec.UInt32Spec(symmetric: bool = False, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.Percentile, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.UInt32'>)[source]#

Quantization specification for uint32 tensors.

class quark.onnx.quantization.config.spec.BFloat16Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.MinMax, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.BFloat16'>)[source]#

Specification for bfloat16 tensors.

class quark.onnx.quantization.config.spec.BFP16Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.MinMax, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.BFP16'>)[source]#

Specification for Block Floating Point (BFP16) tensors.

class quark.onnx.quantization.config.spec.MX4Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.MinMax, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.MX4'>)[source]#

Specification for MX4 tensors.

class quark.onnx.quantization.config.spec.MX6Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.MinMax, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.MX6'>)[source]#

Specification for MX6 tensors.

class quark.onnx.quantization.config.spec.MX9Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.MinMax, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.MX9'>)[source]#

Specification for MX9 tensors.

class quark.onnx.quantization.config.spec.MXFP4E2M1Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.MinMax, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.MXFP4E2M1'>)[source]#

Specification for MXFP4E2M1 tensors.

class quark.onnx.quantization.config.spec.MXFP6E3M2Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.MinMax, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.MXFP6E3M2'>)[source]#

Specification for MXFP6E3M2 tensors.

class quark.onnx.quantization.config.spec.MXFP6E2M3Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.MinMax, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.MXFP6E2M3'>)[source]#

Specification for MXFP6E2M3 tensors.

class quark.onnx.quantization.config.spec.MXFP8E5M2Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.MinMax, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.MXFP8E5M2'>)[source]#

Specification for MXFP8E5M2 tensors.

class quark.onnx.quantization.config.spec.MXFP8E4M3Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.MinMax, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.MXFP8E4M3'>)[source]#

Specification for MXFP8E4M3 tensors.

class quark.onnx.quantization.config.spec.MXInt8Spec(symmetric: bool = True, scale_type: ~quark.onnx.quantization.config.spec.ScaleType = ScaleType.Float32, calibration_method: ~quark.onnx.quantization.config.spec.CalibMethod = CalibMethod.MinMax, quant_granularity: ~quark.onnx.quantization.config.spec.QuantGranularity = QuantGranularity.Tensor, data_type: type[~quark.shares.data_type.BaseDataType] = <class 'quark.onnx.quantization.config.data_type.MXInt8'>)[source]#

Specification for MXInt8 tensors.

class quark.onnx.quantization.config.spec.QLayerConfig(input_tensors: QTensorConfig | None = None, activation: QTensorConfig | None = None, weight: QTensorConfig | None = None, bias: QTensorConfig | None = None, output_tensors: QTensorConfig | None = None)[source]#

Layer-level quantization configuration.

Parameters: