Customize a model - gemma3

MAX Serving
1

As I understood the Model support | Modular part, I tried to get the gemma3 form Huggingface as:

max serve --model-path=unsloth/gemma-3-4b-it-GGUF --weight-path=unsloth/gemma-3-4b-it-GGUF/gemma-3-4b-it-Q6_K.gguf --quantization-encoding "q6_k"

But I got the below, am I doing something wrong?

(quickstart) [abuka@archlinux quickstart]$ max serve --model-path=unsloth/gemma-3-4b-it-GGUF --weight-path=unsloth/gemma-3-4b-it-GGUF/gemma-3-4b-it-Q6_K.gguf --quantization-encoding "q6_k"
23:46:53.995 INFO: 8615 MainThread: root: Logging initialized: Console: INFO, File: None, Telemetry: None
23:46:53.996 INFO: 8615 MainThread: root: Metrics initialized.
23:46:55.808 INFO: 8615 MainThread: max.pipelines: No GPUs available, falling back to CPU
23:46:55.808 INFO: 8615 MainThread: max.pipelines: No GPUs available, falling back to CPU
Traceback (most recent call last):
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/bin/max", line 10, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/lib/python3.13/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/lib/python3.13/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/lib/python3.13/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/lib/python3.13/site-packages/max/entrypoints/pipelines.py", line 66, in invoke
    return super().invoke(ctx)
           ~~~~~~~~~~~~~~^^^^^
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/lib/python3.13/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/lib/python3.13/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/lib/python3.13/site-packages/max/entrypoints/cli/config.py", line 251, in wrapper
    return func(*args, **kwargs)
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/lib/python3.13/site-packages/max/entrypoints/pipelines.py", line 146, in wrapper
    return func(*args, **kwargs)
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/lib/python3.13/site-packages/max/entrypoints/pipelines.py", line 191, in cli_serve
    pipeline_config = PipelineConfig(**config_kwargs)
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/lib/python3.13/site-packages/max/pipelines/lib/config.py", line 351, in __init__
    self.resolve()
    ~~~~~~~~~~~~^^
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/lib/python3.13/site-packages/max/pipelines/lib/config.py", line 416, in resolve
    self._validate_and_resolve_remaining_pipeline_config(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        model_config=self.model_config
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/abuka/Desktop/quickstart/.pixi/envs/default/lib/python3.13/site-packages/max/pipelines/lib/config.py", line 557, in _validate_and_resolve_remaining_pipeline_config
    raise ValueError(msg)
ValueError: quantization_encoding of 'SupportedEncoding.q6_k' not supported by MAX engine.
(quickstart) [abuka@archlinux quickstart]$ 

```
2

Sorry to have issues with max serve. Our model support page Gemma-3-it-4B Model | MAX Builds shows we support Gemma3 on GPU so that’s why you see the CPU quantization doesn’t work. Depending on our priorities we may be able to support it but in the meantime please create an issue here.