Skip to content

mtmd: (WIP) gemma3n vision support #17961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ngxson wants to merge 23 commits into
base: master
Choose a base branch
from
+1,492 −5
Draft

mtmd: (WIP) gemma3n vision support #17961

Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
c5e90a1
convertible to gguf
ngxson May 24, 2025
2e0a9df
weight loaded ok, missing cgraph
ngxson May 24, 2025
fd3b181
wip
ngxson May 24, 2025
122a54f
wip
ngxson May 27, 2025
ba8dbcc
activations matched until attn_post_norm
ngxson May 27, 2025
c533872
matched activations until `l_out-19`
ngxson May 31, 2025
9b90d04
Merge branch 'master' into gemma3n_text
ngxson May 31, 2025
cc80c3b
small clean up
ngxson May 31, 2025
a66ac3f
clean up, break into smaller fn
ngxson Jun 1, 2025
787f73f
cpu-only is ok, still missing icdf
ngxson Jun 1, 2025
80bee4e
replace ggml_set with ggml_concat
ngxson Jun 1, 2025
ee6703a
matched text output
ngxson Jun 1, 2025
4bfbfa1
Merge branch 'master' into gemma3n_text
ngxson Jun 19, 2025
4d3d0ae
fix merge conflict
ngxson Jun 19, 2025
d8589f8
update text conversion
ngxson Jun 19, 2025
e5fe215
add kq weighted norm
ngxson Jun 21, 2025
d42071e
fix f_attention_scale
ngxson Jun 21, 2025
f4e7034
init
ngxson Jun 22, 2025
f8f73e8
conversion script
ngxson Jun 22, 2025
96fd71d
wip
ngxson Jun 22, 2025
325cbe7
add MultiQueryAttentionBlock
ngxson Jun 23, 2025
3161c0c
fix rms_norm_act_2d
ngxson Jun 25, 2025
59e6091
wip
ngxson Jun 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update text conversion
  • Loading branch information
@ngxson
ngxson committed Jun 19, 2025
commit d8589f8bb45bbcaef049dc780b8721f8113ebc7c
7 changes: 5 additions & 2 deletions convert_hf_to_gguf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4262,7 +4262,7 @@ def set_gguf_parameters(self):
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
del bid # unused

if name.startswith("language_model."):
if "language_model." in name:
name = name.replace("language_model.", "")

elif name.startswith("multi_modal_projector.") or name.startswith("vision_tower.") \
Expand Down Expand Up @@ -4336,7 +4336,7 @@ def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iter
return [] # skip other tensors


@ModelBase.register("Gemma3p5ForCausalLM")
@ModelBase.register("Gemma3nForConditionalGeneration")
class Gemma3NModel(Gemma3Model):
model_arch = gguf.MODEL_ARCH.GEMMA3N
norm_shift = 0.0 # same value with Gemma3p5RMSNorm scale_shift on python code
Expand Down Expand Up @@ -4374,6 +4374,9 @@ def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iter

# TODO: implement self.prediction_coefs.weight.clamp_(...)

if "language_model." not in name:
return [] # skip non-language model tensors

if "embed_tokens_per_layer.weight" in name:
hidden_size_per_layer_input = 256
data_torch = data_torch * (hidden_size_per_layer_input**0.5)
Expand Down