Nice speedup ,

Master:

Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes

PR:

Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes

I used 512 as MMQ_ITER_K so that all tile sizes remain the same, and it seems to be faster than the previous version.

To compile -DCMAKE_CUDA_ARCHITECTURES="120a" is required.

120f is most likely what you want - to also cover the DGX Spark (which is sm_121)

Hey everyone, hope we can get working on this together. I have been working on an NVFP4 implementation for llama for Blackwell for months and months now (just for fun in my spare time). I never was sure I would be able to actually do it (first time working on kernels) so was staying quiet, but I kept at it just to see if I could do it. I just fixed the last major bug tonight! I've been fighting weeks and hair pulling of junk tokens and mystery bugs, it's been rough. This was truly far more complicated than it seemed at first comparing to the MXFP4 I was looking at. But, omg, finally, it's all actually working, and itt's flying! I've made a mess so have a lot of cleanup to do before I want to post up anything. I'm using the PTX call with mxf4nvf4.block_scale.scale_vec::4X (m16n8k64) with e2m1/e2m1/UE4M3 scales per 16w and a global FP32 scale, and put in a bunch of other things. So I've got now working: cpu fallback, mmq (native mma), on the fly activations quant'd to nvfp4 (prefill), mmvq is still nvfp4 x q8_1 but I might try it with nvfp4. I have more work to do but I'll share a lot more soon, I just wanted to share the good news that I got it making output for the first time!! :)) :)) I never saw this PR until just now, would have totally got on board sooner.

@michaelw9999 for adding nvfp4 support I would first suggest to create a PR which adds the nvfp4 quantization, i.e how it exists for mxfp4. After this PR (and a PR which generally introduces nvfp4 into ggml) I think it should be easy to add support as only the on-the-fly activations should change, everything else should be relatively the same.

I would not suggest to use the x4 instruction for now as it will lead to a lot of changes and be harder to review

@michaelw9999 for adding nvfp4 support I would first suggest to create a PR which adds the nvfp4 quantization, i.e how it exists for mxfp4. After this PR (and a PR which generally introduces nvfp4 into ggml) I think it should be easy to add support as only the on-the-fly activations should change, everything else should be relatively the same.

I would not suggest to use the x4 instruction for now as it will lead to a lot of changes and be harder to review

I will clean it up a a lot! And try to make as clear and clean as possible. I've spent so much time focusing on getting it to 4X I feel letting that go would be sad and set me back a lot of effort. Nothing should come anywhere near messing with anything else. I could put up a PR for the NVFP4 quantization quite shortly, (I have GGML_TYPE_NVFP4 = 40). It's only 11pm here...so I'm usually at this for a few more hours into the night, and quite motivated tonight!

It is ultimately your call as to how to go about the PR, I would've added nvfp4 support after this PR but I will review whatever you post. The final authority is @JohannesGaessler in merging this part of the code

@michaelw9999 thank you for your work and interest in contributing to llama.cpp/ggml. For context, is the scope you're targeting only models like GPT-OSS that are already released as MXFP4 or more general NVFP4 quantization for any model?

Hello JohannesGaessler! What I have been working on is "true NVFP4" for any model, not really MXFP4, so these really are not at all the same thing. But there can for sure be some overlap and commonality with the new Blackwell PTX MMA functions with block scaling built in, and we can use some common functions between the two, but really these are different quantizations. We should definitely share some of the new helper functions to reduce bloat. I've made an entire new set of conversion functions for dealing with intrinsic fp4 and fp8s (from cuda_fp4.h and cuda_fp8.h) There were many issues to work out with unsigned/signed complexities problems with scaling/clamping/NaN and the pre-existing GGML_CUDA_xx functions not working so I had to re-do a lot of things (and hopefully go back, and un-do things where not necessary anymore). My understanding is the NVFP4 vs "FP4" difference (unfortunately very confusing naming) is that it's one scale per 16 weights block for NVFP4, optionally E4M3 scales(4x) or E8M0 scales(2x), but 32/block for MXFP4/E8M0, so NVFP4 should have higher precision (and be very, very fast on Blackwell SM120+). The NVFP4 implementation I've written (hoping to at least) be true enough to be convertible from any other already trained or quantized down NVFP4 models so full compatibility. I've been working with Qwen3 only the last few months but started on Nemotron3 yesterday... (surprised they did not release an NVFP4 model...) So I've made two entirely different approaches, the first never working at all yet, the second I went starting it by mimicking Q6_K as it seemed the closest related, so I set it up like a K quant with a superblock:

#define QK_NVFP4 16
typedef struct {
uint8_t qs[QK_K/2]; // 8 bytes of quants, so total of 16...4-bit E2M1 quants
uint8_t scales[QK_NVFP4]; // 16 scales in UE4M3 format
float d; // NV spec was one FP32 scale per tensor, but for starters here I put one per 256-weight superblock
} block_nvfp4;

Perhaps we ought to move this to a different thread so this is now deviating a bit from the MXFP4 side?

I would suggest we get on a call or similar to discuss details. To be 100% clear: even if you invest months of work that does not mean that it will make sense to merge (all of) your changes. In fact, if you invest a lot of time I think it's even more important to discuss requirements with maintainers. I am hosting a Mumble server that we can use to talk (contact me via the email on my Github page), but I am open to other means of communication as well.