[BUG]: The model Qwen3.8-27B is not loading

[aropb]

Description

LLama.Exceptions.LoadWeightsFailedException : Failed to load model 'Qwen3.8-27B-Q6_K.gguf'

https://huggingface.co/bartowski/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-Q6_K.gguf

[martindevans]

I guess this will need a llama.cpp update. There's a PR (#1414) pending at the moment that needs some more testing before merge and release, any assistance there would be appreciated :)

[aropb]

This PR doesn’t seem to be in the main branch yet, does it?
Will there be MTP support?

[martindevans]

That's correct, it's not merged yet because the other platforms need testing before it can be merged.

Will there be MTP support?

Unfortunately not. MTP (and all speculative drafting) is not implemented as a core part of the llama.cpp (in llama.h) but instead as part of the common extensions built on top of llama.cpp, which isn't something this project provides a wrapper for.

[zsogitbe]

Will there be MTP support?

Unfortunately not. MTP (and all speculative drafting) is not implemented as a core part of the llama.cpp (in llama.h) but instead as part of the common extensions built on top of llama.cpp, which isn't something this project provides a wrapper for.

Martin, is there a way to inspire people to develop additional functionality with some guidance? For example, by maintaining a collection of projects that interested members can work on.

I’m asking because MTP support can be implemented by adding an executor (MtpExecutor) in C# (correct me if I am wrong), with only minimal changes required in the C++ code. I can provide some instructions and plan for the MTP implementation if needed.

[martindevans]

MTP support can be implemented by adding an executor (MtpExecutor) in C# (correct me if I am wrong)

I think that's probably correct, but the issue is because we don't wrap common we don't have speculative.h available to us. That means you'd have to rebuild everything in speculative.cpp in C#, which is a huge job!

Rather than duplicating all of that work I think the right path would be to propose a c-api for spculative.h, and merge it upstream. That way we wouldn't be duplicating a lot of work. Then we could discuss expanding this project to cover that new API.

For example, by maintaining a collection of projects that interested members can work on.

I guess that's kind of the project board, although in practice nobody uses/maintains that at the moment. I get the impression you were probably thinking of something more than that though?

[zsogitbe]

MTP support can be implemented by adding an executor (MtpExecutor) in C# (correct me if I am wrong)

I think that's probably correct, but the issue is because we don't wrap common we don't have speculative.h available to us. That means you'd have to rebuild everything in speculative.cpp in C#, which is a huge job!

Rather than duplicating all of that work I think the right path would be to propose a c-api for spculative.h, and merge it upstream. That way we wouldn't be duplicating a lot of work. Then we could discuss expanding this project to cover that new API.

I completely understand your concern about duplicating the work in speculative.cpp. My main hesitation with proposing a C-API for speculative.h is that the common directory in llama.cpp changes incredibly fast and isn't intended to be a stable API. Tying ourselves to it might create constant maintenance churn for us on every upstream sync.

If we instead just expose the raw MTP logits from the core library (e.g., something like llama_get_mtp_logits_ith), we can build a native C# MtpExecutor. While it's certainly an upfront investment to write the C# port, it gives us full control over the managed execution loop and insulates us from the volatility of common. While I don't have the bandwidth to take on the full C# implementation myself right now, I've already done quite a bit of research on this. I'd be more than happy to share my notes and help jumpstart anyone who is interested in picking it up.

For example, by maintaining a collection of projects that interested members can work on.

I guess that's kind of the project board, although in practice nobody uses/maintains that at the moment. I get the impression you were probably thinking of something more than that though?

Exactly, I was thinking of something a bit more proactive than a standard project board. I worry we might be losing a bit of momentum compared to the blazing speed of upstream llama.cpp, and relying on the backlog isn't quite inspiring enough to pull in new contributors.

We need to find ways to lower the barrier to entry so C# developers feel comfortable jumping in. Maybe we could try actively highlighting 'Good First Issues', creating a more visible 'Help Wanted' roadmap for cool features (like MTP!), or showcasing community projects built with LLamaSharp to drive interest. I'm not entirely sure what the silver bullet is, but I'd love to brainstorm ways we can actively invite more people into the codebase.

[zsogitbe]

I have added an enhancement request to llama.cpp: ggml-org/llama.cpp#27469

[zsogitbe]

Just to follow up on my previous thought about building a C# MtpExecutor: @martindevans, you were completely right.

After digging deeper into common/speculative.cpp, MTP is far more complex than just reading auxiliary logits. It actually relies on a dual-context setup (ctx_tgt + ctx_dft), feeding dense hidden embeddings directly into batches (batch.embd), and managing dynamic layer offsets across multi-pass autoregressive steps. Replicating all of that plumbing across P/Invoke in C# would indeed be a huge maintenance nightmare.

Because of that, the issue I opened upstream focuses entirely on proposing a formalized, opaque llama_speculative_* API directly in llama.h. Hopefully the maintainers are receptive so we can cleanly wrap it down the road!

[aropb]

In the new version 0.29.0, it works.

[martindevans]

Yep that's the latest version, it's released on GitHub but releasing it to Nuget failed :(

[aropb]

How long does it usually take to issue a key?

[martindevans]

It's not something we have to do frequently, so there's no "usual" time tbh. I can't do it since the nuget package is not administrated by me (that's why the nuget key issue is assigned to SanftMonster).

[aropb]

Now it's clear, thank you. We'll be waiting.