When to support Yi model

[hanmoley]

when I used the llama convert script convert Yi model, It converted successfully, but the engine did not work normally. the output is a character repeatedly

[Advait251206]

This symptom — engine runs but outputs a repeated character / looped token — is a classic sign that the model is technically converted but not semantically compatible with the expected LLaMA-style assumptions in TensorRT-LLM.

---

1. Why Yi models “convert successfully” but fail at runtime

Yi models (01.AI Yi series) are LLaMA-like, but not identical. Even if conversion scripts succeed, small architectural differences can break generation.

Common mismatches:

🔴 A. Tokenizer mismatch (most common cause)

If:

• vocab size is slightly different
• special tokens not aligned
• BOS/EOS handling differs

👉 Result:

• repeated tokens
• stuck generation loops
• nonsense output

---

🔴 B. RoPE / positional encoding differences

Yi models may use:

• different RoPE scaling
• different base frequency
• slight config differences vs LLaMA

If TensorRT-LLM treats it as vanilla LLaMA:

👉 attention becomes unstable → repetitive decoding

---

🔴 C. RMSNorm / layer norm subtle differences

Even small changes in:

• epsilon value
• normalization order

can cause:

👉 logits collapse → model outputs same token repeatedly

---

🔴 D. Weight mapping “looks correct but is not”

The conversion script may:

• map tensor names successfully
• but incorrectly interpret architecture variants

So engine builds fine, but behavior is wrong.

---

2. Why repeated characters happen specifically

This pattern:

"aaaaaaaaaaaa"
"the the the the"
"<EOS> <EOS> <EOS>"

usually means:

❗ logits collapse

The model output distribution becomes extremely peaked:

• one token dominates
• decoding loop keeps selecting it

This is almost always caused by:

• RoPE mismatch
• normalization mismatch
• embedding misalignment

---

3. Why “conversion succeeds” is misleading

TensorRT-LLM conversion scripts:

✔ only check structural compatibility
✔ do not validate numerical correctness
❌ do not verify generation quality

So:

engine build success ≠ correct model behavior

---

4. What is likely wrong in Yi → LLaMA conversion

Yi models typically require attention to:

✔ Correct config differences:

Check:

• rope_theta
• hidden_act
• rms_norm_eps
• tie_word_embeddings
• tokenizer special tokens

---

✔ Correct model type mapping

Yi is “LLaMA-like”, but not always:

• identical MLP structure
• identical attention scaling
• identical KV cache assumptions

---

5. How to fix or debug

✔ Step 1: verify tokenizer

Run HF model directly:

• does it generate correctly in Transformers?

If NOT → TRT-LLM is not the issue

---

✔ Step 2: compare config.json

Compare:

• Yi config
• LLaMA config used by TRT-LLM converter

Look for:

• rope scaling differences
• vocab mismatch
• norm epsilon differences

---

✔ Step 3: test FP16 baseline

Before TRT conversion:

• export HF → run PyTorch inference
• confirm correctness

Then:

• HF → ONNX (optional)
• ONNX → TRT-LLM

---

✔ Step 4: disable sampling issues

Sometimes repetition is exaggerated by:

• greedy decoding
• temperature = 0
• no repetition penalty

Try:

• temperature 0.7
• top-p sampling
• repetition penalty 1.1–1.2

---

6. Why Yi support is tricky in TRT-LLM

Yi models are:

• LLaMA-like but not LLaMA-identical
• trained with slightly different internals

TensorRT-LLM converters assume:

strict LLaMA/Mistral architecture compliance

So even small deviations break correctness.

---

7. Bottom line

• ✔ Engine building success does NOT guarantee correctness
• ❌ Repeated output = usually RoPE / norm / tokenizer mismatch
• ⚠ Yi models need careful config alignment, not just conversion
• ✔ First validate in Hugging Face before blaming TRT-LLM

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206