[quantization] Use QuantLlamaForCausalLM#531
Merged
mhs4670go merged 1 commit intoSamsung:mainfrom Feb 27, 2026
Merged
Conversation
stamalakhov
force-pushed
the
use_quant_autocasual_lm_PR
branch
from
1023a8b to
515381a
Compare
This PR: 1. uses `QuantLlamaForCausalLM` for LLama quantization 2. removes additional transforms of attention_mask/position_ids to prevent their exploding 3. removes unused code 4. adjusts tests accordingly TICO-DCO-1.0-Signed-off-by: s.malakhov <s.malakhov@partner.samsung.com>
stamalakhov
force-pushed
the
use_quant_autocasual_lm_PR
branch
from
515381a to
f2964d8
Compare
Contributor
Author
|
@mhs4670go |
mhs4670go
reviewed
Feb 27, 2026
| hidden_states.device | ||
| ) | ||
| attention_mask = self._fq(attention_mask, self.obs_causal_mask) | ||
| attention_mask = attention_mask.squeeze(0) |
Contributor
There was a problem hiding this comment.
If we generate squeezed attention_mask whose shape is (1, L, L), do you think this squeeze would become redundant? Of course constant folding can be the same thing though.
Contributor
Author
There was a problem hiding this comment.
yes. i think it will be redundant.
Contributor
No, I think I can review them here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR:
In short this PR uses all static data (casual_mask/position_embeddings) calculated at QuantLlamaModel level and uses inside all the layers to prevent their exploding. Currently all layers uses their own static data which induces large overhead for not-small
max_seq_len(~2048).python tico/quantization/wrapq/examples/quantize_full_qmodel_with_gptq.py --model HuggingFaceTB/SmolLM2-135M-Instruct --gptq_mse --max_seq_len 2048 --save_circle_to_folder "."
Namespace(model='HuggingFaceTB/SmolLM2-135M-Instruct', device='cuda', dtype='float32', seed=42, trust_remote_code=False, hf_token=None, no_tqdm=False, no_GPTQ=False, no_PTQ=False, save_circle_to_folder='.', cache_dir='/mnt/storage/transformers_cache', nsamples_for_qcalibration=128, linear_weight_bits=4, gptq_mse=True, max_seq_len=2048, embedding_weight_bits=8, lm_head_weight_bits=4, eval_tasks=None) === Config === Model : HuggingFaceTB/SmolLM2-135M-Instruct Device : cuda DType : float32Loading FP model …
Calculating original perplexities …
Token indices sequence length is longer than the specified maximum sequence length for this model (304978 > 8192). Running this sequence through the model will result in indexing errors
PPL: 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 148/149 [00:18<00:00, 7.86it/s]
┌── Wikitext-2 test perplexity ─────────────
│ FP32 : 17.40
└───────────────────────────────────────────
Applying GPTQ …
Quantizing layers: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [01:34<00:00, 3.15s/layer]
Wrapping layers with PTQWrapper …
Calibrating PTQ obeservers…
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 128/128 [01:26<00:00, 1.48it/s]
Calculating perplexities …
PPL: 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 148/149 [01:15<00:00, 1.95it/s]
┌── Wikitext-2 test perplexity ─────────────
│ int16 : 27.73
└───────────────────────────────────────────
saving the whole model to /mnt/storage/slow_repos/TICO/model.q.circle
it produced
model.q.circlefile of111.2 MBsize instead of> 300 MBof previous version.Draft: #495
TICO-DCO-1.0-Signed-off-by: s.malakhov s.malakhov@partner.samsung.com