@martinakaduc I see some of my comments in here--did you figure out the necessary changes to the preprocess_gemma function? :)
FWIW yours looks right, I realize that I may have been running an old version of the chat template that didn't include the BOS token when I was developing.

preprocess_gemma

@muhark Yes, I have referenced your code. I also figured out the problem with preprocess_gemma where <end_of_turn>\n requires 2 tokens.

I have successfully trained my llava_gemma for Vietnamese! 😄 (https://huggingface.co/ura-hcmut/GemSUraV-7B)

@martinakaduc Sorry to bother. I see some warnings when pretrain Mixtral 8*7B, just like #1417. Did you face the same problem? Thanks in advance!

why did you use
"additional_special_tokens": [
"<start_of_turn>",
"<end_of_turn>"
],
instead of
"bos_token": "",
"eos_token": "",