I'm sure a pure ViT is perfectly useable but I found the hybrid version to be more stable during training.
Another factor is that for smaller model sizes the hybrid versions actually perform better (see Appendix C in the ViT paper)
The images are actually downscaled by the ResNet, but always by the same factor so it’s no problem for the ViT.

Visualising the resnet embeddings seems reasonable but I don’t have code for that I could share. For the ViT, it’s the attention mask you want to see. The model will most likely only pay attention to the current character in the sequence. I don’t think it’s very interesting to do either but I won’t stop you :)