A Primer on the Inner Workings of Transformer-based Language Models
The rapid progress of research aimed at interpreting the inner workings of advanced language models has highlighted a need for contextualizing the insights gained from years of work in this area. This primer provides a concise technical introduction to the current techniques used to interpret the inner workings of Transformer-based language models, focusing on the generative decoder-only architect
PowerPoint Presentation
NTECH2023 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. Novel Transformer Model Based Clustering Method for Standard Cell Design Automation Chia-Tung (Mark) Ho, Ajay Chandna, David Guan, Alvin Ho, Minsoo Kim, Yaguang Li and Haoxing (Mark) Ren | ISPD 2024 NTECH2023 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. OUTLINE • Introduction • Novel Transformer Model Based Clustering Method • Experimental Results • Con
NeurIPS 2023 Tutorial: Reconsidering Overfitting in the Age of Overparameterized Models
NeurIPS 2023 Tutorial: Reconsidering Overfitting in the Age of Overparameterized Models ` Large, overparameterized models such as neural networks are now the workhorses of modern machine learning. These models are often trained to near-zero error on noisy datasets and simultaneously generalize well to unseen data, in contrast to the textbook intuition regarding the perils of overfitting. At the sa
Zoology (Blogpost 2): Simple, Input-Dependent, and Sub-Quadratic Sequence Mixers
Table 1: Perplexity of 355 million parameter models trained for 10 billion tokens on the Pile. Yet, some subquadratic gated-convolutions match attention on the non AR slice! Can we capture the strengths of both gated convolutions and attention in one purely sub-quadratic architecture? We find the AR gap is because gated convolution models (e.g. Hyena, H3, RWKV, RetNet) need model dimension that sc
ã¯ã˜ã‚ã¦ã®è‡ªç„¶è¨€èªžå‡¦ç† Transformer 系モデルã®æŽ¨è«–高速化ã®æ¤œè¨¼ | オブジェクトã®åºƒå ´
今回㯠Transformer ç³»ã®ãƒ¢ãƒ‡ãƒ«ã€å…·ä½“çš„ã«ã¯ BERT, T5, GPT ã®æŽ¨è«–を高速化ã—ã¦ã¿ã¾ã™ã€‚高速化手法ã¨ã—㦠FasterTransformer, Torch-TensorRT, AWS Neuron を用ã„ã€ç´ ã® transfomers ã«æ¯”ã¹ã€ã©ã®ç¨‹åº¦é€Ÿããªã‚‹ã‹ï¼ˆãªã‚‰ãªã„ã‹ï¼‰ã€åˆ©ç‚¹ãƒ»æ¬ 点を確èªã—ã¦ã¿ã¾ã—ょã†ã€‚ 1. ã¯ã˜ã‚㫠今回㯠Transformer ç³»ã®ãƒ¢ãƒ‡ãƒ«ã€å…·ä½“çš„ã«ã¯ BERT, T5, GPT ã®æŽ¨è«–を様々ãªæŠ€è¡“を使ã£ã¦é«˜é€ŸåŒ–ã—ã¦ã¿ã¾ã™ã€‚ 高速化ã®å…ƒãƒã‚¿ã¯ Hugging Face ã® transformers1 縛りã¨ã—ã¦ã€ç´ ã® transformers ã§æŽ¨è«–ã™ã‚‹å ´åˆã«æ¯”ã¹ã€ ã©ã®ç¨‹åº¦é€Ÿããªã‚‹ã‹ï¼ˆãªã‚‰ãªã„ã‹ï¼‰è¦‹ã¦ã¿ã¾ã—ょã†ã€‚ 推論を高速化ã™ã‚‹æŠ€è¡“ã¨ã—ã¦ã¯ FasterTransfomer2, Torch-TensorRT3, AWS Neuron(
On Layer Normalization in the Transformer Architecture
The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show
Understanding the Difficulty of Training Transformers
Transformers have proved effective in many NLP tasks. However, their training requires non-trivial efforts regarding designing cutting-edge optimizers and learning rate schedulers carefully (e.g., conventional SGD fails to train Transformers effectively). Our objective here is to understand $\textit{what complicates Transformer training}$ from both empirical and theoretical perspectives. Our analy
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
RMT: Retentive Networks Meet Vision Transformers
Personal and Psychological Dimensions of AI Researchers Confronting AI Catastrophic Risks - Yoshua Bengio
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles
GCVit/models/gc_vit.py at main · NVlabs/GCVit
ResiDual: Transformer with Dual Residual Connections
Accelerated PyTorch 2 Transformers
GitHub - NVlabs/GCVit: [ICML 2023] Official PyTorch implementation of Global Context Vision Transformers
https://openreview.net/pdf?id=JroZRaRw7Eu
GitHub - Dao-AILab/flash-attention: Fast and memory-efficient exact attention
DETRs Beat YOLOs on Real-time Object Detection
GitHub - NVIDIA/FasterTransformer: Transformer related optimization, including BERT, GPT
bigscience/train/lessons-learned.md at master · bigscience-workshop/bigscience
Transformer Engine documentation — Transformer Engine 1.9.0 documentation
{{#tags}}- {{label}}
{{/tags}}