The Big LLM Architecture Comparison
Last updated: Dec 18, 2025 It has been seven years since the original GPT architecture was developed. At first glance, looking back at GPT-2 (2019) and forward to DeepSeek V3 and Llama 4 (2024-2025), one might be surprised at how structurally similar these models still are. Sure, positional embeddings have evolved from absolute to rotational (RoPE), Multi-Head Attention has largely given way to Gr
「LLMã¯ã‚³ãƒ³ãƒ†ã‚ストãŒã™ã¹ã¦ã€ã‹ã‚‚ã—ã‚Œãªã„
コンテã‚ストエンジニアリングã«ã¤ã„㦠LLM（大è¦æ¨¡è¨€èªžãƒ¢ãƒ‡ãƒ«ï¼‰ã®åˆ†é‡Žã§ã€æœ€è¿‘「コンテã‚ストエンジニアリング（Context Engineering）ã€ã¨ã„ã†è¨€è‘‰ãŒå¤šã使ã‚れるよã†ã«ãªã‚Šã¾ã—ãŸã€‚AIエージェントã®æ–‡è„ˆã§ã‚‚使ã‚れるã“ã¨ãŒå¤šãã€è‡ªåˆ†ã®ä¸ã§ãšã£ã¨ãƒ¢ãƒ¤ãƒ¢ãƒ¤ã—ã¦ã„ãŸã®ã§ã™ãŒã€å°‘ã—自分ãªã‚Šã«æ•´ç†ã—ã¦ã¿ãŸã®ã§ã“ã“ã«æ›¸ã„ã¦ã¿ã¾ã™ã€‚ åŠåˆ†ä»¥ä¸ŠãŠæ°—æŒã¡ã¨ã„ã†ã‹ãƒã‚¨ãƒ ã‚„ç§è¦‹ãŒæ··ã˜ã£ã¦ã„ã¾ã™ã®ã§ã€å¦è¡“çš„ãªå®šç¾©ã®åŽ³å¯†æ€§ã‚ˆã‚Šã€è‡ªåˆ†ãŒæ™®æ®µä½¿ã£ã¦ã„ã¦æ„Ÿã˜ã‚‹å®Ÿè·µç›®ç·šã§ã®ä¸€ã¤ã®è€ƒãˆæ–¹ã¨ã—ã¦æ‰ãˆã¦ã‚‚らãˆã‚‹ã¨ã‚ã‚ŠãŒãŸã„ã§ã™ã€‚ 「プãƒãƒ³ãƒ—トエンジニアリングã€ã‹ã‚‰ã€Œã‚³ãƒ³ãƒ†ã‚ストエンジニアリングã€ã¸ ãã‚‚ãも「コンテã‚ストエンジニアリングã€ã£ã¦ä½•ï¼Ÿã€Œãƒ—ãƒãƒ³ãƒ—トエンジニアリングã€ã¨ä½•ãŒé•ã†ã®ï¼Ÿã¨ã„ã†ã¨ã“ã‚ã‹ã‚‰å§‹ã‚ãŸã„ã¨æ€ã„ã¾ã™ã€‚ プãƒãƒ³ãƒ—トエンジニアリングã¯ã€ã‚‚ã®ã™ã”ã„å˜ç´”ã«ã—ãŸå›³ã«ã™ã‚‹ã¨ä»¥ä¸‹ã«ãªã‚‹ã¨æ€ã„ã¾ã™ã€‚ プãƒãƒ³ãƒ—
GitHub - google/langextract: A Python library for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visualization.
Precise Source Grounding: Maps every extraction to its exact location in the source text, enabling visual highlighting for easy traceability and verification. Reliable Structured Outputs: Enforces a consistent output schema based on your few-shot examples, leveraging controlled generation in supported models like Gemini to guarantee robust, structured results. Optimized for Long Documents: Overcom
2025-05-02 Stanford CS336 Language Modeling from Scratch: GPUã®è¬Žã‚’解ã - Flash Attentionã¾ã§ã®æœ€é©åŒ–技術完全ガイド
2025-05-02 Stanford CS336 Language Modeling from Scratch: GPUã®è¬Žã‚’解ã - Flash Attentionã¾ã§ã®æœ€é©åŒ–技術完全ガイド ※本記事ã¯ã€Stanford CS336 Language Modeling from Scratch Spring 2025ã®è¬›ç¾©å‹•ç”»ã€ŒGPUsã€ã®å†…容を基ã«ä½œæˆã•ã‚Œã¦ã„ã¾ã™ã€‚講義ã®è©³ç´°æƒ…å ±ã¯ https://stanford-cs336.github.io/spri... ã§ã”覧ã„ãŸã ã‘ã¾ã™ã€‚Stanford大å¦ã®ã‚ªãƒ³ãƒ©ã‚¤ãƒ³AIプãƒã‚°ãƒ©ãƒ ã«ã¤ã„ã¦ã¯ https://stanford.io/ai ã€æœ¬è¬›ç¾©ã¸ã®ç™»éŒ²ã«ã¤ã„ã¦ã¯ https://online.stanford.edu/courses/c... ã‚’ã”å‚ç…§ãã ã•ã„。 本記事ã§ã¯ã€è¬›ç¾©ã®å†…容を詳細ã«ã¾ã¨ã‚ã¦ãŠã‚Šã¾ã™ãŒã€è¦ç´„や解釈ã«ã‚ˆã‚‹èª¤ã‚ŠãŒ
Stanford CS336 Language Modeling from Scratch I 2025
Language models serve as the cornerstone of modern natural language processing (NLP) applications and open up a new paradigm of having a single general purpo...
Context Engineering
TL;DRAgents need context to perform tasks. Context engineering is the art and science of filling the context window with just the right information at each step of an agent’s trajectory. In this post, we break down some common strategies — write, select, compress, and isolate — for context engineering by reviewing various popular agents and papers. We then explain how LangGraph is designed to supp
Context Engineering for AI Agents: Lessons from Building Manus
Context Engineering for AI Agents: Lessons from Building Manus At the very beginning of the Manus project, my team and I faced a key decision: should we train an end-to-end agentic model using open-source foundations, or build an agent on top of the in-context learning abilities of frontier models? Back in my first decade in NLP, we didn't have the luxury of that choice. In the distant days of BER
Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research into CoT monitorability and investment in CoT monitoring alon
Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety — AI Alignment Forum
It is interesting to note how views on this topic have shifted with the rise of outcome-based RL applied to LLMs. A couple of years ago, the consensus in the safety community was that process-based RL should be prioritized over outcome-based RL, since it incentivizes choosing actions for reasons that humans endorse. See for example Anthropic's Core Views On AI Safety: Learning Processes Rather tha
AIエージェントã®ã‚µãƒ¼ãƒ“ス構築を検討ã—ã¦ã„ã‚‹ã‚ãªãŸã¸
ã¯ã˜ã‚㫠本記事ã§ã¯ã€æ˜¨ä»Šè©±é¡Œã® AI エージェントサービスをã“ã‚Œã‹ã‚‰ä½œã‚‹æ–¹ã¸ã€AI エージェントシステムを作る上ã§ã®æŠ‘ãˆã‚‹ã¹ããƒã‚¤ãƒ³ãƒˆã¨å®Ÿè·µçš„ãªãƒãƒ³ã‚ºã‚ªãƒ³ã«ã¦ DeepDive ã—ã¦ã„ããŸã„ã¨æ€ã„ã¾ã™ã€‚ 本記事㯠2 部構æˆã«ãªã£ã¦ãŠã‚Šã€ä»¥ä¸‹ã®æ§‹æˆã§ã™ã€‚ 第 1 部: AI エージェント ã®åŸºæœ¬æ¦‚念ã¨ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆã‚·ã‚¹ãƒ†ãƒ 構築ã®ã‚¬ã‚¤ãƒ‰ 第 2 部: Azure AI Agent Service を使ã£ãŸãƒ¯ãƒ¼ã‚¯ãƒ•ãƒãƒ¼ãƒ«ãƒ¼ãƒ†ã‚£ãƒ³ã‚°ã®å®Ÿè£… 第１部ã§ã¯ã€OpenAI 社ã®a-practical-guide-to-building-agents ã‚’å‚考ã«ã€AI エージェントã®åŸºæœ¬æ¦‚念ã¨ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆã‚·ã‚¹ãƒ†ãƒ 構築ã®ã‚¬ã‚¤ãƒ‰ã‚’解説ã—ã¾ã™ã€‚ 第２部ã§ã¯ã€Anthropic ã®ãƒ–ãƒã‚°è¨˜äº‹ Building Effective Agents – Workflow Routing ã§ç´¹ä»‹ã•ã‚Œã¦ã„ã‚‹ ワークフãƒãƒ¼ãƒ«ãƒ¼
vLLMã®Speculative Decodingã«ã‚ˆã‚‹æŽ¨è«–高速化を試ã™
ã¯ã˜ã‚ã« ã“ã®è¨˜äº‹ã§ã¯ã€Speculative Decodingã«ã‚ˆã‚‹LLMã®æŽ¨è«–高速化をvLLMã§è©¦ã—ã€ç°¡å˜ãªãƒ™ãƒ³ãƒãƒžãƒ¼ã‚¯ã‚’è¡Œã£ãŸçµæžœã‚’共有ã—ã¾ã™ã€‚ Speculative Decodingã«ã¤ã„㦠最åˆã«ã€Speculative Decodingã«ã¤ã„ã¦ç°¡å˜ã«è§£èª¬ã—ã¾ã™ã€‚ Speculative Decodingã¨ã¯ã€å¤§åž‹ã®ãƒ¢ãƒ‡ãƒ«ã®æŽ¨è«–ã‚’ã™ã‚‹éš›ã€ã‚ˆã‚Šå°åž‹ã®ãƒ¢ãƒ‡ãƒ«ã‚’利用ã—ã¦æŽ¨è«–を高速化ã™ã‚‹æ‰‹æ³•ã§ã™ã€‚ã“ã®æœ¬æ¥ã®å‡ºåŠ›ã‚’å¾—ãŸã„大型ã®ãƒ¢ãƒ‡ãƒ«ã‚’Target Modelã€é«˜é€ŸåŒ–ã®ãŸã‚ã®å°åž‹ã®ãƒ¢ãƒ‡ãƒ«ã‚’Draft Modelã¨è¨€ã„ã¾ã™ã€‚ Speculative Decodingã§ã¯é€šå¸¸ã®æŽ¨è«–ã¨ã¯é•ã„ã€æŽ¨è«–ã®éš›ã«ã¾ãšå°åž‹ã®Draft ModelãŒä¸€å®šã®Draft Tokens分ã®ç”Ÿæˆã‚’è¡Œã„ã€å€™è£œã¨ãªã‚‹ãƒˆãƒ¼ã‚¯ãƒ³åˆ—ã‚’æ案ã—ã¾ã™ã€‚ãã®å¾ŒTarget Modelã¯ã“ã®Draft Tokensã«å¯¾ã—ã¦ç¢ºçŽ‡åˆ†å¸ƒã‚’å…ƒã«
LLMã®æŽ¨è«–速度を劇的ã«åŠ 速ã™ã‚‹æ–¹æ³• Speculative Decoding ã®è§£èª¬ – blog
ã¯ã˜ã‚㫠皆ã•ã‚“ã«è³ªå•ã§ã™ã€‚ 「モデルã®ç²¾åº¦ã‚’è½ã¨ã•ãšã€è¨ˆç®—リソースも増やã•ãšã€æŽ¨è«–速度ã ã‘ã‚’2å€ã«ã™ã‚‹æ–¹æ³•ã€ ãŒã‚ã‚‹ã¨ã—ãŸã‚‰â€”—ãã‚Œã¯é”法ã§ã—ょã†ã‹ï¼Ÿãã‚Œã¨ã‚‚ç¾å®Ÿã®æŠ€è¡“ã§ã—ょã†ã‹ï¼Ÿ ç”ãˆã¯å¾Œè€…ã§ã™ã€‚Google DeepMindã¨UC BerkeleyãŒå…±åŒé–‹ç™ºã—ãŸSpeculative Decodingã¯ã€ã¾ã•ã«ã“ã®ä¸å¯èƒ½ã‚’å¯èƒ½ã«ã™ã‚‹ã€ŒæŽ¨è«–åŠ é€Ÿã®ãƒ–ラックボックスã€ã€‚自動車ã§ä¾‹ãˆã‚Œã°ã€ãƒŠãƒ“ã®äºˆæ¸¬ãƒ«ãƒ¼ãƒˆå€™è£œã‚’事å‰è¨ˆç®—ã—ã¤ã¤ã€å®Ÿéš›ã®èµ°è¡Œã§æœ€é©çµŒè·¯ã‚’é¸æŠžã™ã‚‹ã‚ˆã†ãªå·§å¦™ãªæ‰‹æ³•ã§ã€LLMã®ç”Ÿæˆé€Ÿåº¦ã«é©å‘½ã‚’èµ·ã“ã—ã¾ã™ã€‚ 「Speculative Decodingã€ã£ã¦ä½•ï¼Ÿ 「Speculative Decodingã€ã¯æ—¥æœ¬èªžã§ã€ŒæŽ¨æ¸¬çš„デコーディングã€ã¨è¨³ã•ã‚Œã‚‹ã“ã¨ãŒå¤šãã€ç›´è¨³ã«è¿‘ã„表ç¾ã¨ã—ã¦ã€ŒæŠ•æ©Ÿçš„デコーディングã€ã¨å‘¼ã°ã‚Œã‚‹ã“ã¨ã‚‚ã‚ã‚Šã¾ã™ã€‚ã“ã®æ‰‹æ³•ã‚’ç°¡å˜ã«è¨€ã†ã¨ã€å°ã•ãªãƒ¢ãƒ‡ãƒ«ï¼ˆãƒ‰ãƒ©ãƒ•ãƒˆãƒ¢ãƒ‡ãƒ«ï¼‰ã§è¤‡æ•°
ランã‚ング
ランã‚ング
メンテナンス
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
GitHub - mjun0812/flash-attention-prebuild-wheels: Provide with pre-build flash-attention package wheels on Linux and Windows platforms using GitHub Actions
GitHub - hemingkx/SpeculativeDecodingPapers: 📰 Must-read papers and blogs on Speculative Decoding âš¡ï¸
Stanford CS336 | Language Modeling from Scratch
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
AI エージェントã¨è€ƒãˆç›´ã™ãƒ‡ãƒ¼ã‚¿åŸºç›¤
Accelerating Large Language Model Decoding with Speculative Sampling
GitHub - FasterDecoding/Medusa: Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads
GitHub - SafeAILab/EAGLE: Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).
{{#tags}}- {{label}}
{{/tags}}