深層学習とベイズ統計
•
57 likes•19,532 views
This document summarizes a presentation on Bayesian deep learning and probabilistic programming. It discusses: 1. The history of Bayesian neural networks from 1987 to present, focusing on key papers. 2. An overview of Bayesian deep learning methodology, including Bayesian inference, variational inference, and Monte Carlo methods. 3. Probabilistic programming libraries like Edward that combine probabilistic modeling with deep learning frameworks like TensorFlow. 4. Examples of using Edward to build Bayesian neural networks and variational autoencoders for classification and generation. 5. References on Bayesian deep learning and the use of variational inference methods like Box's algorithm.
![p(θ|D) KL q(θ)
ELBO
⇤
= argmin KL(q(✓; ) || p(✓ | D))
= argmin Eq(✓; )[logq(✓; ) p(✓ | D)]
ELBO( ) = Eq(✓; )[p(✓, D) logq(✓; )]
⇤
= argmax ELBO( )](https://image.slidesharecdn.com/twm2017-08-05-170805064621/85/-21-320.jpg)
![- KL =ELBO
q(✓; 1)
q(✓; 5)
p(✓, D) p(✓, D)
✓✓
⇤
= argmax ELBO( )
ELBO( ) = Eq(✓; )[p(✓, D) logq(✓; )]](https://image.slidesharecdn.com/twm2017-08-05-170805064621/85/-23-320.jpg)
![[DL輪読会]Control as Inferenceと発展](https://cdn.slidesharecdn.com/ss_thumbnails/20191004-191204055019-thumbnail.jpg?width=560&fit=bounds)
![[DL輪読会]Convolutional Conditional Neural Processesと Neural Processes Familyの紹介](https://cdn.slidesharecdn.com/ss_thumbnails/20191220readingpaperconvcnp-191220034420-thumbnail.jpg?width=560&fit=bounds)
![[DL輪読会]Neural Ordinary Differential Equations](https://cdn.slidesharecdn.com/ss_thumbnails/nnasode1-190111001755-thumbnail.jpg?width=560&fit=bounds)
![[DL輪読会]Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://cdn.slidesharecdn.com/ss_thumbnails/20220325okimura-220405024717-thumbnail.jpg?width=560&fit=bounds)
More Related Content
What's hot (20)
Viewers also liked (14)
Similar to 深層学習とベイズ統計 (20)
![[DL輪読会]Recent Advances in Autoencoder-Based Representation Learning](https://cdn.slidesharecdn.com/ss_thumbnails/20190119dljournalclubweb-190401063633-thumbnail.jpg?width=560&fit=bounds)
More from Yuta Kashino (20)
Recently uploaded (20)
深層学習とベイズ統計
- 1. 2017-08-05 @ Tokyo Web Mining
- 2. Yuta Kashino ( ) BakFoo, Inc. CEO Astro Physics /Observational Cosmology Zope / Python Realtime Data Platform for Enterprise / Prototyping
- 3. Yuta Kashino ( ) arXiv stat.ML, stat.TH, cs.CV, cs.CL, cs.LG math-ph, astro-ph PyCon2016 @yutakashino https://www.slideshare.net/yutakashino/pyconjp2016
- 4. - - -
- 8. - Denker, Schwartz, Wittner, Solla, Howard, Jackel, Hopfield (1987) Denker and LeCun (1991) MacKay (1992) Hinton and van Camp (1993) Neal (1995) Barber and Bishop (1998) Graves (2011) Blundell, Cornebise, Kavukcuoglu, and Wierstra (2015) Hernandez-Lobato and Adam (2015)
- 9. - Yarin Gal Zoubin Ghahramani Shakir Mohamed Dastin Tran Rajesh Ranganath David Blei Ian Goodfellow Columbia U U of Cambridge
- 10. - - : - : - : - : - : SGD + BackProp … …x1 x2 xd ✓(2) ✓(1) x y y(n) = X j ✓ (2) j ( X i ✓ (1) ji x (n) i ) + ✏(n) p(y(n) | x(n) , ✓) = ( X i ✓ (n) i x (n) i ) ✓ D = {x(n) , y(n) }N n=1 = (X, y)
- 11. : - + - 2012 ILSVRC → 2015 - - - -
- 12. : - - ReLU, DropOut, Mini Batch, SGD(Adam), LSTM… - - ImageNet, MSCoCo… - : GPU, - : - Theano, Torch, Caffe, TensorFlow, Chainer, MxNet, PyTorch…
- 15. - = = -
- 16. - - : - : - : - : - : SGD + BackProp … …x1 x2 xd ✓(2) ✓(1) x y y(n) = X j ✓ (2) j ( X i ✓ (1) ji x (n) i ) + ✏(n) p(y(n) | x(n) , ✓) = ( X i ✓ (n) i x (n) i ) ✓ D = {x(n) , y(n) }N n=1 = (X, y)
- 17. - data hypothesis - - P(H | D) = P(H)P(D | H) P H P(H)P(D|H) P(x) = X y P(x, y) P(x, y) = P(x)P(y | x)
- 18. - : - : - - - : P(H | D) = P(H)P(D | H) P H P(H)P(D|H) likelihood priorposterior P(✓ | D, m) = P(D | ✓, m)P(✓ | m) P(D | m) m: P(x | D, m) = Z P(x | ✓, D, m)P(✓ | D, m)d✓ P(m | D) = P(D | m)P(m) P(D)
- 19. - - : - : - : - : … …x1 x2 xd ✓(2) ✓(1) x y ✓ D = {x(n) , y(n) }N n=1 = (X, y) P(✓ | D, m) = P(D | ✓, m)P(✓ | m) P(D | m) m: P(x | D, m) = Z P(x | ✓, D, m)P(✓ | D, m)d✓ prior
- 20. - (Variational Bayes) - (MCMC) P(✓ | D, m) = P(D | ✓, m)P(✓ | m) P(D | m) m:
- 21. p(θ|D) KL q(θ) ELBO ⇤ = argmin KL(q(✓; ) || p(✓ | D)) = argmin Eq(✓; )[logq(✓; ) p(✓ | D)] ELBO( ) = Eq(✓; )[p(✓, D) logq(✓; )] ⇤ = argmax ELBO( )
- 22. - VI - - David MacKay “Lecture 14 of the Cambridge Course” - PRML 10 http://www.inference.org.uk/itprnn_lectures/
- 23. - KL =ELBO q(✓; 1) q(✓; 5) p(✓, D) p(✓, D) ✓✓ ⇤ = argmax ELBO( ) ELBO( ) = Eq(✓; )[p(✓, D) logq(✓; )]
- 24. - p q - : - ADVI: Automatic Differentiation Variational Inference - BBVI: Blackbox Variational Inference q(✓; 1) q(✓; 5) p(✓, D) p(✓, D) ✓✓ arxiv:1603.00788 arxiv:1401.0118
- 25. Reference - Zoubin Ghahramani “History of Bayesian neural networks” NIPS 2016 Workshop Bayesian Deep Learning - Yarin Gal “Bayesian Deep Learning"O'Reilly Artificial Intelligence in New York, 2017
- 27. - Probabilistic Programing Library/Langage - Stan, PyMC3, Anglican, Church, Venture,Figaro, WebPPL, Edward - : Edward / PyMC3 - (VI) Metropolis Hastings Hamilton Monte Carlo Stochastic Gradient Langevin Dynamics No-U-Turn Sampler Blackbox Variational Inference Automatic Differentiation Variational Inference
- 28. Edward
- 29. Edward - Dustin Tran (Open AI) - Blei Lab - (PPL) - Stan, PyMC3, Anglican, Church, Venture,Figaro, WebPPL - 2016 2 PPL - TensorFlow - George Edward Pelham Box Box-Cox Trans., Box-Jenkins, Ljung-Box test box plot Tukey, 3 2 RA Fisher
- 30. PPL Edward TensorFlow(TF) + (PPL) TF: PPL: + + Python/Numpy
- 31. 1. TF: - - :
- 32. 1. TF:
- 33. 1. TF: - - - GPU / TPU Inception v3 Inception v4
- 34. 1. TF: - Keras, Slim - TensorBoard
- 35. 2. x: edward x⇤ s P(x | ↵) ✓⇤ ⇠ Beta(✓ | 1, 1)
- 36. 2. - ( ) Edward p(x, ✓) = Beta(✓ | 1, 1) 50Y n=1 Bernoulli(xn | ✓),
- 38. 3. Edward TF
- 39. 3. 256 28*28
- 40. 4. X, Z Z - (Variational Bayes) - (MCMC) p(z | x) = p(x, z) R p(x, z)dz .
- 41. 4.
- 43. 4. Edward KLqp
- 44. 5. Box’s loop George Edward Pelham Box Blei 2014
- 45. 5. Box’s loop
- 46. Edward - Edward = TensorFlow + + - TensorFlow - - TF GPU, TPU, TensorBoard, Keras - - Box’s Loop - Python
- 48. Refrence •D. Tran, A. Kucukelbir, A. Dieng, M. Rudolph, D. Liang, and D.M. Blei. Edward: A library for probabilistic modeling, inference, and criticism.(arXiv preprint arXiv:1610.09787) •D. Tran, M.D. Hoffman, R.A. Saurous, E. Brevdo, K. Murphy, and D.M. Blei. Deep probabilistic programming.(arXiv preprint arXiv:1701.03757) •Box, G. E. (1976). Science and statistics. (Journal of the American Statistical Association, 71(356), 791–799.) •D.M. Blei. Build, Compute, Critique, Repeat: Data Analysis with Latent Variable Models. (Annual Review of Statistics and Its Application Volume 1, 2014)
- 50. Dropout - Yarin Gal ”Uncertainty in Deep Learning” - Dropout - Dropout : conv - LeNet with Dropout http://mlg.eng.cam.ac.uk/yarin/blog_2248.html
- 51. Dropout - LeNet DNN - conv Dropout MNIST
- 52. Dropout - CO2
- 53. - - -
- 55. BakFoo, Inc. NHK NMAPS: +
- 57. BakFoo, Inc.
- 58. BakFoo, Inc. : SNS +