SlideShare a Scribd company logo

企業における自然言語処理技術利用の最先端

•
•
Yuya Unno
Yuya Unno

1. This document discusses the history and recent developments in natural language processing and deep learning. It covers seminal NLP papers from the 1990s through 2000s and the rise of neural network approaches for NLP from 2003 onward. 2. Recent years have seen increased research and investment in deep learning, with many large companies establishing AI labs in 2012-2014 to focus on neural network techniques. 3. The document outlines some popular deep learning architectures for NLP tasks, including neural language models, word2vec, sequence-to-sequence learning, and memory networks. It also introduces the Chainer deep learning framework for Python.

1 of 65
Downloaded 183 times
1 Preferred Infrastructure 2015/06/19
!  -2008 !  !  2008-2011 t 1 !  t !  2011- 1 t !  t !  !  Jubatus NLP 12014- t 12015,
NLP 1YANS !  YANS 19 !  !  -‐‑‒19/3 9/5 !  YANS 13 !  !  "
1 !  ! 
2 !  !  !  !  !  ! 
!  !  !  !  t PFI
3 1.  2.  t 3. 
1.
!  !  ---- ---- !  Web EC t Web !  1CGM ---- ---- !  SNS1Twitter Facebook !  1Line !  t
!  !  !  !  !  !  t WWW !  Blog !  SNS
企業における自然言語処理技術利用の最先端
!  EC t !  !  !  !  !  !  SNS !  !  t
!  EC t !  11000 !  !  !  !  !  SNS !  ! 
t !  !  !  !  !  !  SNS !  ! 
vs vs vs !  !  etc. !  UI
!  !  !  !  !  !  1 1 SNS !  t
!  !  !  !  !  !  1
2. t
24 http://www.soumu.go.jp/johotsusintokei/whitepaper/ja/h24/html/nc122310.html
!  ! 
t t : Siri https://www.apple.com/jp/ios/siri/ http://v-assist.yahoo.co.jp/ https://www.nttdocomo.co.jp/service/information/shabette_concier/
Gunosy, SmartNews http://gunosy.com https://www.smartnews.be/
11/2 !  !  1970 t PC !  !  !  2000 !  t
12/2 !  !  RSS 2000 !  !  1 !  2000 ! 
!  !  t !  !  !  ! 
t !  !  t !  !  t !  !  !  !  !  ! 
t
t !  ! 
!  !  <12:00 :25
t !  !  !  !  !  !  t
t !  t !  !  t !  t !  !  t
3.
企業における自然言語処理技術利用の最先端
!  !  1993: [Brown+93] !  1996: [Berger+96] !  2001: [Lafferty+01] !  t !  2003: Latent Dirichlet Allocation [Blei+03] !  2006: Pitman-Yor language model [Teh06] !  !  2006: [Clarke+06][Riedel+06] !  2010: [Koo+10][Rush+10] !  !  2003: Neural language model [Bengio+03] !  2010: Recurrent Neural Network [Mikolov+10] !  2013: Skipgram Model (word2vec) [Mikolov+13]
!  !  90 2010 ! 
!  ! 
1. !  2011: 30% # 20% !  2012: 26% # 16% http://image-net.org/challenges/LSVRC/2012/ilsvrc2012.pdf
2. !  2012/3: Google Hinton DNNresearch !  2012/4: Baidu Institute of Deep Learning !  2012/8, 10: Yahoo! IQ Engines LookFlow !  2012/12: Facebook AI Lab LeCun !  2014/1: Google DeepMind !  2014/5: Andrew Ng Baidu !  2014/8: IBM SyNAPSE
3. !  2014/1 !  2014/10 -‐‑‒ !  2015/06
4. t !  !  Sequence-to-sequence (2014, Google) !  !  Deep Q-Networks (2013, DeepMind) !  !  Variational Auto-Encoder (2014, UvA) !  Stochastic Backprop (2014, DeepMind) !  !  Neural Turing Machines (2014, DeepMind) !  !  Memory Networks (2014, Facebook) !  !  Show and Tell (2015, Google)
!  !  !  !  !  !  ! 
!  t !  !  !  1 !  ! 
Neural Network Language Model (NNLM) [Bengio +03] !  N NN !  N-1
Recurrent Neural Network Language Model (RNNLM) [Mikolov+10] !  t-1 t !  NNLM N !  !  http://rnnlm.org
Skip-gram (word2vec) [Mikolov+13] !  !  1CBOW !  Analogical reasoning !  1word2vec
(MV-RNN) [Socher +12] !  !  f !  Root
Sequence-to-sequence learning [Sutskever+14] !  ! 
Show and Tell [Vinyals+15] !  !  CNN RNN CNN CNN RNN
Memory networks [Weston+15][Sukhbaatar+15] !  ! 
!  !  !  1 !  End-to-end !  !  ! 
!  libsvm, liblinear !  !  JUMAN, Chasen, MeCab !  !  Moses (GIZA++) !  !  Stanford CoreNLP !  !  word2vec !  Skipgram !  Theano, Caffe, Torch !  t ! 
Chainer v1.0.0 2015/06/09 Powerful Flexible Intuitive
Chainer !  Chainer !  Python h(f(x) + g(x)) x f g h+
!  !  1 !  !  !  !  !  Chainer
企業における自然言語処理技術利用の最先端
t
PFI 2010 Xappy12012 !  2012 Gunosy !  !  !  t
Xappy t !  !  Blog !  Twitter Facebook SNS !  !  4G !  t !  !  etc. !  GUI t
!  !  !  !  t !  PC t !  !  !  !  10 ! 
企業における自然言語処理技術利用の最先端
(1/5) !  [Brown+93] Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, Robert L. Mercer. The mathematics of statistical machine translation: parameter estimation. Computational Linguistics Vol. 19 (2), 1993. !  [Berger+96] Adam L. Berger, Vincent J. Della Pietra, Stephen A. Della Pietra. A Maximum Entropy Approach to Natural Language Processing. Computational Linguistics, Vol. 22 (1), 1996. !  [Lafferty+01] John Lafferty, Andrew McCallum, Fernando C. N. Pereira. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. ICML2001.
(2/5) !  [Blei+03] David M. Blei, Andrew Y. Ng, Michael I. Jordan. Latent Dirichlet Allocation. JMLR Vol. 3, 2003. !  [Teh06] Yee Whye Teh. A Hierarchical Bayesian Language Model based on Pitman-Yor Processes. ACL 2006. !  [Clarke+06] James Clarke, Mirella Lapata. Constraint-Based Sentence Compression: An Integer Programming Approach. COLING/ACL 2006. !  [Riedel+06] Sebastian Riedel, James Clarke. Incremental Integer Linear Programming for Non-projective Dependency Parsing. COLING/ACL 2006.
(3/5) !  [Koo+10] Terry Koo, Alexander M. Rush, Michael Collins, Tommi Jaakkola, David Sontag. Dual Decomposition for Parsing with Non-Projective Head Automata. EMNLP 2010. !  [Rush+10] Alexander M. Rush, David Sontag, Michael Collins, Tommi Jaakkola. On Dual Decomposition and Linear Programming Relaxations for Natural Language Processing. EMNLP 2010. !  [Bengio+03] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin. A Neural Probabilistic Language Model. JMLR, 2003.
(4/5) !  [Mikolov+10] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan "Honza" Cernocky, Sanjeev Khudanpur. Recurrent neural network based language model. Interspeech, 2010. !  [Mikolov+13] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. CoRR, 2013. !  [Socher+12] Richard Socher, Brody Huval, Christopher D. Manning, Andrew Y. Ng. Semantic Compositionality through Recursive Matrix-Vector Spaces. EMNLP2012. !  [Sutskever+14] I. Sutskever, O. Vinyals, Q. V. Le. Sequence to Sequence Learning with Neural Networks. NIPS 2014.
(5/5) !  [Vinyals+15] O. Vinyals, A. Toshev, S. Bengio, D. Erhan. Show and Tell: A Neural Image Caption Generator. arXiv:1411.4555, 2014. !  [Weston+15] J. Weston, S. Chopra, A. Bordes. Memory Networks. ICLR 2015 !  [Sukhbaatar+15] S. Sukhbaatar, A. Szlam, J. Weston, R. Fergus. End-To-End Memory Networks. arXiv:1503.08895, 2015.

More Related Content

企業における自然言語処理技術利用の最先端