Introduction  | Machine Learning  | Google for Developers
Introduction Stay organized with collections Save and categorize content based on your preferences. Text classification algorithms are at the heart of a variety of software systems that process text data at scale. Email software uses text classification to determine whether incoming mail is sent to the inbox or filtered into the spam folder. Discussion forums use text classification to determine w
GitHub - saffsd/langid.py: Stand-alone language identification system
langid.py is a standalone Language Identification (LangID) tool. The design principles are as follows: Fast Pre-trained over a large number of languages (currently 97) Not sensitive to domain-specific features (e.g. HTML/XML markup) Single .py file with minimal dependencies Deployable as a web service All that is required to run langid.py is >= Python 2.7 and numpy. The main script langid/langid.p
æ ªå¼ä¼šç¤¾ãƒ¬ãƒˆãƒªãƒ
レトリãƒã¨ã¯ 自然言語処ç†ã‚’使ã£ãŸAI技術ã§ã€ 人ã®å‰µé€ çš„ãªä»•äº‹ã‚’支æ´ã™ã‚‹ レトリãƒã¯ã€è‡ªç„¶è¨€èªžå‡¦ç†AIを誰ã«ã§ã‚‚使ã„ã‚„ã™ã„ã‹ãŸã¡ã§æä¾›ã—ã€ æƒ…å ±ã®èª¿æŸ»ã€æ•´ç†ã€åˆ†æžã®åˆ†é‡Žã§å¤šãã®äººã‚’支æ´ã—ã¾ã™ã€‚ Our Solution AIソリューション㮠プãƒãƒ•ã‚§ãƒƒã‚·ãƒ§ãƒŠãƒ«ãŒ 課題解決ã«ç›´çµã™ã‚‹ ソリューションをã”æä¾› レトリãƒã¯ã€é¡§å®¢ãƒ•ã‚¡ãƒ¼ã‚¹ãƒˆãªãƒ†ãƒƒã‚¯ã‚«ãƒ³ãƒ‘ニーを目指ã—ã€é¡§å®¢ã«å¯„ã‚Šæ·»ã„ãªãŒã‚‰ä¸–ã®ä¸ã‚’より良ã便利ã«ã™ã‚‹ã‚½ãƒªãƒ¥ãƒ¼ã‚·ãƒ§ãƒ³ã‚’æä¾›ã—ã¾ã™ã€‚ ソリューションã®å…¨ä½“åƒã‚’知る
有効ãªWikiNameã§ã¯ã‚ã‚Šã¾ã›ã‚“ - LANGUAGE MEDIA PROCESSING LAB
[日本語] / [English] äº¬éƒ½å¤§å¦ å¤§å¦é™¢æƒ…å ±å¦ç ”究科 çŸ¥èƒ½æƒ…å ±å¦ã‚³ãƒ¼ã‚¹ 言語メディア分野（工å¦éƒ¨é›»æ°—é›»åå·¥å¦ç§‘） ç ”ç©¶å®¤ã¸ã®ã‚¢ã‚¯ã‚»ã‚¹ JUMAN ã¯æœ‰åŠ¹ãª WikiName ã§ã¯ã‚ã‚Šã¾ã›ã‚“。 サイトãƒãƒªã‚·ãƒ¼
Statistical Semantic入門 ~分布仮説ã‹ã‚‰word2vecã¾ã§~
1. The document discusses various statistical and neural network-based models for representing words and modeling semantics, including LSI, PLSI, LDA, word2vec, and neural network language models. 2. These models represent words based on their distributional properties and contexts using techniques like matrix factorization, probabilistic modeling, and neural networks to learn vector representatio
Word2Vec：発明ã—ãŸæœ¬äººã‚‚é©šãå˜èªžãƒ™ã‚¯ãƒˆãƒ«ã®é©šç•°çš„ãªåŠ›
Word2Vecã¨ã¯ Word2Vecã§æ¼”算処ç†ã™ã‚‹ Word2Vecã¨ãƒ‹ãƒ¥ãƒ¼ãƒ©ãƒ«ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ Word2Vecã®ä»•çµ„ã¿ CBoW Skip-gram Word2Vecを応用ã™ã‚‹ã“ã¨ãŒã§ãる分野 レコメンド 機械翻訳 Q&A・ãƒãƒ£ãƒƒãƒˆãƒœãƒƒãƒˆ æ„Ÿæƒ…åˆ†æž Word2Vecã®å¼±ç‚¹ Word2Vecã®æ´¾ç”Ÿç³»ã‚„類似ツール GloVe WordNet Doc2Vec fastText ã¾ã¨ã‚ å‚考 世界ä¸ã®Webサイトã®æ•°ã¯2014å¹´ã«10億件を超ãˆãŸã‚ˆã†ã 。ãã—ã¦ã€Facebookã®ãƒ¦ãƒ¼ã‚¶ãƒ¼æ•°ã ã‘ã§ã‚‚16億人を超ãˆã¦ã„る。 ãã—ã¦ã€ãã®ã„ãšã‚Œã‚‚コンテンツã®ä¸èº«ã®å¤§éƒ¨åˆ†ã¯ãƒ†ã‚ストã‹ã‚‰æˆã‚Šç«‹ã£ã¦ã„ã‚‹ã“ã¨ã ã‚ã†ã€‚ ã¨ã„ã†ã“ã¨ã¯ã€èŽ«å¤§ã«å¢—大ã—続ã‘ã‚‹ãƒãƒƒãƒˆä¸Šã®ãƒ‡ãƒ¼ã‚¿ã®ã»ã¨ã‚“ã©ã¯ã©ã“ã‹ã®å›½ã®è¨€è‘‰ã ã£ã¦ã“ã¨ã 。世界ä¸ã®äººãŒæ¯Žæ—¥ãƒ†ã‚ストデータを生æˆã—続ã‘ãŸã“ã¨ã¯ã“ã‚Œã¾ã§ã®æ´å²ä¸Šç„¡ã‹ã£ãŸã‚“ã˜ã‚ƒãªã„ã ã‚ã†ã‹ã€‚ ã‚‚ã—ãã‚“
word2vecã®ã‚½ãƒ¼ã‚¹ã‚’èªã‚“ã§ã¿ãŸ - Qiita
å˜èªžã®æ„味をベクトルã§è¡¨ç¾ã™ã‚‹æ‰‹æ³•ã§ã‚ã‚‹word2vec。検索ã™ã‚‹ã¨ã„ã‚ã‚“ãªæ–¹ã®è§£èª¬ãŒè¦‹ã¤ã‹ã‚Šã¾ã™ã€‚ãã®è§£èª¬ã¨ã‚½ãƒ¼ã‚¹ã‚³ãƒ¼ãƒ‰ã‚’見比ã¹ãªãŒã‚‰ã€è‡ªåˆ†ãªã‚Šã«å‹‰å¼·ã—ã¦ã¿ã¾ã—ãŸã€‚ 今回ã¯word2vecã®C#実装ã§ã‚ã‚‹Word2Vec.Netã®ã‚½ãƒ¼ã‚¹ã§å‹‰å¼·ã—ã¾ã—ãŸã€‚ãƒã‚¸ãƒƒã‚¯ã¯å…ƒã€…ã®C言語ã«ã‚ˆã‚‹å®Ÿè£…ã¨ã»ã¨ã‚“ã©åŒã˜ãªã®ã§ã€ã“ã®ã‚½ãƒ¼ã‚¹ã§å‹‰å¼·ã—ã¦ã‚‚å•é¡Œã‚ã‚Šã¾ã›ã‚“。ã¾ãŸã€ã“ã®æ–¹ãŒVisualStudioã®ãƒ‡ãƒãƒƒã‚¬ãŒä½¿ãˆã‚‹ã®ã§è¿½ã„ã‚„ã™ã„ã§ã™ã€‚ word2vecã«ã¯å¦ç¿’アルゴリズムã¨ã—ã¦ã€ŒC-BOWã€ã¨ã€ŒSkip-gramã€ã®ï¼’種類ã®æ‰‹æ³•ãŒç´¹ä»‹ã•ã‚Œã¦ã„ã¾ã™ãŒã€ä»Šå›žã¯ã€ŒSkip-gramã€ã«ã¤ã„ã¦å‹‰å¼·ã—ã¾ã—ãŸã€‚計算é‡ã‚’抑ãˆã‚‹ã‚„ã‚Šæ–¹ã¨ã—ã¦ã¯ã€ŒéšŽå±¤çš„ソフトマックスã€ã¨ã€ŒNegative Samplingã€ã®ï¼’種類ãŒword2vecã®ãƒ—ãƒã‚°ãƒ©ãƒ ä¸ã«å®Ÿè£…ã•ã‚Œã¦ã„ã¾ã™ãŒã€ä»Šå›žã¯ã€ŒNegative Samplingã€ã‚’勉強ã—ã¾ã—
言語処ç†100本ノック 2015
言語処ç†100本ノックã¯ï¼Œå®Ÿè·µçš„ãªèª²é¡Œã«å–り組ã¿ãªãŒã‚‰ï¼Œãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°ï¼Œãƒ‡ãƒ¼ã‚¿åˆ†æžï¼Œç ”究ã®ã‚¹ã‚ルを楽ã—ãç¿’å¾—ã™ã‚‹ã“ã¨ã‚’目指ã—ãŸå•é¡Œé›†ã§ã™ 実用的ã§ãƒ¯ã‚¯ãƒ¯ã‚¯ã™ã‚‹ã‚ˆã†ãªé¡Œæを厳é¸ã—ã¾ã—㟠言語処ç†ã«åŠ ãˆã¦ï¼Œçµ±è¨ˆã‚„機械å¦ç¿’ãªã©ã®å‘¨è¾ºåˆ†é‡Žã«ã‚‚親ã—ã‚ã¾ã™ ç ”ç©¶ã‚„ãƒ‡ãƒ¼ã‚¿åˆ†æžã®é€²ã‚方，作法，スã‚ルを修得ã§ãã¾ã™ å•é¡Œã‚’解ãã®ã«å¿…è¦ãªãƒ‡ãƒ¼ã‚¿ãƒ»ã‚³ãƒ¼ãƒ‘スをé…布ã—ã¦ã„ã¾ã™ 言語ã¯Pythonを想定ã—ã¦ã„ã¾ã™ãŒï¼Œä»–ã®è¨€èªžã«ã‚‚対応ã—ã¦ã„ã¾ã™
GitHub - tarowatanabe/expgram: expgram: an ngram toolkit with succinct storage
expgram is an ngram toolkit which can efficiently handle large ngram data: A succinct data structure for compactly represent ngram data [1]. Among ngram compression methods mentioned in [1], we do not implement block-wise compression (or zlib every 8k-byte) for computational efficiency reason. Language model is estimated by MapReduce proposed by [2] using pthread and/or MPI. Better rest cost estim
Kneser-Ney Smoothing を試ã—ã¦ã¿ãŸ - 木曜ä¸è¶³
Kneser-Ney Smoothing ã¯é«˜æ€§èƒ½ãªè¨€èªžãƒ¢ãƒ‡ãƒ«ã§ã‚る。ã¨ã€ã‚ˆãèžã‹ã•ã‚Œã¦çŸ¥ã£ã¦ã„ã‚‹ã¤ã‚‚ã‚Šã ã‘ã©ã€ã¾ã 一度も試ã—ãŸã“ã¨ãŒãªã‹ã£ãŸã®ã§ã€è©¦ã—ã¦ã¿ãŸã€‚ コードã¯ã“ã“。 https://github.com/shuyo/iir/blob/master/ngram/knsmooth.py 実験用ã«ã¹ã£ãŸã‚Šæ›¸ã„ã¦ã„ã‚‹ã®ã§ã‚³ãƒ”ペã¯å¤šã„ã—ã€é€Ÿåº¦ã‚„メモリã®åŠ¹çŽ‡ã‚‚悪ã„ãŒã€ã¾ã‚æ°—ã«ã—ãªã„ã§ã€‚ コーパスã¯æœ€åˆæ‰‹å…ƒã®é©å½“ãªãƒ‹ãƒ¥ãƒ¼ã‚¹ã‚³ãƒ¼ãƒ‘スを使ã£ã¦ã„ãŸã‚“ã ã‘ã©ã€ãã‚Œã ã¨å†ç¾æ¤œè¨¼ã§ããªã„ã®ã§ã€nltk ã®ã‚³ãƒ¼ãƒ‘スを使ã†ã‚ˆã†ã«å¤‰æ›´ã—ãŸã€‚ nltk.corpus ã®ä¸‹ã«ã‚るコーパスモジュールåã‚’ -c オプションã§ä¸Žãˆã‚‹ã¨ã€ãã®ã‚³ãƒ¼ãƒ‘スを使ã£ã¦ additive smoothing 㨠Kneser-Ney smoothing ã® perplexity を出ã—ã¦ãれる。 デフォルトã§ã¯ Brown コーパス
H24:Introduction to Statistical Topic Models
統計数ç†ç ”究所 H24年度公開講座 「確率的トピックモデルã€ã‚µãƒãƒ¼ãƒˆãƒšãƒ¼ã‚¸ 講師: æŒæ©‹å¤§åœ° (çµ±æ•°ç ”), 石黒å‹å½¦ (NTTコミュニケーション科å¦åŸºç¤Žç ”究所) 講義スライド æŒæ©‹åˆ† (2013/1/15) [PDF] (12MB) 石黒分 (2013/1/16) [PDF] ソフトウェア UM (Unigram Mixtures) um-0.1.tar.gz DM (Dirichlet Mixtures) dm-0.1.tar.gz, dm-0.2.tar.gz PLSI (Probabilistic Latent Semantic Indexing) plsi-0.03.tar.gz (外部サイト) LDA (Latent Dirichlet Allocation) lda-0.1.tar.gz å‚考文献 「ç§ã®ãƒ–ックマーク: Latent Topic Model (潜在的トピックモデ
Gensim: topic modelling for humans
✔   Train large-scale semantic NLP models ✔   Represent text as semantic vectors ✔   Find semantically related documents from gensim import corpora, models, similarities, downloader # Stream a training corpus directly from S3. corpus = corpora.MmCorpus("s3://path/to/corpus") # Train Latent Semantic Indexing with 200D vectors. lsi = models.LsiModel(corpus, num_topics=200) # Convert another corpus t
SimString - 高速ã‹ã¤ã‚·ãƒ³ãƒ—ルãªé¡žä¼¼æ–‡å—列検索ライブラリ
SimStringã¯ï¼Œé¡žä¼¼æ–‡å—列検索ã®ãŸã‚ã®é«˜é€Ÿã‹ã¤ã‚·ãƒ³ãƒ—ルãªãƒ©ã‚¤ãƒ–ラリã§ã™ï¼Ž 類似文å—列検索ã¨ã¯ï¼Œæ–‡å—列集åˆï¼ˆãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ï¼‰ã®ä¸ã‹ã‚‰ï¼Œã‚¯ã‚¨ãƒªæ–‡å—列ã¨ã®é¡žä¼¼åº¦ãŒé–¾å€¤ä»¥ä¸Šã®ã‚‚ã®ã‚’，見ã¤ã‘出ã™æ“作ã§ã™ï¼Ž クエリ文å—列ã¨å®Œå…¨ã«ä¸€è‡´ã—ãªãã¦ã‚‚，データベースä¸ã®ä¼¼ã¦ã„ã‚‹æ–‡å—列を検索ã™ã‚‹ã“ã¨ãŒã§ãã‚‹ã®ã§ï¼Œã‚¹ãƒšãƒ«è¨‚æ£ï¼Œã‚ã„ã¾ã„計算，柔軟ãªè¾žæ›¸ãƒžãƒƒãƒãƒ³ã‚°ï¼Œé‡è¤‡ãƒ¬ã‚³ãƒ¼ãƒ‰æ¤œå‡ºï¼Œãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹çµ±åˆãªã©ï¼Œæ§˜ã€…ãªã‚¢ãƒ—リケーションを構築ã§ãã¾ã™ï¼Ž SimStringã¯ï¼Œé¡žä¼¼åº¦é–¢æ•°ã¨ã—ã¦ï¼Œã‚³ã‚µã‚¤ãƒ³ä¿‚数，ジャッカード係数，ダイス係数，オーãƒãƒ¼ãƒ©ãƒƒãƒ—ä¿‚æ•°ã«å¯¾å¿œã—ã¦ã„ã¾ã™ï¼Ž æ–‡å—列ã®é¡žä¼¼åº¦ã‚’計算ã™ã‚‹ãŸã‚ã®ç‰¹å¾´é‡ã¨ã—ã¦ã¯ï¼Œæ–‡å—nグラムをサãƒãƒ¼ãƒˆã—ã¦ã„ã¾ã™ï¼Ž SimStringã«ã¯ï¼Œæ¬¡ã®ã‚ˆã†ãªç‰¹å¾´ãŒã‚ã‚Šã¾ã™ï¼Ž 高速ãªé¡žä¼¼æ–‡å—列検索アルゴリズム．Google Web1T ã®è‹±èªžå˜èªžï¼ˆ13,588,391æ–‡å—列）ã‹ã‚‰ï¼Œã‚³ã‚µã‚¤ãƒ³é¡žä¼¼åº¦ãŒ
ランã‚ング
メンテナンス
ãŠçŸ¥ã‚‰ã›
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
GitHub - piskvorky/gensim: Topic Modelling for Humans
GitHub - facebookresearch/fastText: Library for fast text representation and classification.
GitHub - alexandres/lexvec: This is an implementation of the LexVec word embedding model (similar to word2vec and GloVe) that achieves state of the art results in multiple NLP tasks
ã¯ã¦ãªãƒ–ックマーク全文検索ã®ç²¾åº¦æ”¹å–„
Google Code Archive - Long-term storage for Google Code Project Hosting.
GitHub - clips/pattern: Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
潜在的æ„味インデã‚シング（LSI）徹底入門 - ã‚らã³ã日記
{{#tags}}- {{label}}
{{/tags}}