[B! minhash] chezouã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

chezou id:chezou

minhashã«é–¢ã™ã‚‹chezouã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (6)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

ä¹±æŠžãƒ‡ãƒ¼ã‚¿æ§‹é€ ã®æœ€æ–°äº‹æƒ… âˆ’MinHash ã¨ HyperLogLog ã®æœ€è¿‘ã®é€²æ©âˆ’ - iwiwiã®æ—¥è¨˜
æœ¬æ—¥ï¼ŒPFI ã‚»ãƒŸãƒŠãƒ¼ã«ã¦ã€Œä¹±æŠžãƒ‡ãƒ¼ã‚¿æ§‹é€ ã®æœ€æ–°äº‹æƒ… âˆ’MinHash ã¨ HyperLogLog ã®æœ€è¿‘ã®é€²æ©âˆ’ã€ã¨ã„ã†ã‚¿ã‚¤ãƒˆãƒ«ã§è©±ã‚’ã•ã›ã¦ã‚‚ã‚‰ã„ã¾ã—ãŸï¼Žã‚¹ãƒ©ã‚¤ãƒ‰ã¯ä»¥ä¸‹ã«ãªã‚Šã¾ã™ï¼Ž Ustream ã®éŒ²ç”»ã‚‚ã‚ã‚Šã¾ã™ï¼Ž http://www.ustream.tv/recorded/48151077 å†…å®¹ã¨ã—ã¦ã¯ï¼Œä»¥ä¸‹ã®æ“ä½œã‚’åŠ¹çŽ‡çš„ã«è¡Œã†ãŸã‚ã®é›†åˆã«é–¢ã™ã‚‹ãƒ‡ãƒ¼ã‚¿æ§‹é€ (Sketch) ã®æœ€è¿‘ã®é€²æ©ã‚’ç´¹ä»‹ã—ã¾ã—ãŸï¼Ž é›†åˆã®é¡žä¼¼åº¦ã®æŽ¨å®š (Jaccard ä¿‚æ•°) é›†åˆç•°ãªã‚Šæ•°ã®æŽ¨å®š (distinct counting) ã©ã¡ã‚‰ã‚‚é‡è¦ã‹ã¤åŸºç¤Žçš„ãªæ“ä½œã§ï¼Œb-bit MinHash ã‚„ HyperLogLog ãªã©ï¼Œæ—¢ã«å®Ÿç”¨çš„ãªæ‰‹æ³•ãŒææ¡ˆã•ã‚Œã¦ãŠã‚Šï¼Œå®Ÿéš›ã«ã‚‚ä½¿ã‚ã‚Œã¦ã„ã¾ã™ï¼Žã—ã‹ã—ï¼Œ2014 å¹´ã«ãªã£ã¦ï¼ŒOdd Sketch ã‚„ HIP Estimator ã¨ã„ã†ï¼Œã“ã‚Œã‚‰ã‚’ã•ã‚‰ã«æ”¹å–„ã™ã‚‹æ‰‹æ³•ãŒç«‹ã¦ç¶š
chezou 2014/05/30
minhash

HyperLogLog

Algorithm
ãƒªãƒ³ã‚¯
æƒ…å ±ã‚·ã‚¹ãƒ†ãƒ åŸºç¤Žè«–ï¼’ ï¼š ç¬¬11å›ž(æ‹…å½“ï¼š å¤è³€) Jaccardä¿‚æ•°ã¨ Min Hash
chezou 2013/03/22
minhash

lsh
ãƒªãƒ³ã‚¯
ãã•ã‚‚ã¡ç ”ç©¶å®¤ç”Ÿæ´»ãƒ–ãƒã‚°ã ã£ãŸã‚‚ã® LSHã¾ã¨ã‚(1)
LSHã¯è¿‘ä¼¼æœ€è¿‘å‚æŽ¢ç´¢(Approximate Nearest Neighbor)ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ã®ä¸€ã¤ï¼Ž è¿‘ä¼¼æœ€è¿‘å‚æŽ¢ç´¢ã¨ã¯ï¼Œç°¡å˜ã«è¨€ã†ã¨ã‚¯ã‚¨ãƒªqã‹ã‚‰åŠå¾„(1+Îµ)å†…ã«ã‚ã‚‹ç‚¹vã‚’æŽ¢ç´¢ã™ã‚‹ã“ã¨ï¼Ž ã¤ã¾ã‚Šï¼ŒåŠå¾„(1+Îµ)ã®ç‚¹ã®ã†ã¡ï¼Œã©ã‚Œã‹1ã¤ã§ã‚‚æŽ¢ç´¢ã§ãã‚Œã°ãŠï½‹ï¼Ž è¨€è‘‰ã®æ„å‘³ãã®ã¾ã¾ã«æœ€è¿‘å‚æŽ¢ç´¢(Nearest Neighbor)ã®æ¡ä»¶ã‚’å°‘ã—ç·©ãã—ãŸæŽ¢ç´¢ã¨ã„ãˆã‚‹ï¼Ž ï¼ˆå®Ÿã¯ï¼Œç‰¹å¾´ãƒ™ã‚¯ãƒˆãƒ«ã®æ¬¡å…ƒãŒd=2ã®å ´åˆãªã‚‰ï¼ŒãƒœãƒãƒŽã‚¤å›³ã‚’ä½¿ãˆã°è¿‘ä¼¼æœ€è¿‘å‚æŽ¢ç´¢ãŒã§ãã‚‹ï¼‰ LSHã¯ãƒãƒƒã‚·ãƒ¥é–¢æ•°ã‚’ç”¨ã„ãŸç¢ºçŽ‡çš„æŽ¢ç´¢ã§è¿‘ä¼¼æœ€è¿‘å‚æŽ¢ç´¢ã‚’è§£ãï¼Ž ãã†ï¼Œå®Ÿã¯ãƒãƒƒã‚·ãƒ¥é–¢æ•°ã‚’ç”¨ã„ã‚‹ã¨ã„ã†ã“ã¨ä»¥ä¸Šã«ç¢ºçŽ‡çš„æŽ¢ç´¢ã¨ã„ã†ã“ã¨ã«å¤§ããªæ„å‘³ãŒã‚ã‚‹ï¼Žï¼ˆã“ã‚ŒãŒè‡ªåˆ†ã«ã¨ã£ã¦ã¯ã‹ãªã‚Šã‚„ã£ã‹ã„ãªå•é¡Œï¼‰ LSHã§ã¯ï¼Œã‚¯ã‚¨ãƒªqã¨è¿‘å‚(åŠå¾„(1+Îµ)ä»¥å†…)ã«ã‚ã‚‹ç‚¹ã§ã¯ãƒãƒƒã‚·ãƒ¥å€¤ãŒä¸€è‡´ã™ã‚‹ç¢ºçŽ‡ãŒé«˜ãï¼Œ ã‚¯ã‚¨ãƒªqã¨é ã„ä½ç½®ã«ã‚ã‚‹ç‚¹ã§ã¯ãƒãƒƒã‚·ãƒ¥å€¤ãŒä¸€è‡´ã™ã‚‹ç¢ºçŽ‡ãŒä½Ž
chezou 2013/03/11
lsh

minhash
ãƒªãƒ³ã‚¯
ã€ŽMPJoin ã‚’ä½¿ã£ãŸé¡žä¼¼ãƒ‡ãƒ¼ã‚¿æŠ½å‡ºã€€â€•ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ã‚·ãƒªãƒ¼ã‚º 1â€•ã€
Hattoriã€€ã§ã™ã€‚ä»¥å‰æ›¸ã„ãŸè¨˜äº‹ã®å†’é ã§ã€â€ä»Šåº¦ã¯ã‚·ãƒªãƒ¼ã‚ºã§ä½•ã‹ã‚¨ãƒ³ãƒˆãƒªã‚’æ›¸ããŸã„ ! â€ã¨è»½ã„ãƒŽãƒªã§ä¸€æ–‡ã‚’è¡¨è¨˜ã—ã¦ãŠã„ãŸã‚‰ã€ãƒ›ãƒ³ãƒˆã«ã‚„ã‚‹ã“ã¨ã«ãªã‚Šã¾ã—ãŸã€‚ å¼Šç¤¾ã®ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢çµ„ç¹”ã®ç‰¹å¾´ã®ã²ã¨ã¤ã«ã€æ‰‹ã‚’ä¸Šã’ã‚‹ãƒ»å£°ã‚’ä¸Šã’ã‚‹ã¨ã€ã€Žã˜ã‚ƒã€ã‚„ã£ã¦ã‚ˆã€‚ã€ã¨ã„ã†ãƒŽãƒªã§è¿”ã£ã¦ãã‚‹ã¨ã„ã†äº‹ãŒæŒ™ã’ã‚‰ã‚Œã‚‹ã®ã§ã™ãŒã€ä»Šå›žã‚‚ãã®ä¾‹ã«æ¼ã‚Œãªã£ãŸã‚ã‘ã§ã™ããƒ»ãƒ»ãƒ»ã€‚ã‚·ã‚¯ã‚·ã‚¯ãƒ»ãƒ»ãƒ»ã€‚ ã¨ã„ã†ã‚ã‘ã§ã€ä½•ã‚’æ›¸ã“ã†ã‹ãªãã£ã¦è©±ã—ãªã‚“ã§ã™ãŒãƒ»ãƒ»ãƒ»ã€‚ç§ã®å ´åˆã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ç³»ã®è©±ã—ã‹ã§ããã†ã«ãªã„ã®ã§ã€æ¯Žå›žãƒãƒ„ãƒãƒ„ã¨ãƒžã‚¤ãƒŠãƒ¼ã§æ¥µä¸€éƒ¨ã®äººã«ã—ã‹ã‚¦ã‚±ãªã„ãƒ†ãƒ¼ãƒžã‚’ç´¹ä»‹ã—ã¦è¡Œã“ã†ã¨æ€ã„ã¾ã™ã€‚ ã§ã€åˆå›žã®ä»Šå›žã¯ SimilarityJoin é–¢é€£ã®ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ã§ã€€"MPJoin" ã¨ã„ã†ã‚„ã¤ã‚’ç´¹ä»‹ã—ãŸã„ã¨æ€ã„ã¾ã™ã€‚ â– ã€€Similarity Join ã¨ã¯ä½•ãžã‚„ï¼Ÿ ã¾ãšæœ€åˆã« SimilarityJoin [1] ã®å®šç¾©ãªã‚“ã§ã™ãŒã€ã–ã£ãã‚Š
chezou 2012/10/18
minhash

mpjoin
ãƒªãƒ³ã‚¯
http://eigenjoy.com/2011/05/09/cascading-simhash-a-library-to-cluster-by-minhashes-in-hadoop/
chezou 2012/10/18
minhash

hadoop
ãƒªãƒ³ã‚¯
MinHashã‚’ç”¨ã„ãŸSketchSort - Yasuo Tabeiã®æ—¥è¨˜
MinHashã‚’ç”¨ã„ãŸSketchSortã®è«–æ–‡ãŒMolecular Informaticsã«æŽ¡æŠžã•ã‚Œã¾ã—ãŸã€‚ è«–æ–‡ã¯ä¸‹ã®ã‚µã‚¤ãƒˆã‹ã‚‰ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ Yasuo Tabei and Koji Tsuda: SketchSort: Fast All Pairs Similarity Search for Large Databases of Molecular Fingerprints, Molecular Informatics, 2011. Link ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã‚’google codeã«ã¦å…¬é–‹ã—ã¦ã„ã¾ã™ã€‚ æœ¬è«–æ–‡ã§ã¯ã€ä»¥å‰ç´¹ä»‹ã—ãŸCosineè·é›¢ã«åŸºã¥ãé«˜é€Ÿãªå…¨ç‚¹é–“é¡žä¼¼åº¦æ¤œç´¢æ³•(SketchSort)ã‚’Jaccard-Tanimotoè·é›¢ã«åŸºã¥ãæ‰‹æ³•ã«æ‹¡å¼µã—ã¾ã—ãŸã€‚ã“ã®ãŸã‚ã€ä»¥å‰ã¯ä¸Žãˆã‚‰ã‚ŒãŸä»»æ„ã®ï¼’ç‚¹é–“ã®Cosineè·é›¢ã‚’ãƒãƒŸãƒ³ã‚°è·é›¢ã§ä¿å˜ã—ãŸã¾ã¾ãƒã‚¤ãƒŠãƒªæ–‡å—åˆ—ã¸ã¨ãƒãƒƒ
chezou 2012/10/18
minhash
ãƒªãƒ³ã‚¯
1