[B! scraping] libkazzã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

libkazz id:libkazz

scrapingã«é–¢ã™ã‚‹libkazzã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (7)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

ã‚µãƒ¼ãƒã‚µã‚¤ãƒ‰JavaScriptã¨jQueryã§ã‚¹ã‚¯ãƒ¬ã‚¤ãƒ”ãƒ³ã‚°
jQuery ã§ã‚¹ã‚¯ãƒ¬ã‚¤ãƒ”ãƒ³ã‚°ã§ããŸã‚‰ã‚»ãƒ¬ã‚¯ã‚¿ä½¿ãˆã‚‹ã—ä¾¿åˆ©ã‹ãªãƒ¼ã¨æ€ã£ãŸã‚Šã—ãŸã‚“ã§ã™ãŒã€Rhino ã¨ env-js ã‚’ä½¿ã†ã¨è¶…ç°¡å˜ã«ã§ããŸã®ã§ãƒ¬ãƒãƒ¼ãƒˆã—ã¦ã¿ã¾ã™ã€‚ Rhino ã¨ env-js ã£ã¦ä½•ãžã„ï¼Ÿ Rhino ã£ã¦ã®ã¯ Java ã§æ›¸ã‹ã‚ŒãŸ JavaScript ã‚¨ãƒ³ã‚¸ãƒ³ã§ã™ã€‚Rhino ã‚’ä½¿ã†ã¨ã‚³ãƒžãƒ³ãƒ‰ãƒ©ã‚¤ãƒ³ã‹ã‚‰ JavaScript ã‚’å®Ÿè¡Œã§ãã¾ã™ã€‚ Rhino ã«ã¯ DOM ãŒç„¡ã„ã®ã§å˜ä½“ã§ã¯ jQuery ã‚’å®Ÿè¡Œã™ã‚‹ã“ã¨ã¯ã§ãã¾ã›ã‚“ãŒã€env-js ã¨ã„ã† DOM ãƒ©ã‚¤ãƒ–ãƒ©ãƒªã‚’ä½¿ç”¨ã™ã‚‹ã¨ã“ã®è¾ºã¯ã‚¯ãƒªã‚¢ã§ãã¾ã™ã€‚ env-js ã®ä¸ã« jQuery ã‚’ä½¿ã£ãŸã‚µãƒ³ãƒ—ãƒ«ãŒã„ãã¤ã‹å…¥ã£ã¦ã¾ã™ã®ã§ã€Java ã¨ã‹ JavaScript ã¨ã‹è©³ã—ã„æ–¹ã¯ãã¡ã‚‰ã‚’è¦‹ã¦ã‚‚ã‚‰ã£ãŸæ–¹ãŒæ—©ã„ã¨æ€ã„ã¾ã™ã€‚ ä½¿ã£ã¦ã¿ã‚‹ ã¾ãš env-js ã‚’ãƒ“ãƒ«ãƒ‰ã—ãªã„ã¨ã„ã‘ãªã„ã®ã§ãã®è¾ºã®æ‰‹é †ã‹ã‚‰èª¬æ˜Žã—ã¾ã™ã€‚
libkazz 2011/01/10
javascript

jquery

rhino

scraping
ãƒªãƒ³ã‚¯
æ—¥æœ¬èªžç‰ˆ : IBM Bluemix
IBM Developer is your one-stop location for getting hands-on training and learning in-demand skills on relevant techno logies such as generative AI, data science, AI, and open source.
libkazz 2008/04/24
ã‚¯ãƒãƒ¼ãƒ©ãƒ¼ã‚µãƒ³ãƒ—ãƒ«

ruby

python

scraping
ãƒªãƒ³ã‚¯
Webãƒšãƒ¼ã‚¸ã®æœ¬æ–‡æŠ½å‡º (nakatani @ cybozu labs)
Webãƒšãƒ¼ã‚¸ã®è‡ªå‹•ã‚«ãƒ†ã‚´ãƒ©ã‚¤ã‚º ã®ç¶šãã€‚ å‰å›žæ›¸ã„ãŸã¨ãŠã‚Šã€ãƒ‘ã‚¹ãƒˆãƒ©ãƒƒã‚¯ã§è¡Œã£ã¦ã„ã‚‹ Web ãƒšãƒ¼ã‚¸ã®ã‚«ãƒ†ã‚´ãƒ©ã‚¤ã‚ºã§ã¯ã€Web ãƒšãƒ¼ã‚¸ã®æœ¬æ–‡æŠ½å‡ºãŒã²ã¨ã¤ã®éµã«ãªã£ã¦ã„ã¾ã™ã€‚ä»Šå›žã¯ãã®æœ¬æ–‡æŠ½å‡ºãƒ¢ã‚¸ãƒ¥ãƒ¼ãƒ«ã‚’å…¬é–‹ã—ã¤ã¤ã€ä½¿ã£ã¦ã„ã‚‹æŠ€æ³•ã‚’ã–ã£ãã‚Šè§£èª¬ãªã©ã—ã¦ã¿ã¾ã™ã€‚ æœ¬ãƒ¢ã‚¸ãƒ¥ãƒ¼ãƒ«ã®åˆ©ç”¨ã¯è‡³æ¥µç°¡å˜ã€‚require ã—ã¦ analyse ãƒ¡ã‚½ãƒƒãƒ‰ã«è§£æžã—ãŸã„ html ã‚’ä¸Žãˆã‚‹ã ã‘ã€‚æ–‡å—ã‚³ãƒ¼ãƒ‰ã¯ UTF-8 ã§ã™ã€‚ ã€è¿½è¨˜ã€‘å¤§äº‹ãªã“ã¨æ›¸ãå¿˜ã‚Œã€‚æœ¬ãƒ¢ã‚¸ãƒ¥ãƒ¼ãƒ«ã¯ Ruby1.8.5 ã§å‹•ä½œç¢ºèªã—ã¦ã„ã¾ã™ãŒã€ç‰¹åˆ¥ãªã“ã¨ã¯ã—ã¦ã„ãªã„ã®ã§ã€1.8.x ãªã‚‰å‹•ãã¨æ€ã„ã¾ã™ã€‚ $KCODE="u" # æ–‡å—ã‚³ãƒ¼ãƒ‰ã¯ utf-8 require 'extractcontent.rb' # ã‚ªãƒ—ã‚·ãƒ§ãƒ³å€¤ã®æŒ‡å®š opt = {:waste_expressions => /ãŠå•ã„åˆã‚ã›|ä¼šç¤¾æ¦‚è¦/} ExtractCont
libkazz 2007/10/31
ruby

scraping
ãƒªãƒ³ã‚¯
libxml2ã§ã‚¹ã‚¯ãƒ¬ã‚¤ãƒ”ãƒ³ã‚° - ã‚¹ã‚³ãƒˆãƒ—ãƒªã‚´ãƒ‹ã‚¨ãƒ•ã‚¹ã‚¯é€šä¿¡
â– [Python]libxml2ã§ã‚¹ã‚¯ãƒ¬ã‚¤ãƒ”ãƒ³ã‚° å·oãƒ»-ãƒ»ï¼‰ï¼œ2nd life - ruby ã®ã‚¹ã‚¯ãƒ¬ã‚¤ãƒ”ãƒ³ã‚°ãƒ„ãƒ¼ãƒ«ã‚ãƒƒãƒˆ scrAPIã§ã€ç´¹ä»‹ã•ã‚Œã¦ã„ã‚‹scrAPIã¨ã„ã†ãƒ„ãƒ¼ãƒ«ã‚’çŸ¥ã‚Šã¾ã—ãŸã€‚ CSSã‚»ãƒ¬ã‚¯ã‚¿ã§è¦ç´ ã‚’å–å¾—ã™ã‚‹ã¨ã„ã†ã‚¢ã‚¤ãƒ‡ã‚£ã‚¢ã¯é¢ç™½ã„ã¨æ€ã†ã®ã§ã™ãŒã€ã‚„ã£ã±ã‚ŠXPathã‚’ä½¿ã£ãŸæ–¹ãŒæ‰‹ã£å–ã‚Šæ—©ã„ã—ã€ã‚ã¨ã‚ã¨å¿œç”¨ãŒåˆ©ããã†ãªæ°—ã‚‚ã—ã¾ã™ã€‚ è©¦ã—ã«ã€Pythonã¨libxml2ã‚’ä½¿ã£ã¦æ›¸ã„ã¦ã¿ã¾ã™ã€‚libxml2ã®HTMLãƒ‘ãƒ¼ã‚µãƒ¼ã¯ã€ãƒ–ãƒãƒ¼ã‚¯ãƒ³ãªHTMLã‚‚è§£æžã—ã¦ãã‚Œã‚‹ã—ã€ã‚¨ãƒ³ã‚³ãƒ¼ãƒ‡ã‚£ãƒ³ã‚°ã‚‚ä¸Šæ‰‹ãæ‰±ã£ã¦ãã‚Œã‚‹ã®ã§éžå¸¸ã«ä¾¿åˆ©ã§ã™ã€‚ ä¾‹ãˆã°ã€ã™ã¹ã¦ã®ãƒªãƒ³ã‚¯ã‚’å–å¾—ã—ãŸã„å ´åˆã¯ã“ã‚“ãªæ„Ÿã˜ã§ã™ã€‚ import libxml2 doc = libxml2.htmlReadFile( 'http://www.hatena.ne.jp/', # url None, # encoding libxml
libkazz 2007/05/28
XPathï¼žScrAPIã¨ã®ã“ã¨ã§ã™ãŒã€‚ã€‚ã€‚

python

scraping
ãƒªãƒ³ã‚¯
pylori*style wiki - HTMLãƒ‘ãƒ¼ã‚µ Hpricot
Hpricotã¯ Rubyç”¨ã® HTML ãƒ‘ãƒ¼ã‚µã§ã™ã€‚ã‚¹ã‚ãƒ£ãƒ³éƒ¨åˆ†ã¯ C è¨€èªžã§æ›¸ã‹ã‚Œã¦ã„ã¦é«˜é€Ÿã«å‹•ä½œã—ã¾ã™ã€‚ HTMLã®è§£æžã‚„æ›¸ãæ›ãˆã«å¨åŠ›ã‚’ç™ºæ®ã™ã‚‹ä¾¿åˆ©ãªãƒ©ã‚¤ãƒ–ãƒ©ãƒªã§ã™ã€‚ rubygemsã‚’ä½¿ã†ã®ãŒç°¡å˜ã§ã™ã€‚ # gem install hpricot ã™ã‚‹ã¨ä»¥ä¸‹ã®ã‚ˆã†ã«ãƒ—ãƒ©ãƒƒãƒˆãƒ•ã‚©ãƒ¼ãƒ ã‚’èžã„ã¦ãã¾ã™ã€‚Unixç³»OSã§ã‚ã‚Œã° 1 ã‚’ã€mswin32 ã§ã‚ã‚Œã° 2 ã‚’é¸ã³ã¾ã™ã€‚ Select which gem to install for your platform (i386-freebsd6) 1. hpricot 0.4 (ruby) 2. hpricot 0.4 (mswin32) 3. Cancel installation ã‚³ãƒ³ãƒ‘ã‚¤ãƒ«ãŒå§‹ã¾ã‚Šã€ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ãŒè¡Œã‚ã‚Œã¾ã™ã€‚ Building native extensions. This could take a while.
libkazz 2007/04/09
æ›¸ãæ›ãˆå¯èƒ½ï¼Ÿãªã‚‰ä½¿ã†

scraping

Ruby
ãƒªãƒ³ã‚¯
kizasiãŒè§£æž - CNET Japan
ä¾¡æ ¼æ¯”è¼ƒã‚µã‚¤ãƒˆã®ä¾¡æ ¼.comã®æŽ²ç¤ºæ¿ã§ã¯ã€æ—¥ã€…è£½å“ã«å¯¾ã™ã‚‹ç–‘å•ã‚„æ„Ÿæƒ³ãªã©ãŒæ›¸ãè¾¼ã¾ã‚Œã¦ã„ã‚‹ã€‚ã“ã®æ›¸ãè¾¼ã¿ã®ä¸ã§ä»Šæœ€ã‚‚è©±é¡Œã«ãªã£ã¦ã„ã‚‹ã‚ãƒ¼ãƒ¯ãƒ¼ãƒ‰ãŒä½•ãªã®ã‹ã‚’ä¸€è¦§è¡¨ç¤ºã™ã‚‹ã‚µãƒ¼ãƒ“ã‚¹ãŒ11æœˆ28æ—¥ã«å§‹ã¾ã‚‹ã€‚ ã“ã‚Œã¯ã‚«ã‚«ã‚¯ã‚³ãƒ ãŒã€ãƒ–ãƒã‚°ä¸Šã§è©±é¡Œã®è¨€è‘‰ã‚’ç´¹ä»‹ã™ã‚‹kizasi.jpã‚’é‹å–¶ã™ã‚‹ã‚·ãƒ¼ã‚¨ãƒ¼ã‚·ãƒ¼ã¨ææºã—ã¦å®Ÿç¾ã™ã‚‹ã‚‚ã®ã€‚kizasiã§åˆ©ç”¨ã—ã¦ã„ã‚‹è¨€èªžè§£æžã‚¨ãƒ³ã‚¸ãƒ³ã‚’åˆ©ç”¨ã—ã¦ä¾¡æ ¼.comã®æŽ²ç¤ºæ¿ã«æ›¸ã‹ã‚Œã¦ã„ã‚‹æ–‡ç« ã‚’è§£æžã—ã€å‡ºç¾é »åº¦ãªã©ã‚’å…ƒã«è©±é¡Œã®ã‚ãƒ¼ãƒ¯ãƒ¼ãƒ‰ã‚’é¸ã³å‡ºã—ã¦è¡¨ç¤ºã™ã‚‹ã€‚ ã“ã®ã‚ãƒ¼ãƒ¯ãƒ¼ãƒ‰ãŒè¡¨ç¤ºã•ã‚Œã‚‹ã®ã¯æºå¸¯é›»è©±ã‚«ãƒ†ã‚´ãƒªã¨ãªã‚‹ã€‚ã€Œæ–°æ©Ÿç¨®ç™ºå£²ã‚„MNPï¼ˆç•ªå·ãƒãƒ¼ã‚¿ãƒ“ãƒªãƒ†ã‚£ï¼‰ãªã©ã§ãƒ¦ãƒ¼ã‚¶ãƒ¼ã‹ã‚‰ã®æ³¨ç›®åº¦ãŒé«˜ã„ã€ï¼ˆä¸¡ç¤¾ï¼‰ãŸã‚ã¨ã„ã†ã€‚ã€Œå„æºå¸¯ç«¯æœ«ã«ã¤ã„ã¦ä¾¡æ ¼.comã®ãƒ¦ãƒ¼ã‚¶ãƒ¼é–“ã§ä»Šã€ä½•ãŒèªžã‚‰ã‚Œã¦ã„ã‚‹ã®ã‹ã‚’è¦–è¦šçš„ã«ä¼ãˆã‚‹ã“ã¨ã§ã€è†¨å¤§ãªæ›¸ãè¾¼ã¿ã®ä¸ã‹ã‚‰ä¸€ç›®ã§æ—¬ã®ã‚ãƒ¼ãƒ¯ãƒ¼ãƒ‰ã‚„ã€Œç”Ÿã®å£°ã€ã‚’çŸ¥ã‚‹äº‹ãŒå¯èƒ½ã¨ãªã‚Šã€æŽ²ç¤ºæ¿ã®
libkazz 2006/12/21
kizasi

scraping
ãƒªãƒ³ã‚¯
ruby ã®ã‚¹ã‚¯ãƒ¬ã‚¤ãƒ”ãƒ³ã‚°ãƒ„ãƒ¼ãƒ«ã‚ãƒƒãƒˆ scrAPI - å·oãƒ»-ãƒ»ï¼‰ï¼œ2nd life
http://blog.labnotes.org/category/scrapi/ ruby ã§ã‚¹ã‚¯ãƒ¬ã‚¤ãƒ”ãƒ³ã‚°ã—ã¦ web ã®æƒ…å ±ã‚’å–å¾—ã™ã‚‹ã®ã«ã¯ã€ä»Šã¾ã§æ£è¦è¡¨ç¾ã‹ xpath ã§ã‚„ã£ã¦ãŸã®ã§ã€ã‚ã‚Šã¨é¢å€’ã§ã—ãŸã€‚ã§ã€ãµã¨ scrAPI ã¨ã„ã†ã‚¹ã‚¯ãƒ¬ã‚¤ãƒ”ãƒ³ã‚°ãƒ„ãƒ¼ãƒ«ã‚ãƒƒãƒˆã‚’çŸ¥ã£ãŸã®ã§ã™ãŒã€ã“ã‚ŒãŒã‹ãªã‚Šä¾¿åˆ©ãã†ã€‚ ã“ã®ãƒ„ãƒ¼ãƒ«ã‚ãƒƒãƒˆã‚’ä½¿ã†ã¨ã€CSS3 ãªã‚»ãƒ¬ã‚¯ã‚¿ã‚’è¨˜è¿°ã™ã‚‹ã“ã¨ã§ã€è¦ç´ ã‚’å–å¾—ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ãŸã¨ãˆã°ã¨ã‚ã‚‹ã‚µã‚¤ãƒˆã®ãƒªãƒ³ã‚¯ã‚’å…¨éƒ¨å–å¾—ã—ãŸã‘ã‚Œã°ã€ require 'rubygems' require 'scrapi' require 'open-uri' require 'nkf' require 'pp' $KCODE = 'u' links = Scraper.define do process "a[href]", "urls[]"=>"@href" result :urls e
libkazz 2006/12/04
ã“ã‚Œã ã‘ã˜ã‚ƒåˆ†ã‹ã‚‰ã‚“ã€‚æ·»ä»˜ã®RDoc(è‹±)ã‚’èªã‚€å¿…è¦ãŒã‚ã‚Šãã†

scraping

Ruby
ãƒªãƒ³ã‚¯
1