An Introduction to Solr

Searching with Solr Tom Hill [email_address] eBig Java SIG, June 18th, 2008

Tonight's Talk Tonight's Talk should run about 1 1/2 hours About Solr Background & overview Installing & Bringing Up Solr Rest Interface & Java Client Configuring Solr

Why Implement Search? Does your site need search? Do you need to implement it, or is Google enough? Just text or Structured Data? Do you need to control ranking?

What is Solr? Web application for text search A wrapper around Apache Lucene Lucene is a library (.jar file) Solr is a web app (.war file) Written at CNet, now at Apache

What is Lucene? Text search library in Java Fast, feature rich. Written by Doug Cutting Active Apache development community Versions also in C++, C#, Ruby, Python, Delphi, Lisp, etc...

Why Solr? Reliable Fast Supported Open Source Tunable Scoring

Solr Versions Current Version is 1.2 A year old 1.3 is coming "sometime" Large number of features in HEAD Use the latest from subversion for new projects

Alternatives to Solr Just Use Google Use Lucene Use Your Database Commercial Libraries Write your own

What Solr is Not A replacement for a relational database An embedded database* Fully cross platform :-( Replication depends on unix FS Admin scripts are bash(minor)

Solr Sites CNet (Reviews & Products) Internet Archive (Collections) Netflix (Movies) Zvents (Events) StripSearch.ws (Comics) And many more

Features Here's a quick look at some of the features of Solr, as implemented on Zvents.com

Faceted Navigation Groups the results by category Can do multiple facets at once Returns matching counts

Simple Webapp Web Servers[1..n] Database Master Database Slaves[0..n] Solr Master Solr Slaves[0..n]

Scaling Solr Master/Slave architecture Writes to master/reads to slaves Replication: Periodic transfers, not continuous Rsync

Updates Updates flush caches, bad for performance Master therefor much slower than slaves So send all queries to slaves Depends on your update rates

Solr's Data Model Solr maintains a collection of documents A document is a collection of fields & values A field can occur multiple times in a document Documents are immutable. They can be deleted, and a new version added, however.

Querying Http request http://localhost:8080/comix/select/?q=java

Solr Query Syntax Lucene Query Syntax + a bit paris city:paris title:"The Right Way" AND text:go id:[* TO *]

Solr Query Syntax II -inStock:false te?t theat* te*t test~

Using Solr Getting data into Solr Getting data out of Solr

Getting Data Into Solr POST it. <add> <doc> <field name="employeeId">05991</field> <field name="office">Bridgewater</field> <field name="skills">Perl</field> <field name="skills">Java</field> </doc> [<doc> ... </doc>[<doc> ... </doc>]] </add>

Getting Data Into Solr POST it. <add> < doc > <field name="employeeId">05991</field> <field name="office">Bridgewater</field> <field name="skills">Perl</field> <field name="skills">Java</field> </ doc > [<doc> ... </doc>[<doc> ... </doc>]] </add>

Getting Data Into Solr POST it. <add> <doc> <field name=" employeeId "> 05991 </field> <field name="office">Bridgewater</field> <field name="skills">Perl</field> <field name="skills">Java</field> </doc> [<doc> ... </doc>[<doc> ... </doc>]] </add>

Committing Nothing shows up in the index until you commit You can just POST <commit/> to http:// host : port /solr/update

Getting Data Out http://localhost:8080/comix/select/?q=data&indent=on <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">0</int> <lst name="params"> <str name="indent">on</str> <str name="q">data</str> </lst> </lst> <result name="response" numFound="2" start="0"> <doc> <str name="id">strip.3136</str> <str name="release_date">1992-05-07</str> <date name="timestamp">2008-02-28T10:06:01.682Z</date> <str name="type">strip</str> </doc> </result> </response>

Getting Data Out http://localhost:8080/comix/select/?q=data&indent=on <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">0</int> <lst name="params"> <str name="indent">on</str> <str name="q">data</str> </lst> </lst> <result name="response" numFound="2" start="0"> <doc> <str name="id">strip.3136</str> <str name="release_date">1992-05-07</str> <date name="timestamp">2008-02-28T10:06:01.682Z</date> <str name="type">strip</str> </doc> ... </result> </response>

Getting Data Out http://localhost:8080/comix/select/?q=data&indent=on { "responseHeader":{ "status":0, "QTime":1, "params":{ "wt":"json", "rows":["1", "1"], "start":"0", "indent":"on", "q":"data", "version":"2.2"}}, "response":{"numFound":2,"start":0,"docs":[ { "feature_id":"3", "release_date":"1992-05-07", "id":"strip.3136", "timestamp":"2008-02-28T10:06:01.682Z"}] }} JSON format

Debug Query Option Add &debugQuery=on to request params Returns parsed form of query <str name="rawquerystring">c.i.a</str><str name="querystring">c.i.a</str><str name="parsedquery">PhraseQuery(text:"c i a")</str><str name="parsedquery_toString">text:"c i a"</str>

Debug Query Option II Add &debugQuery=on to request params Returns scoring information <str name="id=strip.2781,internal_docid=29854"> 2.6219895 = (MATCH) fieldWeight(text:calvin in 29854), product of: 1.0 = tf(termFreq(text:calvin)=1) 2.6219895 = idf(docFreq=6222) 1.0 = fieldNorm(field=text, doc=29854) </str> <str name="id=strip.4078,internal_docid=31151"> 2.6219895 = (MATCH) fieldWeight(text:calvin in 31151), product of: 1.0 = tf(termFreq(text:calvin)=1) 2.6219895 = idf(docFreq=6222) 1.0 = fieldNorm(field=text, doc=31151) </str>

Deleting Data POST <delete><id>35</id></delete> <delete><query>city:paris</query></delete>

Command Line Control curl http://localhost:8983/solr/update -H "Content-type: text/xml" --data-binary '<commit/>' <?xml version="1.0" encoding="UTF-8"?><response><lst name="responseHeader"> <int name="status">0</int> <int name="QTime">20</int> </lst></response> </lst></response> </lst></response>

Solr in 3 minutes! Download Solr from Apache Untar "ant example" Start the example app Load data into Solr Query

Solr in Ten Minutes <Context docBase="/var/solr/apache-solr-1.2.0.war" debug="0" crossContext="true" > <Environment name="solr/home" type="java.lang.String" value="/var/solr" override="true" /></Context> Copy Solr's example/solr dir to /var/solr Edit schema.xml and solrconfig.xml Load data into Solr In $CATALINA_HOME/conf/Catalina/localhost/foo.xml

Directory Layout ${solr.home}/conf schema.xml solrconfig.xml ${solr.home}/data ${solr.home}/logs ${solr.home}/bin

Java Solr Client Called SolrJ Not in Solr 1.2. I grabbed from the HEAD from svn Works with Solr 1.2 Add/Delete/Query/Commit/Optimize

Adding Docs w/SolrJ Given Map<String, String> fields; CommonsHttpSolrServer server = new CommonsHttpSolrServer( url ); SolrInputDocument doc= new SolrInputDocument(); for (Map.Entry<String, String> e : fields.entrySet()){ doc.addField(e.getKey(), e.getValue()); } UpdateResponse res = server .add( doc);

Deleting Docs w/SolrJ CommonsHttpSolrServer server = new CommonsHttpSolrServer( url ); UpdateResponse res; res = server .deleteById("100"); res = server .deleteByQuery("city:paris");

Simple Query CommonsHttpSolrServer server = new CommonsHttpSolrServer( url ); SolrQuery query = new SolrQuery(); query.setQuery("dance"); QueryResponse rsp = server .query(query);

More Interesting Query CommonsHttpSolrServer server = new CommonsHttpSolrServer( url ); SolrQuery query = new SolrQuery(); query.setQuery("dance"); query.setFacet( true ); query.addFacetField("city"); query.setFacetMinCount(1); query.addSortField( "price", SolrQuery.ORDER.asc ); QueryResponse rsp = server .query(query);

Query Responses QueryResponse qr = server .query(query); SolrDocumentList docs = qr.getResults(); List<FacetField> lf = qr.getFacetFields(); for (FacetField ff: lf) { String fieldName = ff.getName(); List<FacetField.Count> lc = ff.getValues(); for (FacetField.Count c: lc) { String countName = c.getName(); long count = c.getCount(); } }

Other Commands Commit server.commit() Optimize server.optimize() Not too complicated!

Request Handlers Request handler define how the query is processed. Two main types StandardRequestHandler DisMaxRequestHandler You can implement your own Changing in Solr 1.3

"Standard" Request Handler Accepts Solr Query Syntax I tend to use it for my queries, not user queries.

DisMaxRequestHandler Recommended for user queries Allows simple users keywords to be applied to multiple fields, with weighting. Boost Functions Boost Queries

Boost Functions Allow you to influence scoring at run time Computationally Expensive! Really useful for tuning scoring linear(x,2,4) returns 2*x+4 x is a field

The Solr Schema schema.xml Defines types used in this webapp Defines the fields and their types Defines "copyFields" READ THE EXAMPLE SCHEMA.XML

Types Types define processing for a field How the words are split (Whitespace? Punctuation? CIA != C.I.A.) Stemming Case Folding, etc Predefined date, int, float, etc c

Analysis: Index and Query Time Types have two modes Index Time Query Time

Simple Text Field <fieldType name="text" class="solr.TextField" positionIncrementGap= "100"> <analyzer type="index"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/></analyzer><analyzer type="query"><tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/></analyzer></fieldType> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/></analyzer></fieldType> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/></analyzer></fieldType>

Analysis & Facets Make sure to use an untokenized field for faceting. "San Jose" != "San" "Jose"

Fields Elements of a document Both predefined & dynamic Fields may occur multiple times Maybe indexed and/or stored

Example Fields <field name="id" type="string" indexed="true" stored="true" required="true" /><field name="name" type="text" indexed="true" stored="true"/><field name="alphaNameSort" type="alphaOnlySort" indexed="true" stored="false"/>

Copy Fields Two main uses To analyze a field in two different ways To concatenate fields

The Solr Config File solrconfig.xml Defines request handlers, defaults, caches, Read the example solrconfig.xml

Configuring DisMax Parameter defaults set in solrconfig.xml Can be overridden in each request Except for params labeled invariant

DisMax Config Example <requestHandler name="dismax" class="solr.DisMaxRequestHandler" > <lst name="defaults"> <str name="echoParams">explicit</str> <float name="tie">0.01</float> <str name="qf"> text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4 </str> <str name="pf"> text^0.2 features^1.1 name^1.5 manu^1.4 manu_exact^1.9 </str> ... </requestHandler>

DisMax Config Example <requestHandler name="dismax" class="solr.DisMaxRequestHandler" > ... <str name="bf"> ord(poplarity)^0.5 recip(rord(price),1,1000,1000)^0.3 </str> <str name="fl"> id,name,price,score </str>... </requestHandler>

DisMax Config Example <requestHandler name="dismax" class="solr.DisMaxRequestHandler" > ... <str name="mm"> 2<-1 5<-2 6<90% </str> <int name="ps">100</int> <str name="q.alt">*:*</str> </lst> </requestHandler>

Resources Solr http://lucene.apache.org/solr/ wiki, mailing list, jira (bugs/features) Lucene http://lucene.apache.org /

Building Search Applications with Lucene, lingpipe and Gate Manu Konchady Manu Konchady Manu Konchady

Other Presentations Yonik Seely's Solr & Lucene http://people.apache.org/~yonik/presentations/ Slideshare.net Search for solr, or search for lucene

Thanks! Thanks for coming. Feel free to email me if you have questions about Solr Tom Hill [email_address]

Extra Slides Things I didn't have time for in the presentation. Some of them unfinished.

Search Engines are not the Same as Users Search engines have different usage patterns than users

Response Writers http://localhost:8983/solr/select/?q=text_t%3Atiger&version=2.2&start=0&rows=10&indent=on& wt=ruby http://localhost:8983/solr/select/?q=text_t%3Atiger&version=2.2&start=0&rows=10&indent=on& wt=xml

Explain Just why did the documents come up in that order?

Data Matters Gigo The better the data is, the better the search will be.

Watch Your Caches Just like any other app, check your statistics What's the hit rate for your caches?

Setting Up Replication Run rsyncd on the master Run snapshot on the master at intervals Run snappuller on the slaves at (different) intervals. Scripts don't print errors! Check the logs Use bash -xv

Autowarming Runs after an update to the index Updates flush caches Runs some queries to populate caches again Can be a problem, with frequent updates Don't autowarm master, if updating lots

Programming Collective Intelligence A Really Fun Book

Geographic Searching Local Lucene & Local Solr http://locallucene.wiki.sourceforge.net There's also geolucene, but it's not being actively developed, as far as I can tell. http://www.gossamer-threads.com/lists/l ucene/java-dev/53378

http://localhost:8983/solr/admin/stats.jsp#update Are there commits pending?

http://localhost:8983/comix/admin/analysis.jsp?name=text&val=wi-fi Analysis Explanation

An Introduction to Solr

More Related Content

An Introduction to Solr