Lucandra
Download as PPT, PDF25 likes3,339 views
Lucandra is a project that combines Lucene/Solr indexing and search capabilities with Cassandra for storage. It addresses issues with scaling Lucene/Solr by leveraging Cassandra's ability to horizontally scale writes and queries. Lucandra stores Lucene indexes and term vectors in Cassandra to allow distributed indexing and searching across nodes. It has been used successfully in production systems like sparse.ly for twitter search and Wikassandra for Wikipedia search.
1 of 17
Downloaded 247 times
![How is an index stored? { "Lucandra" : { "Docs" : { "Index1/Doc1" : { "Field1" : "T1 T2 T1", ... }, { "Index1/Doc2" : { "Field1" : "T3 T1", ... } }, {"TermVectors" : {"Index1/Field1/T1" : { "Doc1": [0, 2], "Doc2":[1] }, {"Index1/Field1/T2" : { "Doc1": [1] }, {"Index1/Field1/T3" : { "Doc2": [1] }, } }](https://image.slidesharecdn.com/lucandra-100427220402-phpapp02/85/Lucandra-13-320.jpg)
Recommended
Lucene basics
Lucene basicsNitin Pande This document provides an overview of Lucene scoring and sorting algorithms. It describes how Lucene constructs a Hits object to handle scoring and caching of search results. It explains that Lucene scores documents by calling the getScore() method on a Scorer object, which depends on the type of query. For boolean queries, it typically uses a BooleanScorer2. The scoring process advances through documents matching the query terms. Sorting requires additional memory to cache fields used for sorting.
What is in a Lucene index?
What is in a Lucene index?lucenerevolution Presented by Adrien Grand, Software Engineer, Elasticsearch
Although people usually come to Lucene and related solutions in order to make data searchable, they often realize that it can do much more for them. Indeed, its ability to handle high loads of complex queries make Lucene a perfect fit for analytics applications and, for some use-cases, even a credible replacement for a primary data-store. It is important to understand the design decisions behind Lucene in order to better understand the problems it can solve and the problems it cannot solve. This talk will explain the design decisions behind Lucene, give insights into how Lucene stores data on disk and how it differs from traditional databases. Finally, there will be highlights of recent and future changes in Lucene index file formats.
Lucene and MySQL
Lucene and MySQLfarhan "Frank" mashraqi This document provides an overview of Lucene and how it can be used with MySQL. It discusses:
- What Lucene is and its origins as an open source information retrieval library.
- How Lucene works as a toolkit for building search applications rather than a turnkey search engine.
- Core Lucene classes like IndexWriter, Directory, Analyzer, and Document that are used for indexing data.
- Classes like IndexSearcher and Query that support basic search operations through queries and hits.
- Examples of loading data from a MySQL database into a Lucene index and performing searches on that indexed data.
Berlin Buzzwords 2013 - How does lucene store your data?
Berlin Buzzwords 2013 - How does lucene store your data?Adrien Grand This document summarizes how Lucene stores data in segments. It explains that each segment is a fully functional index that stores metadata, inverted indexes, norms, doc values, stored fields, and term vectors. It describes how these components are stored efficiently using techniques like fixed-length data, variable-length data, string compression, and bit packing. The document also outlines improvements to Lucene's storage since version 4.0, including better compression and no seeking on write. Finally, it provides a benchmark showing storage improvements between Lucene 4.0 and 4.4.
Architecture and Implementation of Apache Lucene: Marter's Thesis
Architecture and Implementation of Apache Lucene: Marter's ThesisJosiane Gamgo The thesis studied the architecture and implementation of Apache Lucene. A small search engine called SeboL was implemented to analyze Lucene's indexing and search functionality. Key Lucene components like documents, fields, analyzers, indexes and queries were examined. The indexing process creates an inverted index from documents, while search utilizes the index. The thesis provided an in-depth look at Lucene's architecture and algorithms.
Lucene Introduction
Lucene Introductionotisg Lucene is a free and open source information retrieval (IR) library written in Java. It is widely used to add search functionality to applications. Lucene features fast and scalable indexing and search, and supports various query types including phrase, wildcard, fuzzy and range queries. The Lucene project includes related sub-projects like Solr (search server), Nutch (web crawler), and Mahout (machine learning).
Current advances to bridge the usability-expressivity gap in biomedical seman...
Current advances to bridge the usability-expressivity gap in biomedical seman...Maulik Kamdar I presented a talk at the Protege research meeting on the 'Current advances to bridge the usability-expressivity gap in biomedical semantic search (and visualizing linked data)' https://sites.google.com/site/protegeresearchmeeting/meeting-materials/current-advances-to-bridge-the-usability-expressivity-gap-in-semantic-search
Finite State Queries In Lucene
Finite State Queries In Luceneotisg Finite-State Queries in Lucene:
* Background, improvement/evolution of MultiTermQuery API in 2.9 and Flex
* Implementing existing Lucene queries with NFA/DFA for better performance: Wildcard, Regex, Fuzzy
* How you can use this Query programmatically to improve relevance (I'll use an English test collection/English examples)
Quick overview of other Lucene features in development, such as:
* Flexible Indexing
* "More-Flexible" Scoring: challenges/supporting BM25, more vector-space models, field-specific scoring, etc.
* Improvements to analysis
Bonus:
* Lucene / Solr merger explanation and future plans
About the presenter:
Robert Muir is a super-active Lucene developer. He works as a software developer for Abraxas Corporation. Robert received his MS in Computer Science from Johns Hopkins and BS in CS from Radford University. For the last few years Robert has been working on foreign language NLP problems - "I really enjoy working with Lucene, as it's always receptive to better int'l/language support, even though everyone seems to be a performance freak... such a weird combination!"
Lucene
LuceneMatt Wood The document describes how to index a collection of documents using Lucene. It discusses setting the merge factor and max merge docs parameters, which control how Lucene merges index segments. The code provided shows parsing command line arguments to set these parameters and calls methods to index documents and optimize the index.
Apache Lucene: Searching the Web and Everything Else (Jazoon07)
Apache Lucene: Searching the Web and Everything Else (Jazoon07)dnaber Apache Lucene is a free and open-source search library that provides indexing and searching capabilities. It includes Lucene Java, a core Java library, Solr, a search server with web administration, and Nutch, an open-source web crawler and search engine. Lucene Java provides indexing and searching capabilities, Solr adds web-based administration and HTTP access, and Nutch crawls websites and indexes content.
Introduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of LuceneRahul Jain Rahul Jain gives an introduction to Elasticsearch and its basic concepts like term frequency, inverse document frequency, and boosting. He describes Lucene as a fast, scalable search library that uses inverted indexes. Elasticsearch is introduced as an open source search platform built on Lucene that provides distributed indexing, replication, and load balancing. Logstash and Kibana are also briefly described as tools for collecting, parsing, and visualizing logs in Elasticsearch.
Implementing Semantic Search
Implementing Semantic SearchPaul Wlodarczyk Semantic search uses language processing to analyze the meaning of content and search queries to return more relevant results. It involves classifying content using taxonomies, identifying named entities, extracting relationships between entities, and matching these based on meaning. Implementing semantic search requires preparing content through classification, metadata, and information architecture, as well as technologies for semantic tagging, entity extraction, triple stores, and integrating these capabilities with existing search and content management systems.
Portable Lucene Index Format & Applications - Andrzej Bialecki
Portable Lucene Index Format & Applications - Andrzej Bialeckilucenerevolution See conference video - http://www.lucidimagination.com/devzone/events/conferences/ApacheLuceneEurocon2011
This talk will present a design and implementation of a flexible, version-independent serialization format for Lucene indexes and its applications in index upgrades / downgrades, in distributed document analysis, in distributed indexing, and in integration with external indexing pipelines. This format enables submitting pre-analyzed documents to Lucene/Solr, and transferring parts of indexes between nodes in a distributed setup.
Lucene And Solr Intro
Lucene And Solr Intropascaldimassimo Lucene and Solr are powerful open-source search libraries. Lucene provides the core indexing and search capabilities, while Solr builds on Lucene to provide a ready-to-use search platform with advanced features accessible via HTTP. The document discusses the history and key components of Lucene and Solr, including inverted indexes, analyzers, queries, and schema configuration.
Introduction to Lucene and Solr - 1
Introduction to Lucene and Solr - 1YI-CHING WU this is a solr/lucene search engine induction matrtial. if you would like to know about solr search engine this will be a quick refenence for you.
Lily for the Bay Area HBase UG - NYC edition
Lily for the Bay Area HBase UG - NYC editionNGDATA The document discusses Lily, an open source content application developed by Outerthought that uses HBase for scalable storage and SOLR for search. It provides a high-level overview of Lily's architecture, which maps content to HBase, indexes it in SOLR, and uses a queue implemented on HBase to connect updates between the systems. Future plans for Lily include a 1.0 release with additional features like user management and a UI framework.
Apache lucene
Apache luceneDr. Abhiram Gandhe This document provides an overview of searching and Apache Lucene. It discusses what a search engine is and how it builds an index and answers queries. It then describes Apache Lucene as a high-performance Java-based search engine library. Key features of Lucene like its powerful query syntax, relevance ranking, and flexibility are outlined. Examples of indexing and searching code in Lucene are also provided. The document concludes with a discussion of Lucene's scalability and how it can handle increasing query rates, index sizes, and update rates.
Analytics in olap with lucene & hadoop
Analytics in olap with lucene & hadooplucenerevolution This document discusses reporting and analytics systems at Zanox. It provides an overview of how Zanox handles reporting, including tracking data, building reports using MapReduce, and ensuring system health. It also describes how analytics are performed, including examples of commission and sales data analytics. Finally, it discusses optimizations to the systems like sharding data across search clusters based on important filter dimensions to improve query routing and performance.
Search Lucene
Search LuceneJeremy Coates Zoe Slattery's slides from PHPNW08:
The ability to store large quantities of local data means that many applications require some form of text search and retrieval facility. From the point of view of the application developer there are a number of choices to make, the first is whether to use a complete packaged solution or whether to use one of the available information libraries to build a custom information retrieval (IR) solution. In this talk I’ll look at the options for PHP programmers who choose to embed IR facilities within their applications.
For Java programmers there is clearly a good range of options for text retrieval libraries, but options for PHP programmers are more limited. At first sight for a PHP programmer wishing to embed indexing and search facilities in their application, the choice seems obvious - the PHP implementation of Lucene (Zend Search Lucene). There is no requirement to support another language, the code is PHP therefore easy for PHP programmers to work with and the license is commercially friendly. However, whilst ease of integration and support are key factors in choice of technology, performance can also be important; the performance of the PHP implementation of Lucene is poor compared to the Java implementation.
In this talk I’ll explain the differences in performance between PHP implementation of Lucene and the Java implementation and examine the other options available to PHP programmers for whom performance is a critical factor.
Solr
Solrsortivo Solr is a fulltext search engine based on Lucene that provides fast indexing and retrieval. It has features like distributed search, faceting, highlighting, autocomplete, and extensibility through plugins. Documents are indexed in fields that can be configured and queried using syntax like wildcards, ranges, and proximity searches. Solr allows multiple cores and dynamic fields for flexibility.
Introduction To Apache Lucene
Introduction To Apache LuceneMindfire Solutions Apache Lucene is a high-performance, full-featured text search engine library written in Java. It provides indexing and searching capabilities over various document formats. The Lucene architecture involves indexing documents, building queries, searching the index, and returning results. Core classes for indexing include IndexWriter, Directory, Analyzer, Document, and Field. Core searching classes are IndexSearcher, Query, QueryParser, TopDocs, and ScoreDoc. A demo was presented to index and search documents using Lucene's core classes.
Beyond full-text searches with Lucene and Solr
Beyond full-text searches with Lucene and SolrBertrand Delacretaz Case study using Solr to index the content of existing websites. Presented at ApacheCon EU 2007, Amsterdam.
Devinsampa nginx-scripting
Devinsampa nginx-scriptingTony Fabeen The document discusses extending Nginx functionalities with Lua. It provides an overview of Nginx architecture and how the lua-nginx-module allows running Lua scripts inside Nginx. This provides a powerful and performant programming environment while leveraging Nginx's event-driven architecture. Examples show how to access Nginx variables and APIs from Lua, issue subrequests, and handle requests non-blockingly using cosockets. Libraries like lua-resty-memcached reuse these extensions to build applications in a scalable manner.
Munching & crunching - Lucene index post-processing
Munching & crunching - Lucene index post-processingabial Lucene EuroCon 10 presentation on index post-processing (splitting, merging, sorting, pruning), tiered search, bitwise search, and a few slides on MapReduce indexing models (I ran out of time to show them, but they are there...)
Index types
Index typesVolodymyr Zhabiuk The document discusses different types of indexes that can be used for columnar data, including inverted indexes and forward indexes. It explains that inverted indexes are cheap to store if the column is sorted, as each unique value only needs to be stored once along with its positions. Forward indexes require storing a mapping from unique values to their IDs. This mapping can be compressed using dictionary compression or fixed bit encoding to reduce the space needed. In real use cases, Pinot is able to compress indexes to require only 16 bytes per document while other systems may use over 100 bytes, allowing Pinot to store 7 times more documents in memory.
From Lucene to Elasticsearch, a short explanation of horizontal scalability
From Lucene to Elasticsearch, a short explanation of horizontal scalabilityStéphane Gamard What makes it that Elasticsearch is "horizontaly" scalable while Lucene is not? How does the technology of one affect the other? How does ElasticSearch scale over Lucene and what are the limiting factor?
Text Indexing / Inverted Indices
Text Indexing / Inverted IndicesCarlos Castillo (ChaTo) Part of the course "Algorithmic Methods of Data Science". Sapienza University of Rome, 2015.
http://aris.me/index.php/data-mining-ds-2015
An Introduction to Solr
An Introduction to Solrtomhill The document discusses the open source search platform Solr, describing how it provides a RESTful web interface and Java client for full text search capabilities. It covers installing and configuring Solr, adding and querying data via its HTTP API, and using the SolrJ Java client library. The presentation also highlights key Solr features like faceting, filtering, and scaling for performance.
BNC Tech Forum 09: Lexcycle Stanza demo
BNC Tech Forum 09: Lexcycle Stanza demoBookNet Canada This document provides an overview and details of Lexcycle's Stanza e-reading technology, including:
- Stanza Mobile apps for iPhone and future Android platforms, with over 1.5 million downloads.
- Stanza Desktop apps for Mac and Windows with over 250,000 downloads.
- An online catalog using Atom/XML that provides book metadata and is implemented by Lexcycle and third parties like Smashwords and Feedbooks.
- The catalog includes book listings, popular titles, authors and categories that can be accessed via the Stanza apps.
More Related Content
Viewers also liked (20)
Lucene
LuceneMatt Wood The document describes how to index a collection of documents using Lucene. It discusses setting the merge factor and max merge docs parameters, which control how Lucene merges index segments. The code provided shows parsing command line arguments to set these parameters and calls methods to index documents and optimize the index.
Apache Lucene: Searching the Web and Everything Else (Jazoon07)
Apache Lucene: Searching the Web and Everything Else (Jazoon07)dnaber Apache Lucene is a free and open-source search library that provides indexing and searching capabilities. It includes Lucene Java, a core Java library, Solr, a search server with web administration, and Nutch, an open-source web crawler and search engine. Lucene Java provides indexing and searching capabilities, Solr adds web-based administration and HTTP access, and Nutch crawls websites and indexes content.
Introduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of LuceneRahul Jain Rahul Jain gives an introduction to Elasticsearch and its basic concepts like term frequency, inverse document frequency, and boosting. He describes Lucene as a fast, scalable search library that uses inverted indexes. Elasticsearch is introduced as an open source search platform built on Lucene that provides distributed indexing, replication, and load balancing. Logstash and Kibana are also briefly described as tools for collecting, parsing, and visualizing logs in Elasticsearch.
Implementing Semantic Search
Implementing Semantic SearchPaul Wlodarczyk Semantic search uses language processing to analyze the meaning of content and search queries to return more relevant results. It involves classifying content using taxonomies, identifying named entities, extracting relationships between entities, and matching these based on meaning. Implementing semantic search requires preparing content through classification, metadata, and information architecture, as well as technologies for semantic tagging, entity extraction, triple stores, and integrating these capabilities with existing search and content management systems.
Portable Lucene Index Format & Applications - Andrzej Bialecki
Portable Lucene Index Format & Applications - Andrzej Bialeckilucenerevolution See conference video - http://www.lucidimagination.com/devzone/events/conferences/ApacheLuceneEurocon2011
This talk will present a design and implementation of a flexible, version-independent serialization format for Lucene indexes and its applications in index upgrades / downgrades, in distributed document analysis, in distributed indexing, and in integration with external indexing pipelines. This format enables submitting pre-analyzed documents to Lucene/Solr, and transferring parts of indexes between nodes in a distributed setup.
Lucene And Solr Intro
Lucene And Solr Intropascaldimassimo Lucene and Solr are powerful open-source search libraries. Lucene provides the core indexing and search capabilities, while Solr builds on Lucene to provide a ready-to-use search platform with advanced features accessible via HTTP. The document discusses the history and key components of Lucene and Solr, including inverted indexes, analyzers, queries, and schema configuration.
Introduction to Lucene and Solr - 1
Introduction to Lucene and Solr - 1YI-CHING WU this is a solr/lucene search engine induction matrtial. if you would like to know about solr search engine this will be a quick refenence for you.
Lily for the Bay Area HBase UG - NYC edition
Lily for the Bay Area HBase UG - NYC editionNGDATA The document discusses Lily, an open source content application developed by Outerthought that uses HBase for scalable storage and SOLR for search. It provides a high-level overview of Lily's architecture, which maps content to HBase, indexes it in SOLR, and uses a queue implemented on HBase to connect updates between the systems. Future plans for Lily include a 1.0 release with additional features like user management and a UI framework.
Apache lucene
Apache luceneDr. Abhiram Gandhe This document provides an overview of searching and Apache Lucene. It discusses what a search engine is and how it builds an index and answers queries. It then describes Apache Lucene as a high-performance Java-based search engine library. Key features of Lucene like its powerful query syntax, relevance ranking, and flexibility are outlined. Examples of indexing and searching code in Lucene are also provided. The document concludes with a discussion of Lucene's scalability and how it can handle increasing query rates, index sizes, and update rates.
Analytics in olap with lucene & hadoop
Analytics in olap with lucene & hadooplucenerevolution This document discusses reporting and analytics systems at Zanox. It provides an overview of how Zanox handles reporting, including tracking data, building reports using MapReduce, and ensuring system health. It also describes how analytics are performed, including examples of commission and sales data analytics. Finally, it discusses optimizations to the systems like sharding data across search clusters based on important filter dimensions to improve query routing and performance.
Search Lucene
Search LuceneJeremy Coates Zoe Slattery's slides from PHPNW08:
The ability to store large quantities of local data means that many applications require some form of text search and retrieval facility. From the point of view of the application developer there are a number of choices to make, the first is whether to use a complete packaged solution or whether to use one of the available information libraries to build a custom information retrieval (IR) solution. In this talk I’ll look at the options for PHP programmers who choose to embed IR facilities within their applications.
For Java programmers there is clearly a good range of options for text retrieval libraries, but options for PHP programmers are more limited. At first sight for a PHP programmer wishing to embed indexing and search facilities in their application, the choice seems obvious - the PHP implementation of Lucene (Zend Search Lucene). There is no requirement to support another language, the code is PHP therefore easy for PHP programmers to work with and the license is commercially friendly. However, whilst ease of integration and support are key factors in choice of technology, performance can also be important; the performance of the PHP implementation of Lucene is poor compared to the Java implementation.
In this talk I’ll explain the differences in performance between PHP implementation of Lucene and the Java implementation and examine the other options available to PHP programmers for whom performance is a critical factor.
Solr
Solrsortivo Solr is a fulltext search engine based on Lucene that provides fast indexing and retrieval. It has features like distributed search, faceting, highlighting, autocomplete, and extensibility through plugins. Documents are indexed in fields that can be configured and queried using syntax like wildcards, ranges, and proximity searches. Solr allows multiple cores and dynamic fields for flexibility.
Introduction To Apache Lucene
Introduction To Apache LuceneMindfire Solutions Apache Lucene is a high-performance, full-featured text search engine library written in Java. It provides indexing and searching capabilities over various document formats. The Lucene architecture involves indexing documents, building queries, searching the index, and returning results. Core classes for indexing include IndexWriter, Directory, Analyzer, Document, and Field. Core searching classes are IndexSearcher, Query, QueryParser, TopDocs, and ScoreDoc. A demo was presented to index and search documents using Lucene's core classes.
Beyond full-text searches with Lucene and Solr
Beyond full-text searches with Lucene and SolrBertrand Delacretaz Case study using Solr to index the content of existing websites. Presented at ApacheCon EU 2007, Amsterdam.
Devinsampa nginx-scripting
Devinsampa nginx-scriptingTony Fabeen The document discusses extending Nginx functionalities with Lua. It provides an overview of Nginx architecture and how the lua-nginx-module allows running Lua scripts inside Nginx. This provides a powerful and performant programming environment while leveraging Nginx's event-driven architecture. Examples show how to access Nginx variables and APIs from Lua, issue subrequests, and handle requests non-blockingly using cosockets. Libraries like lua-resty-memcached reuse these extensions to build applications in a scalable manner.
Munching & crunching - Lucene index post-processing
Munching & crunching - Lucene index post-processingabial Lucene EuroCon 10 presentation on index post-processing (splitting, merging, sorting, pruning), tiered search, bitwise search, and a few slides on MapReduce indexing models (I ran out of time to show them, but they are there...)
Index types
Index typesVolodymyr Zhabiuk The document discusses different types of indexes that can be used for columnar data, including inverted indexes and forward indexes. It explains that inverted indexes are cheap to store if the column is sorted, as each unique value only needs to be stored once along with its positions. Forward indexes require storing a mapping from unique values to their IDs. This mapping can be compressed using dictionary compression or fixed bit encoding to reduce the space needed. In real use cases, Pinot is able to compress indexes to require only 16 bytes per document while other systems may use over 100 bytes, allowing Pinot to store 7 times more documents in memory.
From Lucene to Elasticsearch, a short explanation of horizontal scalability
From Lucene to Elasticsearch, a short explanation of horizontal scalabilityStéphane Gamard What makes it that Elasticsearch is "horizontaly" scalable while Lucene is not? How does the technology of one affect the other? How does ElasticSearch scale over Lucene and what are the limiting factor?
Text Indexing / Inverted Indices
Text Indexing / Inverted IndicesCarlos Castillo (ChaTo) Part of the course "Algorithmic Methods of Data Science". Sapienza University of Rome, 2015.
http://aris.me/index.php/data-mining-ds-2015
Similar to Lucandra (20)
An Introduction to Solr
An Introduction to Solrtomhill The document discusses the open source search platform Solr, describing how it provides a RESTful web interface and Java client for full text search capabilities. It covers installing and configuring Solr, adding and querying data via its HTTP API, and using the SolrJ Java client library. The presentation also highlights key Solr features like faceting, filtering, and scaling for performance.
BNC Tech Forum 09: Lexcycle Stanza demo
BNC Tech Forum 09: Lexcycle Stanza demoBookNet Canada This document provides an overview and details of Lexcycle's Stanza e-reading technology, including:
- Stanza Mobile apps for iPhone and future Android platforms, with over 1.5 million downloads.
- Stanza Desktop apps for Mac and Windows with over 250,000 downloads.
- An online catalog using Atom/XML that provides book metadata and is implemented by Lexcycle and third parties like Smashwords and Feedbooks.
- The catalog includes book listings, popular titles, authors and categories that can be accessed via the Stanza apps.
SPARQLing Services
SPARQLing ServicesLeigh Dodds The document discusses SPARQL, a query language for RDF data. It describes the key components of SPARQL, including its specification, query types, results format, and protocol. It also covers implementation issues for SPARQL services and provides examples of using SPARQL to query RSS feeds, geographical data, and more. Extensions discussed include querying by reference, XSLT transformation of results, and a JSON results format.
Best practices in museum search
Best practices in museum searchNate Solas The document discusses best practices for search on museum websites. It recommends using Google search for the website but also implementing internal search to better understand the museum's unique content. It provides examples of autocomplete, faceted search, highlighting search terms, and correcting spelling to improve the user search experience. Solr is recommended as an open source tool for implementing these search features.
Ks2007 Semanticweb In Action
Ks2007 Semanticweb In ActionRinke Hoekstra The document discusses the Semantic Web, including its languages (RDF, RDFS, OWL), storage and querying using SPARQL, and methods for browsing and viewing semantic data through techniques like faceted browsing and Fresnel lenses. While the core technologies exist, broader adoption of the Semantic Web on the mainstream web still has challenges to overcome.
Lecture 3 - Comm Lab: Web @ ITP 
Lecture 3 - Comm Lab: Web @ ITP yucefmerhi This document provides an overview and summary of a lecture on using the Sinatra web framework.
The lecture covers:
- Announcements about assignment deadlines and office hours
- An introduction to Sinatra including what it is, how it works, and its benefits
- Explanations of routes, parameters, strings, and if statements for building web applications in Sinatra
- Examples of creating forms and handling requests
XMLT
XMLTKunal Gaind The document discusses XML transformations using XSLT and XPath. It provides an overview of these technologies and how they are used to transform and navigate XML documents. Examples are given demonstrating how to select nodes and attributes in an XML document using XPath location paths and apply templates and output values using XSLT elements like xsl:template, xsl:value-of, and xsl:for-each.
Ruby off Rails---rack, sinatra and sequel
Ruby off Rails---rack, sinatra and sequelJiang Wu A presentation on Shanghai on Rails, March 14 2009, includes rack middelware introduction, one-file application using sinatra and sequel, and sinatra program running under Rails Metal.
Is There Room For Another Elephant In Tucson
Is There Room For Another Elephant In TucsonAndy Lenards Would you like to scale data-intensive tasks horizontally? Would you like an open source project that gave you that foundation?
Well, there is: Apache Hadoop. It's a Java software framework for supporting data-intensive distributed applications. The framework was inspired by Google papers on their MapReduce framework and Google File System.
Who uses Hadoop? Here's a short list: Yahoo!, A9.com, LinkedIn, Facebook, ImageShack, eHarmony, Hulu, Last.fm, and The New York Times. The highest profile user, Yahoo!, is also a major contributor to the project. They use it extensively in their web search and advertising divisions.
In this talk, titled "Is there room for another elephant in Tucson?", Andrew Lenards will tell us about Hadoop and describe how it could be applied to several practical problems, even if you aren't as big as Google.
Forum Presentation
Forum PresentationAngus Pratt The document discusses the Diamond community forums that were created using Drupal to serve the Business Objects developer community. It provides an overview of the forums and improvements made over time, including moving the forums to the new SAP Developer Network. Template and code snippets are included that show how forums, categories, comments, users and other elements were structured in Drupal.
Faceted Search – the 120 Million Documents Story
Faceted Search – the 120 Million Documents StorySourcesense Upayavira's presentation at Online Information 2010 in London: the case study of an Enterprise-critical migration from custom Lucene indexes to Apache Solr, with a significant focus on scalability.
The solution needed to providing search against rapidly changing data-sets and multi-million document indexes, enabling complex queries with sub second responses and maintaining high availability.
XML and Web Services with PHP5 and PEAR
XML and Web Services with PHP5 and PEARStephan Schmidt The document discusses XML and web services with PHP5 and PEAR. It provides an introduction and overview of XML including related technologies like DTD, XML Schema, Relax NG, XPath, and XSLT. It also covers using XML in PHP5 with SAX, DOM, SimpleXML, XPath, and XSLT. Finally, it discusses web services in PHP5 and PEAR.
Weird Plsql
Weird Plsqlwebanddb You probably think that PL/SQL is dull and ordinary programming language. Not so! Parts of it can be downright WEIRD. In this presentation, Steven offers what he considers to be some of the stranger nooks and crannies of the PL/SQL language, perhaps in the process making them a little bit less weird.
Improving Soap Message Serialization
Improving Soap Message SerializationPrabath Siriwardena SOAP web services are slower than other technologies like RMI and Remoting due to the overhead of XML processing and serialization to text format. Template generation and caching of SOAP messages can help improve performance by reducing serialization costs. The document discusses generating SOAP message templates from a WSDL file and implementing caching of templates and SOAP envelopes on the client side to gain performance improvements, especially for requests with complex objects.
Code4Lib 2007: MyResearch Portal
Code4Lib 2007: MyResearch Portaleby Presentation by Andrew Nagy at Code4Lib 2007 in Athens, GA.
Villanova University’s Falvey Memorial Library has longed for a beautiful pig; however, we determined in early 2006 that pigs were only good at searching for truffles, so we decided to build our own OPAC.
After developing our own custom Digital Library from a Native XML Database, we quickly appreciated the ease of development with XQuery and XSLT. We then launched full speed ahead into the development of a new OPAC from scratch using XML technologies and MARCXML.
This presentation will describe the process of choosing an NXDB and optimizing it for large data set performance. Developing searches that take about 2 minutes to process and optimizing them down to about 2 seconds. I will also describe the development processes of the OPAC interface including the AJAX features we have implemented. I will share our success stories and our failures.
Sax Dom Tutorial
Sax Dom Tutorialvikram singh This document provides an overview of DOM and SAX, two common XML APIs in Java. It describes the key differences between DOM and SAX, including that DOM builds an in-memory tree representation, while SAX parses the XML as a stream of events. The document also provides code examples for using SAX to parse XML and extract data, and examples of how to access and manipulate DOM trees after parsing XML.
Sphinx on Rails
Sphinx on Railsfreelancing_god Presentation from OSDC 2008 in Sydney, on using Sphinx with Rails. The sldes don't work that well without the talk though, but it may be useful as a reference, I guess.
How to use cache scope component
How to use cache scope componentprathyusha vadla The document discusses how to use the Cache Scope component in Mule applications. The Cache Scope processes messages, delivers the output to the parent flow, and caches responses. On subsequent messages of the same type, the Cache Scope can return a cached response without re-invoking the process. An example Mule flow is provided that uses a Cache Scope to cache the response of a SOAP request to Sabre for 10 minutes, returning the cached response for subsequent identical requests.
Beyond the Node: Arkestration with Noah
Beyond the Node: Arkestration with Noahlusis This document discusses Noah, a system for orchestration and coordination of infrastructure components like configuration management tools, applications, servers, and services. Noah uses a hierarchical data model and API to represent these components as objects that can be manipulated and watched. It aims to address the need for orchestration beyond single nodes and provide synchronization between different aspects of infrastructure.
XML processing with perl
XML processing with perlJoe Jiang The document discusses using the XML::XSH2 tool in Perl to process XML documents, describing how it allows working with XML using XPath queries and verbs to navigate, select, and transform XML content. Examples are provided of using XML::XSH2 to count elements, copy content between XML files, and convert between XML formats like transforming XML into HTML.
Lucandra
- 1. Lucandra Lucene + Cassandra http://github/tjake/Lucandra http://twitter.com/tjake Jake Luciani
- 2. What we'll cover today: Search use-cases Problems scaling and maintaining Lucene/Solr Cassandra Lucandra Lucandra in Action Q&A
- 3. Types of search apps:
- 4. Types of search apps:
- 5. Lucene/Solr Scaling Problems Writes are expensive on a live system Merge, Reopen, Optimize, Sorting "Too many open files" Solr replication too many moving parts Scaling writes requires client side sharding Lots of grid management -> ZooKeeper? Backups? Monitoring? Failures? Ops Team? Oh my! This sounds a lot like mysql doesn't it?....
- 6. Cassandra - Love Child of BigTable and Dynamo Peer to peer (easy to add new nodes) CAP Configurable Multi-level TreeMap (sorta) Pluggable replication/sorting Writes are very fast! Low latency Integrates with Hadoop Major adoption and development
- 7. Cassandra's Data Model { "bloghost.com" : // Keyspace { "Posts" : // ColumnFamily { " tjake.bloghost.com " : // Key { "20100426-Lucandra" : "lucandra talk today!" } // Columns } }, { "Comments" : // SuperColumnFamily { " tjake.bloghost.com " : // Key { "20100426-Lucandra-1": // SuperColumn {"From" : "Otis","Comment": "Don't Suck!"}, // Columns }, { "20100426-Lucandra-2": // SuperColumn {"From" : "Jake","Comment": "O.K."}, // Columns }, } }}
- 9. Cassandra - Scale Up / Scale Down
- 13. How is an index stored? { "Lucandra" : { "Docs" : { "Index1/Doc1" : { "Field1" : "T1 T2 T1", ... }, { "Index1/Doc2" : { "Field1" : "T3 T1", ... } }, {"TermVectors" : {"Index1/Field1/T1" : { "Doc1": [0, 2], "Doc2":[1] }, {"Index1/Field1/T2" : { "Doc1": [1] }, {"Index1/Field1/T3" : { "Doc2": [1] }, } }
- 15. Lucandra In Action Sparse.ly and Wikassandra
- 16. sparse.ly - twitter search for friends only ~4k Indexes on 2 boxes
- 17. Wikassandra - Search wikipedia 4 node cluster 3k writes per sec (over thrift from single node) Solr interface
Editor's Notes
- #2: test