SlideShare a Scribd company logo

Lost In The Clouds

•
•
G
george.james

This document summarizes OpenStreetMap, a free editable map of the world created by volunteers. OpenStreetMap data is stored on a cloud-based database server containing over 300 million nodes and 3.6 billion tags. The XAPI service provides a RESTful API for querying OpenStreetMap data in chunks up to 100 square degrees using tags and indexes. It currently runs on 5 server instances around the world but needs more to scale to increased usage.

1 of 15
Download to read offline
Lost in the Clouds An example of a schemaless database application in the cloud SlipstreamUSA :: March 25, 2009
The Project OpenStreetMap  A free map of the planet  Free geospatial data  Licensed under CC-BY-SA  Created from  Crowd-sourced contributions  90,000 registered users  1,000+ contributors per day  Used for:  Wikipedia  Flikr  OpenCycleMap.org  OpenPisteMap.org  YourNavigation.org  Mobile devices  © Constantin Litvak
Coverage CC-BY-SA 2.0 © OpenStreetMap
The OpenStreetMap Database Server In the cloud  MySQL  Ways: 24,956,171  Nodes: 305,956,583  Tags: 3.6 billion  highway=motorway  name=Broadway  foo=bar  CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
Interfaces API  For updates  (1,000 users per day) Queries of up to  0.25² degrees Weekly Planet Dump  100Gbyte XML file  Changes  Feed of changes  Daily / Hourly / Minutely  One file per minute  (~30Kb compressed) CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
Problem Statement Problem  Want to be able to query the data in bigger chunks and to do it more selectively Solution  Cloud based schemaless database with simple to use query interface and lots of indexes CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
XAPI Service GT.M or Cache  250Gbyte database  Nodes: 30Gb  Node tags: 115Gb  Node indexes: 70Gb  Ways: 21Gb  Way tags: 10Gb  Way indexes: 18Gb  Apache + serverLink  Licensed as AGPL  CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
XAPI Service In the cloud  Queries of up to 100² degrees  Query by tags and tag values  Scaleable  REST style interface  http://xapi.openstreetmap.org/api/0.6/* [tourism=hotel] [bbox=-75.7,40.0,-75.4,40.2] Import planet dump  Import minute diff  CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
XAPI Service Geospatial Index – quadstrings  adaabcdcabaadab
XAPI Service <way id='27016525'> <nd ref='296138118'/> <nd ref='296138119'/> ^way(27016525)=quot;adaabcdcabaadabquot; <nd ref='296138120'/> ^way(27016525,1)=296138118 <nd ref='296138121'/> ^way(27016525,2)=296138119 <nd ref='296138118'/> <tag k='addr:housenumber' v='2'/> ^way(27016525,3)=296138120 <tag k='building' v='yes'/> ^way(27016525,4)=296138121 </way> ^way(27016525,5)=296138118 ^waytag(27016525,quot;addr:housenumberquot;)=2 ^waytag(27016525,quot;buildingquot;)=quot;yesquot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27016525)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028298)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028299)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028326)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028327)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035972)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035973)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035974)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035975)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035984)=quot;“
Deployment Currently 5 instances:  London  Paris  San Diego  Zurich  Amsterdam  (Wikimedia-de) More servers needed:  Internet connectivity  300Gb disk storage  CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap
Wikipedia: Query to Map {{osm-query |way=name=Arizona Canal}} CC-BY-SA 2.0 © OpenStreetMap
Internals Debian  Apache web-server  REST  URI represents a  resource Xpath flavoring  serverLink  WebLink emulator  GT.M or Cache  POM  AGPL 
XAPI Future Load Sharing / balancing  More Xpath-like queries  JSON output  Generic indexing service  More servers 
Thank you Thank you Questions? CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap

More Related Content

Lost In The Clouds

  • 1. Lost in the Clouds An example of a schemaless database application in the cloud SlipstreamUSA :: March 25, 2009
  • 2. The Project OpenStreetMap  A free map of the planet  Free geospatial data  Licensed under CC-BY-SA  Created from  Crowd-sourced contributions  90,000 registered users  1,000+ contributors per day  Used for:  Wikipedia  Flikr  OpenCycleMap.org  OpenPisteMap.org  YourNavigation.org  Mobile devices  © Constantin Litvak
  • 3. Coverage CC-BY-SA 2.0 © OpenStreetMap
  • 4. The OpenStreetMap Database Server In the cloud  MySQL  Ways: 24,956,171  Nodes: 305,956,583  Tags: 3.6 billion  highway=motorway  name=Broadway  foo=bar  CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
  • 5. Interfaces API  For updates  (1,000 users per day) Queries of up to  0.25² degrees Weekly Planet Dump  100Gbyte XML file  Changes  Feed of changes  Daily / Hourly / Minutely  One file per minute  (~30Kb compressed) CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
  • 6. Problem Statement Problem  Want to be able to query the data in bigger chunks and to do it more selectively Solution  Cloud based schemaless database with simple to use query interface and lots of indexes CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
  • 7. XAPI Service GT.M or Cache  250Gbyte database  Nodes: 30Gb  Node tags: 115Gb  Node indexes: 70Gb  Ways: 21Gb  Way tags: 10Gb  Way indexes: 18Gb  Apache + serverLink  Licensed as AGPL  CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
  • 8. XAPI Service In the cloud  Queries of up to 100² degrees  Query by tags and tag values  Scaleable  REST style interface  http://xapi.openstreetmap.org/api/0.6/* [tourism=hotel] [bbox=-75.7,40.0,-75.4,40.2] Import planet dump  Import minute diff  CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
  • 9. XAPI Service Geospatial Index – quadstrings  adaabcdcabaadab
  • 10. XAPI Service <way id='27016525'> <nd ref='296138118'/> <nd ref='296138119'/> ^way(27016525)=quot;adaabcdcabaadabquot; <nd ref='296138120'/> ^way(27016525,1)=296138118 <nd ref='296138121'/> ^way(27016525,2)=296138119 <nd ref='296138118'/> <tag k='addr:housenumber' v='2'/> ^way(27016525,3)=296138120 <tag k='building' v='yes'/> ^way(27016525,4)=296138121 </way> ^way(27016525,5)=296138118 ^waytag(27016525,quot;addr:housenumberquot;)=2 ^waytag(27016525,quot;buildingquot;)=quot;yesquot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27016525)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028298)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028299)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028326)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028327)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035972)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035973)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035974)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035975)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035984)=quot;“
  • 11. Deployment Currently 5 instances:  London  Paris  San Diego  Zurich  Amsterdam  (Wikimedia-de) More servers needed:  Internet connectivity  300Gb disk storage  CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap
  • 12. Wikipedia: Query to Map {{osm-query |way=name=Arizona Canal}} CC-BY-SA 2.0 © OpenStreetMap
  • 13. Internals Debian  Apache web-server  REST  URI represents a  resource Xpath flavoring  serverLink  WebLink emulator  GT.M or Cache  POM  AGPL 
  • 14. XAPI Future Load Sharing / balancing  More Xpath-like queries  JSON output  Generic indexing service  More servers 
  • 15. Thank you Thank you Questions? CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap