I have played around with Elasticsearch for a while and it has been my first time I was working with a NoSQL database. It’s been a lot of fun to load some data and see what querying techniques are available. I was particularly interested in seeing what kind of support for geospatial operations does the elasticsearch provides. As it turned out, there is very good support for GeoJSON data structures for storage and visualization (both as point data and as areas). You can even run the spatial queries (e.g. find all points within a specified polygon) and draw the results on a tiled basemap as an operational layer (using Kibana) along with using regular SQL-like attribute data filtering.
You can load your geospatial datasets into elasticsearch NoSQL database and then visualize it on a basemap as well as run some spatial querying, draw heat maps and choropleth maps using own regions (polygons that can be enriched with the point data). This is just a few maps I’ve created really quickly loading the cities and states geodatabase feature classes into the NoSQL database and then using Kibana web interface to author visualizations:
It all seems to be very powerful yet fairly easy to configure. Loading the data into the database was not very straightforward, though, because as I understood, you cannot feed the source GeoJSON files into the elasticsearch when using the _bulk api to load documents into an index. You can read more about GeoJSON support in elastic in the docs page Geo-Shape datatype.
I’ve written a tiny Python script that will convert a geodatabase feature class into a GeoJSON and then construct a new .json file with the proper format that can be loaded into the elasticsearch database.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ''' | |
| Convert file geodatabase feature class features into GeoJSON and | |
| then construct text file for loading into elastic search bulk api | |
| ''' | |
| import json | |
| import arcpy | |
| arcpy.env.overwriteOutput = True | |
| #create a geojson file | |
| geo_json_cities_path = r'C:\GIS\Temp\Elastic\usa_cities.json' | |
| arcpy.FeaturesToJSON_conversion(in_features=r'C:\Program Files (x86)\ArcGIS\Desktop10.5\TemplateData\TemplateData.gdb\USA\cities', | |
| out_json_file=geo_json_cities_path, geoJSON=True) | |
| with open(geo_json_cities_path) as f: | |
| data = json.load(f) | |
| # | |
| geo_shapes = [] | |
| for feat in data['features']: | |
| geo_shape = {} | |
| geo_shape['name'] = feat['properties']['CITY_NAME'] | |
| geo_shape['state'] = feat['properties']['STATE_NAME'] | |
| geo_shape['pop'] = feat['properties']['POP1990'] | |
| geo_shape['location'] = feat['geometry']['coordinates'] | |
| geo_shapes.append(geo_shape) | |
| with open(r'C:\GIS\Temp\Elastic\ready\usa_cities_elastic.json','wb') as f: | |
| for city in geo_shapes: | |
| f.write('{ "index": {"_index": "cities", "_type": "city"} }\n') | |
| f.write(json.dumps(city)) | |
| f.write('\n') | |
| #sample rows from output .json file: | |
| #{ "index": {"_index": "cities", "_type": "city"} } | |
| #{"state": "Washington", "name": "Bellingham", "pop": 52179, "location": [-122.46818353792992, 48.74387985436505]} | |
| #{ "index": {"_index": "cities", "_type": "city"} } | |
| #{"state": "Montana", "name": "Havre", "pop": 10201, "location": [-109.67985528815007, 48.54381826401885]} |
To play with the elasticsearch API interface, I’ve used Postman. It’s a very handy application that will let you construct all kinds of HTTP requests and save them to reuse later on. Using Postman, I was able to submit the PUT request to load documents from the .json file I’ve created by running the Python script using the binary body and browsing to the source data file.
Alex Tereshenkov 