Introduction data structure for GraphDB
29 likes3,804 views
The document discusses the data structure used for nodes in a graph database. It describes how nodes are stored in a node cache managed by a node manager. Relationships between nodes are stored in a relationship cache. When a new node is created, it is assigned an ID and added to the node cache and persistence manager. Creating a relationship involves linking two existing nodes in the relationship cache. The cache uses a least recently used (LRU) algorithm to manage memory usage.
More Related Content
Similar to Introduction data structure for GraphDB (20)
Recently uploaded (20)
Introduction data structure for GraphDB
- 1. Introduction data structure for GraphDB mixi.inc Shunya Kimura @kimuras
- 2. agenda • Motivation • Introduction • Structure for Node • Create Node • Appendix
- 3. Motivation
- 4. • Just interesting in GraphDB data structure • Why fast • Mysterious
- 5. Introduction
- 6. Neo4j • OSS implement of GraphDB • High availability • Scales to billions of nods and relationships • High speed querying through traversals • For more Information about Neo4j • http://www.slideshare.net/skimura/
- 7. Introduction for Graph Node Name: Kimura Job: Eng Relationship
- 8. Introduction for Graph Node Name: Kimura Job: Eng Relationship
- 9. Introduction for Graph Node Data Structure Create Node Relationship
- 11. A node data structure Node Manager
- 12. A node data structure Node Manager Node Cache
- 13. A node data structure Node Manager Node Cache Relation Cache
- 14. A node data structure Node Manager Node Cache Relation Cache Cache Manager
- 15. A node data structure Node Manager Node Cache Relation Cache Cache Manager Property Index Manager
- 16. A node data structure Node Manager Node Cache Relation Cache Cache Manager Property Index Transaction Manager Manager
- 17. A node data structure Node Manager Node Cache Relation Cache Cache Manager Property Index Transaction Manager Persistence Manager Manager
- 18. A node data structure Node Manager Node Cache Relation Cache Cache Manager Property Index Transaction Manager Persistence Manager Manager id id id id relationships relationships relationships relationships Node Node Node Node
- 19. Node Manager • NodeCache: Caching node id and node object • RelationCache: Caching rel id and Relation object • Cache Manager: Management cache threads • Property Index Manager: Management all nodes Properties • Persistence Manager: Management store
- 20. Relationship structure id relationships Node
- 21. Relationship structure id relationships Node
- 22. Relationship structure Array structure for relationships id relationships Node
- 23. Relationship structure relationships In Blocks Out Blocks Node
- 24. Relationship structure Node Node relationships In Blocks Node Node Out Blocks Node Node Node Node Node
- 25. Relationship structure In Direction Node Node relationships In Blocks Node Node Out Blocks Node Node Node Node Node
- 26. Relationship structure In Direction Node Node relationships In Blocks Node Node Out Blocks Node Node Node Out Direction Node Node
- 27. Relationship structure Node id relationships
- 28. Relationship structure Node id relationships relationships
- 29. Relationship structure Node id relationships relationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9
- 30. Relationship structure Node id relationships relationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9 Out blocks ID:10 ID:11 ID:12 ID:13 ID:14 ID:15 ID:16 ID:17 ID:18
- 31. Relationship structure Node id Last In block relationships relationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9 Out blocks ID:10 ID:11 ID:12 ID:13 ID:14 ID:15 ID:16 ID:17 ID:18
- 32. Relationship structure Node id Last In block relationships Last Out block relationships In blocks ID:1 ID:2 ID:3 ID:4 ID:5 ID:6 ID:7 ID:8 ID:9 Out blocks ID:10 ID:11 ID:12 ID:13 ID:14 ID:15 ID:16 ID:17 ID:18
- 33. Create Nodes
- 34. Create Node • Understanding data structure with creating nodes • Create Node A • Create Node B • Create Relationship A to B
- 35. Create Node GraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH ); Node NodeA = graphDb.createNode(); Node NodeB = graphDb.createNode(); Relationship relationship = NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS );
- 36. Create Node GraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH ); Node NodeA = graphDb.createNode(); Node NodeB = graphDb.createNode(); Relationship relationship = NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS ); NodeA
- 37. Create Node GraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH ); Node NodeA = graphDb.createNode(); Node NodeB = graphDb.createNode(); Relationship relationship = NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS ); NodeA NodeB
- 38. Create Node GraphDatabaseService graphDb = new EmbeddedGraphDatabase( DB_PATH ); Node NodeA = graphDb.createNode(); Node NodeB = graphDb.createNode(); Relationship relationship = NodeA.createRelationshipTo( NodeB, RelTypes.KNOWS ); NodeA NodeB
- 40. Introduction Create Node NodeA
- 41. Introduction Create Node Node Manager Node Cache Persistence Manager Relation Cache NodeA
- 42. Introduction Create Node Create New ID id Node Manager Node Cache Persistence Manager Relation Cache NodeA
- 43. Introduction Create Node Create New ID id Create Node Node Manager id Node Cache relationships Persistence Manager NodeA Relation Cache NodeA
- 44. Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager NodeA Relation Cache NodeA
- 45. Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager register Persistence Manager NodeA Relation Cache NodeA
- 46. Cache • You can choice some cache type • Strong Reference Cache • Soft LRU Cache (default) • Weak LRU Cache • No Cache
- 47. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms
- 48. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE 1 VAL_1 2 VAL_2 3 VAL_3 4 VAL_4 5 VAL_5
- 49. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE 1 VAL_1 2 VAL_2 Priority 3 VAL_3 4 VAL_4 5 VAL_5
- 50. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE GET KEY:5 1 VAL_1 2 VAL_2 Priority 3 VAL_3 4 VAL_4 5 VAL_5
- 51. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE GET KEY:5 5 VAL_5 1 VAL_1 Priority 2 VAL_2 3 VAL_3 4 VAL_4
- 52. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE 5 VAL_5 1 VAL_1 Priority 2 VAL_2 3 VAL_3 4 VAL_4
- 53. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE PUT KEY:6 5 VAL_5 1 VAL_1 Priority 2 VAL_2 3 VAL_3 4 VAL_4
- 54. What is LRU Cache • discards the least recently used items first. http://en.wikipedia.org/wiki/Cache_algorithms On memory KEY VALUE PUT KEY:6 6 VAL_6 5 VAL_5 Priority 1 VAL_1 2 VAL_2 3 VAL_3
- 55. Soft LRU Cache
- 56. Soft LRU Cache Soft LRU Cache Cache (ConcurrentHashMap) Queue (SoftReferenceQueue)
- 57. Soft LRU Cache ConcurrentHashMap KEY VALUE 1 ref_1 Soft LRU Cache 2 ref_2 3 ref_3 Cache (ConcurrentHashMap) Queue (SoftReferenceQueue)
- 58. Soft LRU Cache ConcurrentHashMap KEY VALUE 1 ref_1 Soft LRU Cache 2 ref_2 3 ref_3 Cache (ConcurrentHashMap) SoftRefrenceQueue Queue (SoftReferenceQueue) KEY VALUE(※SoftRefrence) 1 ref_1 2 ref_2 3 ref_3 ※Soft reference objects, which are cleared at the discretion of the garbage collector in response to memory demand. http://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html
- 59. Soft LRU Cache ConcurrentHashMap KEY VALUE 1 ref_1 Soft LRU Cache 2 ref_2 3 ref_3 Cache (ConcurrentHashMap) SoftRefrenceQueue Queue (SoftReferenceQueue) KEY VALUE(※SoftRefrence) 1 ref_1 2 ref_2 3 ref_3 ※Soft reference objects, which are cleared at the discretion of the garbage collector in response to memory demand. http://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html
- 60. Comparing Cache type Cache Type Queue Type Reference Type Weak LRU Cache WeakReferenceQueue WeakReference Soft LRU Cache SoftRefrenceQueue SoftReference Strong Reference None None Cache Remaining Reference on memory Strong > Soft > Weak
- 61. Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager register Persistence Manager NodeA Relation Cache NodeA
- 62. Introduction Create Node Create New ID id Create Node Node Manager id register Node cache Node Cache relationships Persistence Manager register Persistence Manager NodeA Relation Cache NodeA NodeB
- 63. Introduction Create Node NodeA NodeB
- 64. Introduction Create Node NodeA NodeB
- 65. Introduction Create Node id Node Manager relationships Node Cache NodeA Persistence Manager id Relation Cache relationships NodeB NodeA NodeB
- 66. Introduction Create Node id Node Manager relationships Node Cache NodeA Register OUT direction to NodeB Persistence Manager id Relation Cache relationships NodeB NodeA NodeB
- 67. Introduction Create Node id Node Manager relationships Node Cache NodeA Register OUT direction to NodeB Persistence Manager id Relation Cache relationships Register IN direction from NodeA NodeB NodeA NodeB
- 68. Introduction Create Node id Node Manager relationships Node Cache NodeA Persistence Manager id Relation Cache relationships NodeB NodeA NodeB
- 69. Introduction Create Node Create New Relation ID rel id id Node Manager relationships Node Cache NodeA Persistence Manager id Relation Cache relationships NodeB NodeA NodeB
- 70. Introduction Create Node Create New Relation ID create Relationship rel id id Node Manager relationships Node Cache NodeA Relation Persistence Manager id Relation Cache relationships NodeB NodeA NodeB
- 71. Introduction Create Node Create New Relation ID create Relationship rel id id Node Manager relationships Node Cache NodeA Relation Persistence Manager id Relation Cache relationships NodeB NodeA NodeB
- 72. Appendix
- 73. Implemented Graph Algorithms • Centrality • Betweenness Centrality • Closeness Centrality • Path • A* • Dijkstart • Shortest Path
- 74. announcement
- 75. (NL ) (DSIRNLP) (Tokyo NLP) (Japan.R) ( ) (TokyoWebmining) (MongoDB Japan) http://www.ipsj.or.jp/10jigyo/taikai/74kai/event_2-6.html
- 76. Thanks!
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n
- #47: \n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n
- #54: \n
- #55: \n
- #56: \n
- #57: \n
- #58: \n
- #59: \n
- #60: \n
- #61: \n
- #62: \n
- #63: \n
- #64: \n
- #65: \n
- #66: \n
- #67: \n
- #68: \n
- #69: \n
- #70: \n
- #71: \n
- #72: \n
- #73: \n
- #74: \n
- #75: \n
- #76: \n
- #77: \n
- #78: \n
- #79: \n
- #80: \n
- #81: \n
- #82: \n
- #83: \n
- #84: \n
- #85: \n
- #86: \n
- #87: \n
- #88: \n
- #89: \n
- #90: \n
- #91: \n
- #92: \n
- #93: \n
- #94: \n
- #95: \n
- #96: \n
- #97: \n
- #98: \n
- #99: \n
- #100: \n
- #101: \n
- #102: \n
- #103: \n
- #104: \n
- #105: \n
- #106: \n
- #107: \n
- #108: \n
- #109: \n
- #110: \n
- #111: \n
- #112: \n
- #113: \n
- #114: \n
- #115: \n
- #116: \n
- #117: \n
- #118: \n
- #119: \n
- #120: \n
- #121: \n
- #122: \n
- #123: \n
- #124: \n
- #125: \n
- #126: \n
- #127: \n
- #128: \n
- #129: \n
- #130: \n
- #131: \n
- #132: \n
- #133: \n
- #134: \n
- #135: \n
- #136: \n
- #137: \n
- #138: \n
- #139: \n
- #140: \n
- #141: \n
- #142: \n
- #143: \n
- #144: \n
- #145: \n
- #146: \n
- #147: \n
- #148: \n
- #149: \n
- #150: \n
- #151: \n
- #152: \n
- #153: \n
- #154: \n
- #155: \n
- #156: \n
- #157: \n
- #158: \n
- #159: \n
- #160: \n
- #161: \n
- #162: \n
- #163: \n
- #164: \n
- #165: \n
- #166: \n
- #167: \n
- #168: \n
- #169: \n
- #170: \n
- #171: \n
- #172: \n
- #173: \n
- #174: \n
- #175: \n
- #176: \n
- #177: \n
- #178: \n
- #179: \n
- #180: \n
- #181: \n
- #182: \n
- #183: \n
- #184: \n
- #185: \n
- #186: \n
- #187: \n
- #188: \n
- #189: \n
- #190: \n
- #191: \n
- #192: \n
- #193: \n
- #194: \n
- #195: \n
- #196: \n
- #197: \n
- #198: \n
- #199: \n
- #200: \n
- #201: \n
- #202: \n
- #203: \n