Tabular + graph datasets with CRUD/query, persistence, and extensible algorithm components.
In Unity: Window → Package Manager → + → Add package from git URL... and enter:
https://github.com/Stlouislee/DataCore-for-Unity.git
Add to your Packages/manifest.json:
{
"dependencies": {
"com.aroaro.datacore": "https://github.com/Stlouislee/DataCore-for-Unity.git"
}
}- Open Window → Package Manager
- Click the + button
- Select Add package from git URL
- Enter:
https://github.com/Stlouislee/DataCore-for-Unity.git - Click Add
This package includes precompiled DLLs:
- Apache.Arrow.dll - Tabular data serialization
- LiteDB.dll - Embedded document database
- Microsoft.Data.Analysis.dll - DataFrame support
These dependencies are included in the package and will be automatically available after installation.
Add a DataCoreEditorComponent to a GameObject in your scene. Other scripts can then access it as a shared singleton:
using AroAro.DataCore;
// From any script, access the shared DataCore instance
var store = DataCoreEditorComponent.Instance.GetStore();
// Create and work with datasets
var playerData = store.CreateTabular("player-stats");
playerData.AddNumericColumn("score", new double[] { 100, 200, 300 });
playerData.AddStringColumn("name", new[] { "Alice", "Bob", "Charlie" });
// Query the data
var highScorers = playerData.Query()
.Where("score", QueryOp.Gt, 150)
.ToRowIndices();
// Data persists automatically via LiteDB.
// You can force a write to disk if needed:
store.Checkpoint();Setup:
- Create a GameObject in your scene (e.g., "DataCore Manager")
- Add the
DataCoreEditorComponentcomponent - Configure the persistence path (default: "DataCore/")
- Enable auto-save on exit (enabled by default)
- Access from any script using
DataCoreEditorComponent.Instance
using AroAro.DataCore;
// Initialize a store at a specific path
var store = new DataCoreStore("Data/my_database.db");
var t = store.GetOrCreateTabular("my-table");
t.AddNumericColumn("x", new double[] { 1, 2, 3 });
var indices = t.Query().Where("x", QueryOp.Gt, 1.0).ToRowIndices();
// Ensure data is written to disk
store.Checkpoint();
var g = store.GetOrCreateGraph("my-graph");
g.AddNode("a");
g.AddNode("b");
g.AddEdge("a", "b");
store.Checkpoint();- GameObject → Data Core → Create Data Core - Creates a Data Core GameObject with full editor support
- Tools → DataCore → Create Data Core GameObject - Same as above
When you select a Data Core GameObject, the Inspector shows:
- Persistence Configuration: Set storage path and auto-save options
- Datasets Panel: View all datasets with type, size, and delete options
- Create Buttons: Create new Tabular or Graph datasets
- Actions: Save All, Load All
For each dataset, you can see:
- Tabular: Row count, column names
- Graph: Node count, edge count
- Inline Preview: View first 5 rows of tabular data or sample nodes/edges in the Inspector
- Full Preview Window: Click "Open Full Preview" to open a dedicated window showing:
- Tabular Data: First 100 rows with column headers and formatted values
- Graph Data: First 50 nodes and edges with detailed properties
- Scrollable Interface: Handle large datasets with ease
DataCore now includes a comprehensive event system for monitoring dataset changes:
using AroAro.DataCore.Events;
// Subscribe to dataset events
DataCoreEventManager.DatasetCreated += (sender, args) =>
{
// Handle dataset creation
Console.WriteLine($"Dataset created: {args.DatasetName}");
};
DataCoreEventManager.DatasetModified += (sender, args) =>
{
// Handle dataset modifications
Console.WriteLine($"Dataset modified: {args.DatasetName}");
};
// Available events:
// - DatasetCreated: When a new dataset is created
// - DatasetDeleted: When a dataset is deleted
// - DatasetLoaded: When a dataset is loaded from file
// - DatasetSaved: When a dataset is saved to file
// - DatasetModified: When data is modified (add/remove rows, columns, nodes, edges)
// - DatasetQueried: When a query is executedA built-in sample dataset containing housing data for California districts.
Usage:
// Load the sample dataset
CaliforniaHousingDataset.LoadIntoDataCore();
// Access and query the dataset
var store = DataCoreEditorComponent.Instance.GetStore();
var housingData = store.Get<Tabular.TabularData>("california-housing");
var expensiveHouses = housingData.Query()
.Where("median_house_value", Tabular.TabularOp.Gt, 500000)
.ToRowIndices();Features:
- Built-in sample data with housing statistics
- Ready-to-use queries and examples
- Automatic persistence across play mode sessions
DataCore supports lazy loading for better performance with large datasets:
// Register metadata without loading data
store.RegisterMetadata("large-dataset", DataSetKind.Tabular, "path/to/large-dataset.arrow");
// Data is loaded on first access
var dataset = store.Get<Tabular.TabularData>("large-dataset");Import CSV files with optimized performance:
var dataCore = DataCoreEditorComponent.Instance;
dataCore.ImportCsvToTabular("path/to/file.csv", "MyDataset", true, ',');- Tabular Data: Store and query structured data with numeric and string columns
- Graph Data: Create and manipulate graph datasets with nodes and edges
- Persistence: Automatic saving and loading of datasets
- Built-in Algorithms: PageRank, Connected Components, Min-Max Normalization
- Composable Pipelines: Chain algorithms sequentially with automatic data flow
- Algorithm Registry: Discover, register, and look up algorithms at runtime
- Extensible: Create custom algorithms by extending base classes
- Inspector Preview: View dataset contents directly in Unity Inspector
- CSV Import: Import CSV files with automatic type detection
- Event System: Monitor dataset changes and algorithm execution with comprehensive events
- Lazy Loading: Load large datasets only when needed
- Optimized Processing: Efficient batch processing for CSV import
- Memory Efficient: Better handling of large datasets
// Access preview functionality through DataCoreEditorComponent
var component = DataCoreEditorComponent.Instance;
// Preview is automatically available in the Inspector
// Or open full preview window programmatically
DataCorePreviewWindow.ShowWindow(component, "my-dataset");// Subscribe to events
DataCoreEventManager.DatasetCreated += OnDatasetCreated;
DataCoreEventManager.DatasetModified += OnDatasetModified;
void OnDatasetCreated(object sender, DatasetCreatedEventArgs args)
{
// Handle new dataset creation
Console.WriteLine($"New dataset: {args.Dataset.Name}");
}
void OnDatasetModified(object sender, DatasetModifiedEventArgs args)
{
// Handle dataset modifications
Console.WriteLine($"Dataset modified: {args.DatasetName}");
}// Register metadata for delayed loading
store.RegisterMetadata("big-data", DataSetKind.Tabular, "path/to/big-data.arrow");
// Data loads automatically on first access
var data = store.Get<Tabular.TabularData>("big-data");using AroAro.DataCore.Algorithms;
using AroAro.DataCore.Algorithms.Graph;
using AroAro.DataCore.Algorithms.Tabular;Run a single algorithm on a graph:
// Look up a registered algorithm
var algo = AlgorithmRegistry.Default.Get("PageRank");
// Configure parameters and execute
var context = AlgorithmContext.Create()
.WithParameter("dampingFactor", 0.85)
.WithParameter("maxIterations", 100)
.Build();
var result = algo.Execute(myGraph, context);
if (result.Success)
{
var rankedGraph = result.OutputDataset as IGraphDataset;
double score = (double)rankedGraph.GetNodeProperties("nodeA")["pagerank"];
int iterations = (int)result.Metrics["iterations"];
bool converged = (bool)result.Metrics["converged"];
}Normalize tabular data:
var result = new MinMaxNormalizeAlgorithm().Execute(myTable,
AlgorithmContext.Create()
.WithParameter("rangeMin", -1.0)
.WithParameter("rangeMax", 1.0)
.WithParameter("columns", new[] { "score", "age" })
.Build());
var normalizedTable = result.OutputDataset as ITabularDataset;Chain algorithms in a pipeline:
var pipeline = new AlgorithmPipeline("GraphAnalysis")
.Add(new PageRankAlgorithm(), b => b.WithParameter("dampingFactor", 0.9))
.Add(new ConnectedComponentsAlgorithm());
var pipelineResult = pipeline.Execute(myGraph);
// pipelineResult.FinalOutput has both "pagerank" and "componentId" on every node
// pipelineResult.StepResults gives per-step metrics
// pipelineResult.GetAllMetrics() aggregates all metricsCreate a custom algorithm:
public class ShortestPathAlgorithm : GraphAlgorithmBase
{
public override string Name => "ShortestPath";
public override string Description => "Dijkstra's shortest path from a source node.";
public override IReadOnlyList<AlgorithmParameterDescriptor> Parameters { get; } =
new List<AlgorithmParameterDescriptor>
{
new("sourceNode", "Starting node ID", typeof(string), required: true),
};
protected override AlgorithmResult ExecuteGraph(
IGraphDataset input, AlgorithmContext context)
{
string source = context.GetRequired<string>("sourceNode");
// ... your algorithm logic ...
return AlgorithmResult.Succeeded(Name, outputGraph, metrics);
}
}
// Register it
AlgorithmRegistry.Default.Register(new ShortestPathAlgorithm());Monitor algorithm execution via events:
DataCoreEventManager.AlgorithmStarted += (sender, args) =>
Debug.Log($"Algorithm started: {args.AlgorithmName}");
DataCoreEventManager.AlgorithmCompleted += (sender, args) =>
Debug.Log($"Algorithm completed: {args.AlgorithmName} in {args.Duration.TotalMilliseconds}ms");
DataCoreEventManager.PipelineCompleted += (sender, args) =>
Debug.Log($"Pipeline '{args.PipelineName}': {args.StepCount} steps in {args.Duration.TotalMilliseconds}ms");