Use buildings and terrain around you on Unity

The Streetscape Geometry APIs provide the geometry of terrain, buildings, or other structures in a scene. The geometry can be used for occlusion, rendering, or placing AR content via hit-test APIs. Streetscape Geometry data is obtained through Google Street View imagery.

Try the sample

The Geospatial sample app demonstrates how to obtain and render Streetscape Geometries.

Set up the Geospatial API

To use Streetscape Geometry, you'll need to set up the Geospatial API in your project. Follow instructions on Enabling the Geospatial API to set up the Geospatial API.

Enable Streetscape Geometry

The Geospatial API obtains Streetscape Geometry data when the GeospatialMode is set to GeospatialMode.Enabled and StreetscapeGeometryMode is set to StreetscapeGeometryMode.Enabled.

Obtain Streetscape Geometry in an ARCore session

Add an ARStreetscapeGeometryManager component to a GameObject. When Streetscape Geometries are added, updated, or removed, the ARStreetscapeGeometryManager.StreetscapeGeometriesChanged event is triggered.

public Material streetscapeGeometryMaterial;

List<ARStreetscapeGeometry> _addedStreetscapeGeometries = new List<ARStreetscapeGeometry>();
List<ARStreetscapeGeometry> _updatedStreetscapeGeometries = new List<ARStreetscapeGeometry>();
List<ARStreetscapeGeometry> _removedStreetscapeGeometries = new List<ARStreetscapeGeometry>();

public void OnEnable()
{
    StreetscapeGeometryManager.StreetscapeGeometriesChanged +=
        GetStreetscapeGeometry;
}

public void Update() {
  foreach (ARStreetscapeGeometry streetscapegeometry in _addedStreetscapeGeometries)
  {
    GameObject renderObject = new GameObject(
       "StreetscapeGeometryMesh", typeof(MeshFilter), typeof(MeshRenderer));

    if (renderObject)
    {
        renderObject.transform.position = streetscapegeometry.pose.position;
        renderObject.transform.rotation = streetscapegeometry.pose.rotation;
        renderObject.GetComponent<MeshFilter>().mesh = streetscapegeometry.mesh;
        renderObject.GetComponent<MeshRenderer>().material = streetscapeGeometryMaterial;
    }
  }
}

public void OnDisable()
{
    StreetscapeGeometryManager.StreetscapeGeometriesChanged -=
        GetStreetscapeGeometry;
}

private void GetStreetscapeGeometry(ARStreetscapeGeometriesChangedEventArgs eventArgs)
{
    _addedStreetscapeGeometries = eventArgs.Added;
    _updatedStreetscapeGeometries = eventArgs.Updated;
    _removedStreetscapeGeometries = eventArgs.Removed;
}

Understand ARStreetscapeGeometry

ARStreetscapeGeometry contains information about a building:

• ARStreetscapeGeometry.streetscapeGeometryType
  Identifies the StreetscapeGeometry as either terrain or a building.
• ARStreetscapeGeometry.mesh
  Obtain a polygon Mesh that corresponds to this terrain or building.
• ARStreetscapeGeometry.quality
  Provides the quality of the mesh data. Levels of detail are described in the CityGML 2.0 standard.

Building LOD 1

BuildingLOD1 consists of building footprints extruded upwards to a flat top. Building heights may be inaccurate.

Building LOD 2

BuildingLOD2 will have higher fidelity geometry. Mesh walls and roofs will more closely match the building's shape. Smaller features like chimneys or roof vents may still poke outside of the mesh.

Understand Mesh

Mesh is a polygon mesh representing a surface reconstruction of the Streetscape Geometry. See Mesh and MeshRenderer. Note that the normals are not calculated by default; see Mesh.RecalculateNormals() to calculate them.

Attach AR content to a ARStreetscapeGeometry

Use ARAnchorManager.AddAnchor() to create an anchor at a given pose near vertices in ARStreetscapeGeometry.mesh. This anchor will inherit its tracking state from the parent ARStreetscapeGeometry.

Perform a hit-test against ARStreetscapeGeometry

ARRaycastManagerExtensions.RaycastStreetscapeGeometry can be used to hit-test against Streetscape Geometry. If intersections are found, XRRaycastHit contains pose information about the hit location as well as a reference to the ARStreetscapeGeometry which was hit. This Streetscape Geometry can be passed to ARAnchorManager.AddAnchor() to create an anchor attached to it.

Vector2 screenTapPosition = Input.GetTouch(0).position;
List<XRRaycastHit> hitResults = new List<XRRaycastHit>();
if (RaycastManager.RaycastStreetscapeGeometry(screenTapPosition, ref hitResults)){
  ARStreetscapeGeometry streetscapegeometry =
      StreetscapeGeometryManager.GetStreetscapeGeometry(hitResults[0].trackableId);
  if (streetscapegeometry != null)
  {
      ARAnchor anchor = StreetscapeGeometryManager.AttachAnchor(streetscapegeometry, hitResults[0].pose);
  }
}

Enable Geospatial Depth

Geospatial Depth combines Streetscape Geometry with local sensor input to enhance depth data. When Geospatial Depth is enabled, the output depth and raw depth images are modified to include rasterized Streetscape Geometry in addition to locally observed depth. This may improve the accuracy of poses using Depth.