feat(v2): display mesh and pointcloud #1113

anna-charlotte · 2023-02-09T07:56:27Z

Goals:

Add display methods to PointCloud 3D and Mesh3D urls and the corresponding tensors. For the tensors, first add classes to store those in: VerticesAndFaces for mesh (to store vertices: AnyTensor, faces: AnyTensor), PointAndColors for point cloud (to store points: AnyTensor, colors: AnyTensor).

For PointCloud3D you can call .display() on its url or tensors:

doc = PointCloud3D(url='tests/toydata/tetrahedron.obj')
doc.tensors = doc.url.load(samples=10000)
doc.tensors.display()
# or via url
doc.url.display()

For Mesh3D you can call .display() on its url or tensors:

doc = Mesh3D(url='tests/toydata/tetrahedron.obj')
doc.tensors = doc.url.load()
doc.tensors.display()
# or via url
doc.url.display()

check and update documentation, if required. See guide

anna-charlotte · 2023-02-09T10:34:01Z

Problem with poetry:
To display a mesh, we need to install trimesh[easy] instead of only trimesh, because we need scipy and pyglet, too. There is a problem when adding this with poetry though, as the dependency scipy does not support the python versions 3.7 and 3.12:

Because scipy (1.10.0) requires Python <3.12,>=3.8

I guess therefore for now one needs to install trimesh[easy] via pip, such as we do for tensorflow as of right now, yes?

docarray/documents/mesh.py

docarray/documents/point_cloud.py

anna-charlotte · 2023-02-10T09:19:29Z

After talking to @samsja we think that it would be nice to not have the .display() method on Document level, but rather type level, so that you call pc_doc.url.display() or pc_doc.tensor.display() instead of pc_doc.display(display_from='url'). For Image, Audio, Video this should work, since we have ImageTensor, ImageUrl and so on.
For Mesh3D however we don't have such a tensor type, and also the display depends on both vertices and face. Same goes for the pointcloud display, which depends on a vertices tensor and optional color tensor.
We could introduce documents for those instead of type, something like:

class VerticesAndFaces(BaseDocument):
    vertices: AnyTensor
    faces: AnyTensor

    def display():
        """display mesh from tensors"""

@JohannesMessner do you have any thought on this?

JohannesMessner · 2023-02-10T14:17:30Z

After talking to @samsja we think that it would be nice to not have the .display() method on Document level, but rather type level, so that you call pc_doc.url.display() or pc_doc.tensor.display() instead of pc_doc.display(display_from='url'). For Image, Audio, Video this should work, since we have ImageTensor, ImageUrl and so on. For Mesh3D however we don't have such a tensor type, and also the display depends on both vertices and face. Same goes for the pointcloud display, which depends on a vertices tensor and optional color tensor. We could introduce documents for those instead of type, something like:
class VerticesAndFaces(BaseDocument):
    vertices: AnyTensor
    faces: AnyTensor

    def display():
        """display mesh from tensors"""
@JohannesMessner do you have any thought on this?

fully agree with putting it at the type level.

I am not 100% convinced with having "magic" documents such as VerticesAndFaces that add display, but probably it's the only way, right?

JohannesMessner · 2023-02-10T14:20:14Z

docarray/typing/url/url_3d/mesh_url.py

-        return Mesh3DLoadResult(vertices=vertices, faces=faces)
+        return VerticesAndFaces(vertices=vertices, faces=faces)
+
+    def display(self) -> None:


url.display feels a bit weird to me, because in this case it doesn't really display the url, it displays the thing the url points to.
And to do that, it has to load from that url under the hood.

So is it necessary to expose this? Why not url.load().display()?

yeah i see your point, but its not quite the same, when displaying it from url as it is now, we can display it with color because we just call .show() on the trimesh instance. For url.load().display() we extract the vertices and faces information but as of right now there is no way to extract the color information. Therefore this displays without colors. I think it would be nice to keep the color display if url content includes this information.
What do you think?

but this is true that it is weird to display some data (the color) that we cannot load in our tools

yes true, can be a bit misleading or confusing, too. do u suggest to remove it then for the url, and do the url.load().display() if someone doesn't want to load it into the tensors? I think this won't change on trimesh side any time soon, to easily extract color information.

let me think about it

so what's the conclusion here? keep it as it is? I see the point of being able to show colors, so I don't have a strong opinion anymore

@anna-charlotte what did you decided ?

I would keep it like this, for the colors and also to have the display method for all urls

docarray/documents/mesh/vertices_and_faces.py

docarray/typing/url/url_3d/point_cloud_url.py

Signed-off-by: anna-charlotte <[email protected]>

samsja · 2023-02-15T15:23:24Z

docarray/typing/url/url_3d/mesh_url.py

+        Plot mesh in notebook from url.
+        This loads the Trimesh instance of the 3D mesh, and then displays it.
+        """
+        from IPython.display import display


what ahppened if we are not inside IPython ?

I think for mesh and point cloud it would still open a pyglet window and display it, but for the other IPython displays it would just print something like '< IPython.display.Audio obj >'.

poetry.lock

Signed-off-by: anna-charlotte <[email protected]>

github-actions · 2023-02-16T07:40:26Z

📝 Docs are deployed on https://ft-feat-display-mesh-v2--jina-docs.netlify.app 🎉

anna-charlotte changed the base branch from main to feat-rewrite-v2 February 9, 2023 07:56

anna-charlotte closed this Feb 9, 2023

anna-charlotte reopened this Feb 9, 2023

github-actions bot added size/s area/core labels Feb 9, 2023

anna-charlotte mentioned this pull request Feb 9, 2023

Meta: DocArray v2 Roadmap #780

Closed

47 tasks

anna-charlotte force-pushed the feat-display-mesh-v2 branch from 317d3f7 to d2ed66a Compare February 9, 2023 10:02

github-actions bot added size/l area/setup and removed size/s labels Feb 9, 2023

github-actions bot added area/testing area/typing labels Feb 9, 2023

anna-charlotte force-pushed the feat-display-mesh-v2 branch from 2ad2c4f to c8e8c90 Compare February 9, 2023 16:06

anna-charlotte commented Feb 10, 2023

View reviewed changes

docarray/documents/mesh.py Outdated Show resolved Hide resolved

docarray/documents/point_cloud.py Outdated Show resolved Hide resolved

anna-charlotte requested review from JohannesMessner and samsja and removed request for samsja February 10, 2023 08:14

github-actions bot added size/m and removed size/l labels Feb 10, 2023

anna-charlotte marked this pull request as ready for review February 10, 2023 14:08

JohannesMessner requested changes Feb 10, 2023

View reviewed changes

samsja reviewed Feb 10, 2023

View reviewed changes

docarray/typing/url/url_3d/point_cloud_url.py Outdated Show resolved Hide resolved

anna-charlotte mentioned this pull request Feb 13, 2023

v2: plot image, video, audio, mesh #1127

Closed

2 tasks

anna-charlotte linked an issue Feb 13, 2023 that may be closed by this pull request

v2: plot image, video, audio, mesh #1127

Closed

2 tasks

anna-charlotte force-pushed the feat-display-mesh-v2 branch from e0e8d1a to fbb652e Compare February 15, 2023 13:27

feat: display mesh and pointcloud

943d636

Signed-off-by: anna-charlotte <[email protected]>

anna-charlotte added 16 commits February 15, 2023 16:18

fix: mypy

c91ab6c

Signed-off-by: anna-charlotte <[email protected]>

fix: add display from param to mesh and pc display

e979c32

Signed-off-by: anna-charlotte <[email protected]>

fix: clean up

21fb8ab

Signed-off-by: anna-charlotte <[email protected]>

fix: mypy

75b33d8

Signed-off-by: anna-charlotte <[email protected]>

fix: move display from url to mesh and pc url classes

7890194

Signed-off-by: anna-charlotte <[email protected]>

chore: remove pyglet dependency

0129c3a

Signed-off-by: anna-charlotte <[email protected]>

chore: update pyproject toml

322a718

Signed-off-by: anna-charlotte <[email protected]>

refactor: copy is notebook function from hubble sdk

05d8461

Signed-off-by: anna-charlotte <[email protected]>

fix: introduce vertices and faces doc

0244816

Signed-off-by: anna-charlotte <[email protected]>

fix: introduce points and colors class for point cloud

255795c

Signed-off-by: anna-charlotte <[email protected]>

fix: mypy and tests

db42712

Signed-off-by: anna-charlotte <[email protected]>

docs: add display example to docs

dca04ce

Signed-off-by: anna-charlotte <[email protected]>

fix: apply johannes suggestion from review

788f834

Signed-off-by: anna-charlotte <[email protected]>

fix: apply samis suggestion

57fb1e1

Signed-off-by: anna-charlotte <[email protected]>

docs: update docstring

3a8dc5e

Signed-off-by: anna-charlotte <[email protected]>

fix: only display in notebook

38f771d

Signed-off-by: anna-charlotte <[email protected]>

anna-charlotte force-pushed the feat-display-mesh-v2 branch from 1dba1db to 38f771d Compare February 15, 2023 15:18

docs: update docstring

8c31318

Signed-off-by: anna-charlotte <[email protected]>

samsja reviewed Feb 15, 2023

View reviewed changes

poetry.lock Outdated Show resolved Hide resolved

anna-charlotte commented Feb 15, 2023

View reviewed changes

poetry.lock Outdated Show resolved Hide resolved

anna-charlotte added 2 commits February 15, 2023 16:41

chore: get poetry lock file from feat rewrite v2

bbef411

Signed-off-by: anna-charlotte <[email protected]>

docs: update docstrings

b376ddd

Signed-off-by: anna-charlotte <[email protected]>

samsja approved these changes Feb 15, 2023

View reviewed changes

Merge branch 'feat-rewrite-v2' into feat-display-mesh-v2

c623f9c

JohannesMessner approved these changes Feb 16, 2023

View reviewed changes

anna-charlotte merged commit 96cd383 into feat-rewrite-v2 Feb 16, 2023

anna-charlotte deleted the feat-display-mesh-v2 branch February 16, 2023 08:56

anna-charlotte mentioned this pull request Feb 20, 2023

DocArray v2 third alpha release note #1153

Closed

feat(v2): display mesh and pointcloud #1113

feat(v2): display mesh and pointcloud #1113

Uh oh!

Conversation

anna-charlotte commented Feb 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anna-charlotte commented Feb 9, 2023

Uh oh!

Uh oh!

Uh oh!

anna-charlotte commented Feb 10, 2023

Uh oh!

JohannesMessner commented Feb 10, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

anna-charlotte commented Feb 9, 2023 •

edited

Loading