Skip to content

feat(v2): add audio url and predefined document #940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JoanFM merged 46 commits into from
Jan 3, 2023
Merged

feat(v2): add audio url and predefined document #940

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
bebc9d4
feat: add audio url class
Dec 14, 2022
6025c2f
fix: typos
Dec 14, 2022
9a599e5
test: add tests for audio and audio url
Dec 15, 2022
04abdae
feat: add audio url and audio predefined class
Dec 15, 2022
f8d700d
Merge remote-tracking branch 'origin/feat-rewrite-v2' into feat-add-a…
Dec 21, 2022
d58f804
chore: add types-request
Dec 22, 2022
bdf8e88
feat: add audio tensors torch and ndarray
Dec 22, 2022
6572df8
fix: mypy type hints
Dec 22, 2022
9cd4baa
test: empty test file
Dec 22, 2022
b3c1948
test: add more unit and integration tests
Dec 28, 2022
7774181
fix: update audio tensors and audio url
Dec 28, 2022
af840d4
fix: remove print statements
Dec 28, 2022
797f488
docs: add documentation
Dec 28, 2022
8b48a77
refactor: rename test audio py to test audio tensor py
Dec 28, 2022
e135438
fix: typo in torch tensor py
Dec 28, 2022
14fcf6b
feat: add proto stuff to audio tensors
Dec 28, 2022
c623a13
test: add tests for proto and set tensors
Dec 28, 2022
1be8e3f
fix: set tensor to tensor int, since no inplace change
Dec 28, 2022
17786eb
refactor: rename to save to wav file
Dec 28, 2022
97355f7
docs: fix typo
Dec 28, 2022
20e2344
docs: fix docs for save tensor to wav file
Dec 28, 2022
7fc06e1
Merge branch 'feat-rewrite-v2' into feat-add-audio-v2
Dec 28, 2022
b34d783
fix: apply suggestions from code review
Dec 28, 2022
130d8ab
fix: apply suggestions from code review
Dec 29, 2022
2954351
test: fix assertions
Dec 29, 2022
61cb103
fix: move max int multiplication to abstract class
Dec 29, 2022
5943c0f
feat: add ndim method to abstract tensor class and concrete classes
Dec 29, 2022
131c5ff
fix: ndim
Dec 29, 2022
83ef649
fix: revert ndim in abstract tensor and torch tensor and ndarray
Dec 29, 2022
eecca41
fix: mypy checks
Dec 29, 2022
4762c3c
docs: add docstring to n dim
Dec 29, 2022
6948122
refactor: move n dim to abstract tensor and subclasses
Dec 29, 2022
d174087
refactor: make to protobuf abstract, change node to protobuf signature
Dec 29, 2022
3a52303
fix: remove not needed methods
Dec 29, 2022
a0be12e
fix: change remote audio file to file from github
Dec 30, 2022
9623d29
fix: raw content from remote file
Dec 30, 2022
6efdcf2
fix: path to github remote file
Dec 30, 2022
5026543
refactor: tensor field name to proto field name
Jan 2, 2023
703de43
test: remove redundant test in test audio tensor
Jan 2, 2023
83ece31
fix: load audio url to audio ndarray instead of np ndarray
Jan 2, 2023
de079e2
refactor: move n dim to computational backend
Jan 2, 2023
2ef1350
docs: update docstrings for audio tensors
Jan 3, 2023
d51d38e
feat: make dtype in audiourl load optional
Jan 3, 2023
3901cfa
Merge branch 'feat-rewrite-v2' into feat-add-audio-v2
Jan 3, 2023
a571898
test: fix document refactor and ndarray import
Jan 3, 2023
71af630
fix: fix mypy check
Jan 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: update audio tensors and audio url 
Signed-off-by: anna-charlotte <[email protected]>
  • Loading branch information
anna-charlotte committed Dec 28, 2022
commit 7774181d6c9768d31c8a8b400e67a5fca6156d49
5 changes: 4 additions & 1 deletion docarray/typing/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from docarray.typing.id import ID
from docarray.typing.tensor import NdArray, Tensor
from docarray.typing.tensor.audio import AudioNdArray
from docarray.typing.tensor.embedding import Embedding
from docarray.typing.url import (
AnyUrl,
Expand All @@ -11,6 +12,7 @@
)

__all__ = [
'AudioNdArray',
'NdArray',
'Embedding',
'ImageUrl',
Expand All @@ -29,5 +31,6 @@
pass
else:
from docarray.typing.tensor import TorchEmbedding, TorchTensor # noqa: F401
from docarray.typing.tensor.audio.audio_torch_tensor import AudioTorchTensor # noqa

__all__.extend(['TorchEmbedding', 'TorchTensor'])
__all__.extend(['AudioTorchTensor', 'TorchEmbedding', 'TorchTensor'])
12 changes: 12 additions & 0 deletions docarray/typing/tensor/audio/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
from docarray.typing.tensor.audio.audio_ndarray import AudioNdArray

__all__ = ['AudioNdArray']

try:
import torch # noqa: F401
except ImportError:
pass
else:
from docarray.typing.tensor.audio.audio_torch_tensor import AudioTorchTensor # noqa

__all__.extend(['AudioTorchTensor'])
23 changes: 22 additions & 1 deletion docarray/typing/tensor/audio/audio_ndarray.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,29 @@
import wave
from typing import BinaryIO, TypeVar, Union
from typing import TYPE_CHECKING, BinaryIO, TypeVar, Union

from docarray.typing import NdArray

T = TypeVar('T', bound='AudioNdArray')

if TYPE_CHECKING:
from docarray.proto import NodeProto


class AudioNdArray(NdArray):
""" """

def _to_node_protobuf(self: T, field: str = 'ndarray') -> 'NodeProto':
"""Convert itself into a NodeProto protobuf message. This function should
be called when the Document is nested into another Document that need to be
converted into a protobuf
:param field: field in which to store the content in the node proto
:return: the nested item protobuf message
"""
from docarray.proto import NodeProto

nd_proto = self.to_protobuf()
return NodeProto(**{field: nd_proto})

def save_audio_tensor_to_file(
self: 'T',
file_path: Union[str, BinaryIO],
Expand All @@ -26,7 +41,11 @@ def save_audio_tensor_to_file(

# Convert to (little-endian) 16 bit integers.
max_int16 = 2**15
print(f"self = {self}")
print(f"self.__class__ = {self.__class__}")

tensor = (self * max_int16).astype('<h')
print(f"tensor = {tensor}")
n_channels = 2 if self.ndim > 1 else 1

with wave.open(file_path, 'w') as f:
Expand All @@ -35,4 +54,6 @@ def save_audio_tensor_to_file(
# 2 bytes per sample.
f.setsampwidth(sample_width)
f.setframerate(sample_rate)
print(f"tensor = {tensor}")
print(f"tensor.tobytes() = {tensor.tobytes()}")
f.writeframes(tensor.tobytes())
14 changes: 5 additions & 9 deletions docarray/typing/tensor/audio/audio_torch_tensor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,7 @@
import wave
from typing import BinaryIO, TypeVar, Union

import numpy as np

from docarray.typing import TorchTensor
from docarray.typing.tensor.torch_tensor import metaTorchAndNode
from docarray.typing.tensor.torch_tensor import TorchTensor, metaTorchAndNode

T = TypeVar('T', bound='AudioTorchTensor')

Expand All @@ -26,17 +23,16 @@ def save_audio_tensor_to_file(
:param sample_rate: sampling frequency
:param sample_width: sample width in bytes
"""
np_self: np.ndarray = self.cpu().detach().numpy()
import torch

# Convert to (little-endian) 16 bit integers.
max_int16 = 2**15
tensor = (np_self * max_int16).astype('<h')
n_channels = 2 if np_self.ndim > 1 else 1
tensor = torch.tensor(self * max_int16, dtype=torch.int16)
n_channels = 2 if self.ndim > 1 else 1

with wave.open(file_path, 'w') as f:
# 2 Channels.
f.setnchannels(n_channels)
# 2 bytes per sample.
f.setsampwidth(sample_width)
f.setframerate(sample_rate)
f.writeframes(tensor.tobytes())
f.writeframes(tensor.cpu().detach().numpy().tobytes())
27 changes: 24 additions & 3 deletions docarray/typing/url/audio_url.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,20 @@
import wave
from typing import TYPE_CHECKING, TypeVar, Union
from typing import TYPE_CHECKING, Any, Type, TypeVar, Union

import numpy as np

from docarray.typing.url.any_url import AnyUrl

if TYPE_CHECKING:
from pydantic import BaseConfig
from pydantic.fields import ModelField

from docarray.proto import NodeProto

T = TypeVar('T', bound='AudioUrl')

AUDIO_FILE_FORMATS = ['wav']


class AudioUrl(AnyUrl):
"""
Expand All @@ -28,9 +33,25 @@ def _to_node_protobuf(self: T) -> 'NodeProto':

return NodeProto(audio_url=str(self))

@classmethod
def validate(
cls: Type[T],
value: Union[T, np.ndarray, Any],
field: 'ModelField',
config: 'BaseConfig',
) -> T:
url = super().validate(value, field, config) # basic url validation
has_audio_extension = any(url.endswith(ext) for ext in AUDIO_FILE_FORMATS)
if not has_audio_extension:
raise ValueError(
f'Audio URL must have one of the following extensions:'
f'{AUDIO_FILE_FORMATS}'
)
return cls(str(url), scheme=None)

def load(self: T) -> np.ndarray:
Copy link
Contributor Author

@anna-charlotte anna-charlotte Jan 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@samsja Should I return a np.ndarray here, or rather AudioNdArray or AudioTorchTensor?

Copy link
Member

@JohannesMessner JohannesMessner Jan 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say AudioNdArray.
Normally we just load the np.ndarray, bc the framework specific conversion can happen when putting it back into the document, and having it as np.ndarray makes the least assumptions about the sorrounding code.
In this case, since there are actual audio features that come with the array, I would go with AudioNdArray. It can still be treated like a normal np.ndarray, but bring the aforementioned features.

But just verify in a test that setting a AudioNdArray to a field with type AudioTorchArray actually works without issue?

Copy link
Contributor Author

@anna-charlotte anna-charlotte Jan 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But just verify in a test that setting a AudioNdArray to a field with type AudioTorchArray actually works without issue

Do you mean like this?

def test_load_audio_url_to_audio_torch_tensor(file_url):
    class MyAudioDoc(Document):
        audio_url: AudioUrl
        tensor: Optional[AudioTorchTensor]

    doc = MyAudioDoc(audio_url=file_url)
    doc.tensor = doc.audio_url.load()

    assert isinstance(doc.tensor, np.ndarray)
    assert isinstance(doc.tensor, AudioNdArray)

Copy link
Contributor Author

@anna-charlotte anna-charlotte Jan 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I moved it to the computational backends

Copy link
Member

@samsja samsja Jan 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great I like it better like this

"""
Load the data from the url into a numpy.ndarray audio tensor.
Load the data from the url into a numpy.ndarray.

EXAMPLE USAGE

Expand All @@ -46,7 +67,7 @@ class MyDoc(Document):
audio_url: AudioUrl


doc = MyDoc(mesh_url="toydata/hello.wav")
doc = MyDoc(audio_url="toydata/hello.wav")

audio_tensor = doc.audio_url.load()
assert isinstance(audio_tensor, np.ndarray)
Expand Down