Skip to content

fix: fix search in memory with AnyEmbedding #1696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
JoanFM merged 6 commits into from
Jul 12, 2023
Merged

fix: fix search in memory with AnyEmbedding #1696

JoanFM merged 6 commits into from
Jul 12, 2023

Conversation

@JoanFM
Copy link
Member

@JoanFM JoanFM commented Jul 10, 2023 •
edited
Loading

Allows to do easy search in Memory when AnyEmbedding is used as with TextDoc or ImageDoc:

index = InMemoryExactNNIndex[TextDoc]()

docs = DocList[TextDoc](
    [TextDoc(text='hey', embedding=np.random.rand(128)) for _ in range(200)]
)
index.index(docs)
res = index.find_batched(docs[0:10], search_field='embedding', limit=5)

@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch from e484ff0 to f96171f Compare July 10, 2023 17:01
@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch from f96171f to bc0e40d Compare July 10, 2023 17:11
jupyterjazz
jupyterjazz reviewed Jul 10, 2023
View reviewed changes
Copy link
Contributor

@jupyterjazz jupyterjazz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why was AnyEmbedding failing?

@JoanFM
Copy link
Member Author

JoanFM commented Jul 10, 2023

why was AnyEmbedding failing?

It does not have a computational backend attached, so it cannot stack or do anything on tensors

@JoanFM JoanFM marked this pull request as draft July 10, 2023 22:22
samsja
samsja reviewed Jul 11, 2023
View reviewed changes
Copy link
Member

@samsja samsja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to change

_extract_embedding directly to look at the type to the AnyTensor

@JoanFM
Copy link
Member Author

JoanFM commented Jul 11, 2023

I think it would be better to change

_extract_embedding directly to look at the type to the AnyTensor

and what about the compBackend, is it okey If I return the NumpyBackend for AnyTensor as default?

test: add tests for torch and tf
e7315e8 
Signed-off-by: Joan Fontanals Martinez <[email protected]>
@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch from d36a0b5 to e7315e8 Compare July 11, 2023 07:24
@github-actions github-actions bot added size/m and removed size/s labels Jul 11, 2023
@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch 2 times, most recently from a41e3f7 to 4e013b4 Compare July 11, 2023 10:26
fix: try to pass all mypy checks
2babe51 
Signed-off-by: Joan Fontanals Martinez <[email protected]>
@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch from 4e013b4 to 2babe51 Compare July 11, 2023 10:51
@JoanFM JoanFM requested review from jupyterjazz and samsja July 11, 2023 10:56
@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch from 4d8a20a to 57bf37b Compare July 11, 2023 13:47
test: undo changes in docker compose
bab8099 
Signed-off-by: Joan Fontanals Martinez <[email protected]>
@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch from 57bf37b to bab8099 Compare July 11, 2023 14:30
@JoanFM JoanFM marked this pull request as ready for review July 11, 2023 14:46
@github-actions
Copy link

github-actions bot commented Jul 11, 2023

📝 Docs are deployed on https://ft-anytensor-in-memory-search-fix--jina-docs.netlify.app 🎉

@JoanFM JoanFM merged commit 94a479e into main Jul 12, 2023
@JoanFM JoanFM deleted the anytensor-in-memory-search-fix branch July 12, 2023 07:11
@JoanFM JoanFM linked an issue Jul 13, 2023 that may be closed by this pull request
TextDoc not easily indexable with InMemory and HNSWLib #1695
Closed
6 tasks
@JoanFM JoanFM mentioned this pull request Jul 14, 2023
Merged
8 tasks
@jupyterjazz jupyterjazz mentioned this pull request Jul 18, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@samsja samsja samsja approved these changes

@JohannesMessner JohannesMessner Awaiting requested review from JohannesMessner

@jupyterjazz jupyterjazz Awaiting requested review from jupyterjazz

Assignees

No one assigned

Labels

area/core area/testing area/typing component/array size/m

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

TextDoc not easily indexable with InMemory and HNSWLib

4 participants

@JoanFM @jupyterjazz @samsja