Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
3d45f57
feat: add apply function
Feb 28, 2023
9a07808
test: add benchmark tests
Feb 28, 2023
5709f97
fix: apply
Feb 28, 2023
366b6df
fix: benchmark test
Mar 1, 2023
6e8ff7c
test: benchmark
Mar 2, 2023
89eaf62
fix: apply
Mar 2, 2023
545d9ec
fix: clean up
Mar 2, 2023
d871363
chore: remove benchmark tests from general tests
Mar 2, 2023
8a436ae
chore: fix ci
Mar 2, 2023
c581b58
feat: add threading option and benchmark test
Mar 2, 2023
8bf57fb
test: use both backend options in tests
Mar 2, 2023
0c3524c
feat: add batching to abstract array
Mar 2, 2023
c335895
feat: add apply_batch and _map_batch and tests
Mar 2, 2023
cc23e4e
test: fix load from da
Mar 2, 2023
73c0d84
docs: update docstrings
Mar 2, 2023
b7c2cae
docs: add example for apply
Mar 2, 2023
3eb0c30
fix: mypy
Mar 2, 2023
7c6cb2f
refactor: clean up
Mar 2, 2023
afa5837
refactor: make batch method private
Mar 2, 2023
c69585a
fix: apply
Mar 2, 2023
8a3437a
Test: add for apply batch
Mar 2, 2023
66b78b3
fix: benchmark test increase ndocs
Mar 3, 2023
35e090a
test: clean up
Mar 3, 2023
3019522
test: try to fix
Mar 3, 2023
313d318
test: try to fix test
Mar 3, 2023
0afd5bd
fix: test
Mar 3, 2023
fdcfa23
fix: test
Mar 3, 2023
fc91dbf
fix: apply suggestions from code review
Mar 3, 2023
0d7cd1b
fix: remove print statemetns
Mar 3, 2023
b4c672b
fix: apply samis suggestion
Mar 3, 2023
18a377b
fix: add tests for func da to doc and da to other len da
Mar 3, 2023
245283f
fix: revert last commit
Mar 3, 2023
76fe8b7
test: add len assert
Mar 3, 2023
34b7f9c
test: add assertions
Mar 3, 2023
c7a968d
test: add test to for da extend in batch apply
Mar 3, 2023
6cf8ed2
test: extend with only one doc
Mar 3, 2023
5dc9e6d
test: fix
Mar 3, 2023
d3fc203
fix: test
Mar 3, 2023
45cdc4a
fix: test
Mar 3, 2023
9839602
fix: set docs in apply
Mar 3, 2023
87a93ff
fix: indices
Mar 3, 2023
eeb7fae
fix: indices
Mar 3, 2023
72aaf21
fix: indices
Mar 3, 2023
c0f8029
fix: indices
Mar 3, 2023
9b83c1f
fix:test
Mar 3, 2023
7638d86
fix: mypy
Mar 3, 2023
4a3a290
fix: type hint
Mar 3, 2023
38aae7a
fix: remove apply, only keep map
Mar 3, 2023
01900c9
refactor: map to map_docs
Mar 3, 2023
f6921e0
fix: apply suggestion
Mar 3, 2023
c3fb041
docs: add example usage
Mar 3, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: indices
Signed-off-by: anna-charlotte <[email protected]>
  • Loading branch information
anna-charlotte committed Mar 3, 2023
commit eeb7fae634ca7ee9f043010ce34b89f61139690a
5 changes: 4 additions & 1 deletion docarray/utils/apply.py
Original file line number Diff line number Diff line change
Expand Up @@ -205,12 +205,15 @@ def upper_case_name(da: DocumentArray[MyDoc]) -> DocumentArray[MyDoc]:
be responsible for closing the pool.
:param show_progress: show a progress bar. Defaults to False.
"""
diff = 0
for i, batch in enumerate(
_map_batch(
da, func, batch_size, backend, num_worker, shuffle, pool, show_progress
)
):
start = i * (batch_size + abs(len(batch) - batch_size))
if i == 0:
diff = max(len(batch) - batch_size, 0)
start = i * (batch_size + diff)
stop = (i + 1) * batch_size
if isinstance(batch, BaseDocument):
da[start:stop] = da.__class_getitem__(da.document_type)([batch])
Expand Down