Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve upload and download features #1310

Merged
merged 116 commits into from
Apr 21, 2021
Merged

Improve upload and download features #1310

merged 116 commits into from
Apr 21, 2021

Conversation

Hironsan
Copy link
Member

Upload

  • Support a large file upload
  • Support a folder upload
  • Support multiple files upload
  • Support asynchronous upload
  • Enable to show upload progress

Ingestion

  • Support batch import to speed up the process
  • Support file validation
  • Import as much of the file's content as possible
  • Provide feedback on which file and what line is wrong and why.
  • Support 90 over encodings
  • Support auto encoding detection
  • Support saving filename

File format

  • Expand the available formats
  • Support for specifying columns for labels and text
  • Support for specifying schema of CoNLL(IOB2, IOE2, IOBES, BILOU)
  • Support for specifying separator(CSV, CoNLL)

Download

  • Support a large file download
  • Support asynchronous download
  • Support zip download
  • Support JSON download

Related issues

fix #1299
fix #1292
fix #1291
fix #1286
fix #1267
fix #1265
fix #1257
fix #1202
fix #1187
fix #1046
fix #1015
fix #999
fix #972
fix #929
fix #521
fix #482
fix #454
fix #295

@Hironsan
Install filepond and vue-filepond
abf0a7a
@Hironsan
Install django-drf-filepond
104a892
@Hironsan
Add filepond variables to settings.py
2f2efa2
@Hironsan
Add filepond endpoints to urls.py
86e1566
@Hironsan
Add upload experiment page
ad29f88
@Hironsan
Install celery dependencies
e15910d
@Hironsan
Add celery settings
1e93d97
@Hironsan
Add example celery tasks
e25a490
@Hironsan
Add datasets for upload
55f73e0
@Hironsan
Add encoding as a parameter
abb5d6d
@Hironsan
Update dataset
e869e7e
@Hironsan
Add test cases for dataset
bc294cb
@Hironsan
Add label classes
84f5a28
@Hironsan
Update dataset classes to use label class
10acca1
@Hironsan
Update test cases
5b67cd3
@Hironsan
Add data classes
4536151
@Hironsan
Update dataset to use data class
25cde88
@Hironsan
Update test cases
32781f3
@Hironsan
Add dict method to Record
4248596
@Hironsan
Add append_field function
ca8cddf
@Hironsan
Add replace method to label
d0fea9c
@Hironsan
Add labels
afb97b5
@Hironsan
Update dataset record
c2d1470
@Hironsan
Add factories
c53b243
@Hironsan
Add data factory
b0b253e
@Hironsan
Add inject_data task
c629ecd
@Hironsan
Remove empty label name
02d1a22
@Hironsan
Update injest_data
60fef6e
@Hironsan
Update injest_data to avoid unique constraint error
aa57786
@Hironsan
Apply isort
5003518
Hironsan and others added 27 commits April 20, 2021 10:38
@Hironsan
Move download view
25de2a6
@Hironsan
Move upload view
c4f27f9
@Hironsan
Enable to pass task id to download api
2171426
@Hironsan
Add domain object for downloading dataset
0a84f4a
@Hironsan
Add repositories for downloading dataset
6b66e53
@Hironsan
Add services for downloading dataset
7ee9a0b
@Hironsan
Add services for download to the plugin
7d0559f
@Hironsan
Enable to download dataset
1ea31ba
@Hironsan
Remove polling on before destroy
16d52fd
@Hironsan
Avoid deep directory structure for doanloaded zip
846e588
@Hironsan
Support JSON download
7cfa585
@Hironsan
Show overlay progress circle
ae9cdc2
@Hironsan
Support only approved option for downloading dataset
bf3817a
@Hironsan
Merge branch 'master' into v1.3.0
1a6e762
@Hironsan
Enable to reset download option
7183014
@Hironsan
Remove unnecessary code
f84997a
@Hironsan
Comment out unnecessary testings
769a06c
@Hironsan
Fix seq2seq repository
d5db36b
@Hironsan
Merge branch 'master' into v1.3.0
ffce583
@Hironsan
Add a merge migration
be8dd25
@Hironsan
Reduce exported user if the project is collaborative
e53077d
@Hironsan
Enable to export a doc without label
e5344d6
@Hironsan
Rename app dir to backend for pip installation
a3a054c
@Hironsan
Update workflows
f520c29
@Hironsan
Fix export data without labels
38de295
@Hironsan
Fix dev-celery.sh
fbb48df
@Hironsan
Update Dockerfile to avoid permission denied
bab7978
@Hironsan Hironsan merged commit 29dfa3a into master Apr 21, 2021
@Hironsan Hironsan deleted the v1.3.0 branch April 21, 2021 07:20
@Hironsan Hironsan mentioned this pull request Nov 12, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@Hironsan @kuraga