Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ci): carry-forward flags for codecov #1999

Merged
merged 41 commits into from
Nov 13, 2021
Merged

feat(ci): carry-forward flags for codecov #1999

merged 41 commits into from
Nov 13, 2021

Conversation

aarnphm
Copy link
Contributor

@aarnphm aarnphm commented Nov 13, 2021

We want to carry forward unchanged reports from main branch into PR so that CodeCov doesn't remove unchanged files. Refers to https://docs.codecov.com/docs/carryforward-flags. I also migrate to codecov/github-actions@v2 due to upcoming depreciation

This PR also did some refactoring for our frameworks CI, now we can control dependencies and file run for given frameworks via ./scripts/ci/.frameworks.yml. Developers can quickly run tests from framework_integration_tests.sh like so:

$ ./scripts/ci/framework_integration_tests.sh pytorch

Some assumptions:

  • Developers should define new frameworks under ./scripts/ci/.frameworks.yml (wip getting all keys for frameworks)
  • scripts only takes one arguments, which is the given framework

Given templates for creating new framework:

<framework>:
   root_test_dir: "tests/integration/frameworks" (r"^[\w\-\.]+$")
   is_dir: Or('false', 'true')
   override_fname: string (r"^[\w\-\.]+$")
   dependencies: [] (array of string)
   external_scripts: |
     apt-get install -y curl # example

The expected tests file will have name test_<framework>_impl.py. If developers uses different name (NOT RECOMMEND) then they should pass that name under override_fname

dependencies will accept an array of pypi dependencies that is needed from given tests

 dependencies:
   - "spacy==3.1.2"
   - "pyyaml"
   - "pandas"
   - "-f https://download.pytorch.org/whl/torch_stable.html"
   - "torch==1.9.0+cpu"
   - "torchvision==0.10.0+cpu"
   - "tensorflow==2.6.0"

underlying I just put all of this into a .txt file and then do pip install -r /tmp/requirements.txt

external_scripts can be used if given frameworks need some additional setup, such as onnxmlir:

 external_scripts: |
   export PATH="/workdir/onnx-mlir/build/Debug/lib/:/workdir/onnx-mlir/build/Debug/bin/:${PATH}"
   export PYTHONPATH="${PYTHONPATH}:/workdir/onnx-mlir/build/Debug/lib/"
   apt-get update && apt-get install curl -y

aarnphm added 30 commits November 12, 2021 16:06
@aarnphm
chore(ci): carryforward flags for frameworks
1c20e04 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: tabs
81c3403 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
chore(ci): ci on fork?
b0334d5 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: errors
e345d08 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: errors
2a22eb5 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
feat: flags for CI
c342d8d 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
feat: codecov config
d6f8152 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
style: ignore carry-forward flags in comments
d440853 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
feat: cleanup CI and remove duplication
73b1ba6 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
chore: cleanup
355b11a 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: ci
dec28fc 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: TTY
d1a679c 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
feat(ci): detectron2
8a6a57b 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
style: migrate to new CI
a2a1427 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: codecov actions update
bf40222 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: incremental
b60536e 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: incremental
6f959c9 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: incremental
fcc413c 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: incremental
7da5116 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: incremental
3ef4e29 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: uses CI
6dba65b 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: codecov actions v2 compatible
517eb37 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix(ci/onnxmlir): missing docker as CI dependencies
afb2bbc 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: onnxmlir -y
e1cdf83 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: onnxmlir setup docker
5777028 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: onnxmlir no need to use sudo
34b8235 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: codecov comments
b472fa5 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: codecov comments
417ce63 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: comments codecov
e58cfa3 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: please codecov bot post comments
368cc9b 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: maybe onnxmlir is fixed
f0b13f1 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
fix: debugging onnxmlir
484ba0d 
Signed-off-by: Aaron Pham <[email protected]>
@aarnphm
style: codecov config
02f4851 
Signed-off-by: Aaron Pham <[email protected]>
@codecov
Copy link

codecov bot commented Nov 13, 2021 •
edited
Loading

Codecov Report

Merging #1999 (9eca672) into bentoml-1.0 (6b58b76) will increase coverage by 8.62%.
The diff coverage is n/a.

❗ Current head 9eca672 differs from pull request most recent head 4549d9d. Consider uploading reports for the commit 4549d9d to get more accurate results
Impacted file tree graph

@@               Coverage Diff               @@
##           bentoml-1.0    #1999      +/-   ##
===============================================
+ Coverage        46.69%   55.32%   +8.62%     
===============================================
  Files               97      101       +4     
  Lines             6360     6809     +449     
===============================================
+ Hits              2970     3767     +797     
+ Misses            3390     3042     -348
Flag Coverage Δ
bento-server 33.49% <ø> (?)
frameworks 41.83% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
bentoml/spacy.py 0.00% <0.00%> (-84.29%) ⬇️
bentoml/_internal/bento/pip_pkg.py 0.00% <0.00%> (-23.48%) ⬇️
bentoml/testing/server.py 0.00% <0.00%> (ø)
bentoml/_internal/types.py 63.70% <0.00%> (ø)
bentoml/_internal/typing_extensions/numpy.py 0.00% <0.00%> (ø)
bentoml/_version.py 44.80% <0.00%> (ø)
bentoml/io.py 100.00% <0.00%> (ø)
bentoml/_internal/runner/utils.py 45.14% <0.00%> (+0.48%) ⬆️
bentoml/_internal/service/loader.py 66.00% <0.00%> (+1.00%) ⬆️
bentoml/_internal/runner/runner.py 77.77% <0.00%> (+5.55%) ⬆️
... and 29 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b58b76...4549d9d. Read the comment docs.

@aarnphm aarnphm merged commit b6bce5d into bentoml:bentoml-1.0 Nov 13, 2021
@aarnphm aarnphm deleted the ci/codecov branch November 13, 2021 18:17
aarnphm added a commit to aarnphm/BentoML that referenced this pull request Jul 29, 2022
@aarnphm
feat(ci): carry-forward flags for codecov (bentoml#1999)
6f45fdc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@aarnphm