Skip to content

Commit

Permalink
Merge branch 'main' into feature/ci-check-for-memory-leaks
Browse files Browse the repository at this point in the history
  • Loading branch information
@wochinge
wochinge committed Apr 6, 2021
2 parents 7250604 + 493e520 commit 0b7bd61
Show file tree
Hide file tree
Showing 102 changed files with 1,897 additions and 684 deletions.
17 changes: 17 additions & 0 deletions .github/matchers/flake8-error-matcher.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"problemMatcher": [
{
"owner": "flake8-error",
"severity": "error",
"pattern": [
{
"regexp": "^([^:]+):(\\d+):(\\d+):\\s+([DCFNWE]\\d+\\s+.+)$",
"file": 1,
"line": 2,
"column": 3,
"message": 4
}
]
}
]
}
205 changes: 201 additions & 4 deletions .github/workflows/ci-model-regression-on-schedule.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,6 @@ jobs:
poetry install --extras full
make install
poetry run python -m spacy download de_core_news_md
poetry run python -m spacy link --force de_core_news_md de
- name: Validate that GPUs are working
if: steps.set_dataset_config_vars.outputs.is_dataset_exists == 'true' && steps.set_dataset_config_vars.outputs.is_config_exists == 'true'
Expand All @@ -172,6 +171,7 @@ jobs:
env:
TFHUB_CACHE_DIR: ~/.tfhub_cache/
OMP_NUM_THREADS: 1
TF_DETERMINISTIC_OPS: 1 # Run GPUs in a deterministic mode
run: |-
poetry run rasa --version
Expand Down Expand Up @@ -221,6 +221,56 @@ jobs:
name: report.json
path: ./report.json

- name: Pick an assignee randomly
if: failure()
id: pick_assignee
run: |
ASSIGNEES_LIST=("dakshvar22" "Ghostvv" "JEM-Mosig" "tttthomasssss" "samsucik" "koernerfelicia" "kedz" "aeshky")
# Randomly select an item from assignees list
ASSIGNEE=${ASSIGNEES_LIST[ $(( RANDOM % ${#ASSIGNEES_LIST[@]} )) ]}
echo "::set-output name=assignee::$ASSIGNEE"
- name: Open GitHub Issue 📬
if: failure()
uses: rishabhgupta/git-action-issue@v2
id: open_issue
with:
token: ${{ secrets.GITHUB_TOKEN }}
title: "Scheduled Model Regression Test Failed"
body: "*This PR is automatically created by the Scheduled Model Regression Test workflow. Checkout the Github Action Run [here](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}).* <br> --- <br> **Description of Problem:** <br> Scheduled Model Regression Test failed. <br> **Configuration**: ${{ matrix.config }} <br> **Dataset**: ${{ matrix.dataset}}"
assignees: ${{ steps.pick_assignee.outputs.assignee }}

- name: Add Label to the Issue
if: failure()
id: add_label
env:
LABEL: "type:bug :bug:"
run: |
# Get issue number
ISSUE_NUMBER=$(echo '${{ steps.open_issue.outputs.issue }}' | jq -r '.number')
echo "::set-output name=issue_number::$ISSUE_NUMBER"
# Add label to the issue
curl \
-X POST \
-H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/$GITHUB_REPOSITORY/issues/${ISSUE_NUMBER}/labels \
-d '{"labels":["${{ env.LABEL }}"]}'
- name: Assign Issue to Project Column
if: failure()
run: |
ISSUE_ID=$(echo '${{ steps.open_issue.outputs.issue }}' | jq -r '.id')
# Add this issue to a project column
curl \
-X POST \
-s -H "Authorization: token ${{ secrets.RASABOT_ASSIGN_ISSUE_RESEARCH_BOARD_TOKEN }}" \
-H 'Accept: application/vnd.github.inertia-preview+json' \
-d "{\"content_type\": \"Issue\", \"content_id\": $ISSUE_ID}" \
"https://api.github.com/projects/columns/${{ secrets.RESEARCH_BOARD_INBOX_COLUMN_ID }}/cards"
- name: Notify Slack of Failure 😱
if: failure()
uses: 8398a7/action-slack@v3
Expand All @@ -229,12 +279,11 @@ jobs:
fields: workflow,job,commit,repo,ref,author,took
custom_payload: |
{
text: "Model regression tests CI",
attachments: [{
fallback: 'fallback',
color: '${{ job.status }}' === 'success' ? 'good' : '${{ job.status }}' === 'failure' ? 'danger' : 'warning',
title: `${process.env.AS_WORKFLOW}`,
text: 'Scheduled model regression test failed',
text: 'Scheduled model regression test failed :no_entry:️',
fields: [{
title: 'Configuration',
value: '${{ matrix.config }}',
Expand All @@ -245,10 +294,158 @@ jobs:
value: '${{ matrix.dataset }}',
short: false
},
{
title: 'GitHub Issue',
value: `https://github.com/${{ github.repository }}/issues/${{ steps.add_label.outputs.issue_number }}`,
short: false
},
{
title: 'GitHub Action',
value: `https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}`,
short: false
}],
actions: [{
}]
}]
}
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_CI_MODEL_REGRESSION_TEST }}

analyse_performance:
name: Analyse Performance
runs-on: ubuntu-latest
if: always()
needs:
- deploy_runner_gpu
- model_regression_test_gpu

steps:
- name: Checkout
uses: actions/checkout@v2

- name: Download an artifact with the results for the schedule version
run: |
# Get ID of the schedule workflow
SCHEDULE_ID=$(curl -X GET -s -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' -H "Accept: application/vnd.github.v3+json" \
"https://api.github.com/repos/${{ github.repository }}/actions/workflows" \
| jq -r '.workflows[] | select(.name == "${{ github.workflow }}") | select(.path | test("schedule")) | .id')
ARTIFACT_URL=$(curl -s -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' -H "Accept: application/vnd.github.v3+json" \
"https://api.github.com/repos/${{ github.repository }}/actions/workflows/${SCHEDULE_ID}/runs?event=schedule&status=completed&branch=main&per_page=1" | jq -r .workflow_runs[0].artifacts_url)
DOWNLOAD_URL=$(curl -s -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' -H "Accept: application/vnd.github.v3+json" "${ARTIFACT_URL}" \
| jq -r '.artifacts[] | select(.name="report.json") | .archive_download_url')
# Download the artifact
curl -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' -LJO -H "Accept: application/vnd.github.v3+json" $DOWNLOAD_URL
# Unzip and change name
unzip report.json.zip && mv report.json report_main.json
- name: Download the report
uses: actions/download-artifact@v2
with:
name: report.json

- name: Download gomplate
run: |-
sudo curl -o /usr/local/bin/gomplate -sSL https://github.com/hairyhenderson/gomplate/releases/download/v3.6.0/gomplate_linux-amd64
sudo chmod +x /usr/local/bin/gomplate
- name: Analyse Performance 🔍
id: performance
run: |
OUTPUT="$(gomplate -d data=report.json -d results_main=report_main.json -f .github/templates/model_regression_test_results.tmpl)"
OUTPUT="${OUTPUT//$'\n'/'%0A'}"
OUTPUT="${OUTPUT//$'\r'/'%0D'}"
echo "::set-output name=report_description::${OUTPUT}"
# Remove all (-0.00) from OUTPUT.
# We don't want to count (-0.00) as a performance drop
OUTPUT=${OUTPUT//(-0.00)/}
# Any (negatives) found means there is a performance drops.
if [[ "${OUTPUT}" =~ \(-[0-9.]+\) ]]; then
echo "Some test performance scores decreased."
echo "::set-output name=is_dropped::true"
else
echo "No test performance drops."
echo "::set-output name=is_dropped::false"
fi
- name: Pick an assignee randomly
if: steps.performance.outputs.is_dropped == 'true'
id: pick_assignee
run: |
ASSIGNEES_LIST=("dakshvar22" "Ghostvv" "JEM-Mosig" "tttthomasssss" "samsucik" "koernerfelicia" "kedz" "aeshky")
# Randomly select an item from assignees list
ASSIGNEE=${ASSIGNEES_LIST[ $(( RANDOM % ${#ASSIGNEES_LIST[@]} )) ]}
echo "::set-output name=assignee::$ASSIGNEE"
- name: Open GitHub Issue 📬
if: steps.performance.outputs.is_dropped == 'true'
uses: rishabhgupta/git-action-issue@v2
id: open_issue
with:
token: ${{ secrets.GITHUB_TOKEN }}
title: "Scheduled Model Regression Test Performance Drops"
body: "*This PR is automatically created by the Scheduled Model Regression Test workflow. Checkout the Github Action Run [here](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}).* <br> --- <br> **Description of Problem:** <br> Some test performance scores **decreased**. Please look at the following table for more details. <br> ${{ steps.performance.outputs.report_description }}"
assignees: ${{ steps.pick_assignee.outputs.assignee }}

- name: Add Label to the Issue
if: steps.performance.outputs.is_dropped == 'true'
id: add_label
env:
LABEL: "type:bug :bug:"
run: |
# Get issue number
ISSUE_NUMBER=$(echo '${{ steps.open_issue.outputs.issue }}' | jq -r '.number')
echo "::set-output name=issue_number::$ISSUE_NUMBER"
# Add label to the issue
curl \
-X POST \
-H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/$GITHUB_REPOSITORY/issues/${ISSUE_NUMBER}/labels \
-d '{"labels":["${{ env.LABEL }}"]}'
- name: Assign Issue to Project Column
if: steps.performance.outputs.is_dropped == 'true'
run: |
ISSUE_ID=$(echo '${{ steps.open_issue.outputs.issue }}' | jq -r '.id')
# Add this issue to a project column
curl \
-X POST \
-s -H "Authorization: token ${{ secrets.RASABOT_ASSIGN_ISSUE_RESEARCH_BOARD_TOKEN }}" \
-H 'Accept: application/vnd.github.inertia-preview+json' \
-d "{\"content_type\": \"Issue\", \"content_id\": $ISSUE_ID}" \
"https://api.github.com/projects/columns/${{ secrets.RESEARCH_BOARD_INBOX_COLUMN_ID }}/cards"
- name: Notify Slack when Performance Drops 💬
if: steps.performance.outputs.is_dropped == 'true'
uses: 8398a7/action-slack@v3
with:
status: custom
fields: workflow,job,commit,repo,ref,author,took
custom_payload: |
{
attachments: [{
fallback: 'fallback',
color: 'danger',
title: `${process.env.AS_WORKFLOW}`,
text: 'Scheduled model regression test performance drops :chart_with_downwards_trend:',
fields: [{
title: 'GitHub Issue',
value: `https://github.com/${{ github.repository }}/issues/${{ steps.add_label.outputs.issue_number }}`,
short: false
},
{
title: 'GitHub Action',
value: `https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}`,
short: true
short: false
}],
actions: [{
}]
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/ci-model-regression.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,6 @@ jobs:
poetry install --extras full
make install
poetry run python -m spacy download de_core_news_md
poetry run python -m spacy link --force de_core_news_md de
- name: Validate that GPUs are working
if: steps.set_dataset_config_vars.outputs.is_dataset_exists == 'true' && steps.set_dataset_config_vars.outputs.is_config_exists == 'true'
Expand Down Expand Up @@ -400,7 +399,6 @@ jobs:
poetry install --extras full
make install
poetry run python -m spacy download de_core_news_md
poetry run python -m spacy link --force de_core_news_md de
- name: Run test
id: run_test
Expand Down
13 changes: 13 additions & 0 deletions .github/workflows/continous-integration.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,11 @@ jobs:
# can calculate the proper diff between the branches
git fetch --unshallow origin "${{ github.ref }}"
- name: Add github workflow problem matchers
if: needs.changes.outputs.backend == 'true'
run: |
echo "::add-matcher::.github/matchers/flake8-error-matcher.json"
- name: Lint Code 🎎
if: needs.changes.outputs.backend == 'true'
run: |
Expand Down Expand Up @@ -243,6 +248,14 @@ jobs:
make install-full
make prepare-tests-windows
- name: Add github workflow problem matchers
if: needs.changes.outputs.backend == 'true' && matrix.python-version == 3.6 && matrix.os == 'ubuntu-latest'
# only annotate based on test runs on ubuntu: otherwise
# all errors will be duplicated for each python / os combination
# therefore, we only enable for the one where most tests are run
# (tests will still run in other envs, they will just not create annotations)
run: pip install pytest-github-actions-annotate-failures

- name: Test Code 🔍
if: needs.changes.outputs.backend == 'true' && !(matrix.python-version == 3.8 && matrix.os == 'windows-latest')
env:
Expand Down
18 changes: 18 additions & 0 deletions CHANGELOG.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,24 @@ https://github.com/RasaHQ/rasa/tree/main/changelog/ . -->

<!-- TOWNCRIER -->

## [2.4.2] - 2021-03-25


### Bugfixes
- [#7835](https://github.com/rasahq/rasa/issues/7835): Fix `UnicodeException` in `is_key_in_yaml`.
- [#8258](https://github.com/rasahq/rasa/issues/8258): Fixed the bug that events from previous conversation sessions would be re-saved in the [`SQLTrackerStore`](tracker-stores.mdx#sqltrackerstore) or [`MongoTrackerStore`](tracker-stores.mdx#mongotrackerstore) when `retrieve_events_from_previous_conversation_sessions` was true.


## [2.4.1] - 2021-03-23


### Bugfixes
- [#8194](https://github.com/rasahq/rasa/issues/8194): Fix `TEDPolicy` training e2e entities when no entities are present in the stories
but there are entities in the domain.
- [#8198](https://github.com/rasahq/rasa/issues/8198): Fixed missing model configuration file validation.
- [#8223](https://github.com/rasahq/rasa/issues/8223): In Rasa 2.4.0, support for using `template` in `utter_message` when handling a custom action was wrongly deprecated. Both `template` and `response` are now supported, though note that `template` will be deprecated at Rasa 3.0.0.


## [2.4.0] - 2021-03-11


Expand Down
3 changes: 0 additions & 3 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -116,10 +116,7 @@ types:
--disable-error-code index \
--disable-error-code misc \
--disable-error-code return \
--disable-error-code type-var \
--disable-error-code has-type \
--disable-error-code valid-type \
--disable-error-code no-redef \
--disable-error-code func-returns-value \
--disallow-untyped-calls

Expand Down
25 changes: 19 additions & 6 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -307,14 +307,16 @@ release, as well as:

1. **Go over the milestone and evaluate the status of any PR merging that's happening. Follow up with people on their
bugs and fixes.** If the release introduces new bugs or regressions that can't be fixed in time, we should discuss on
Slack about this and take a decision to go forward or postpone the release. The PR / issue owners are responsible for
communicating any issues which might be release relevant.
Slack about this and take a decision on how to move forward. If the issue is not ready to be merged in time, we remove the issue / PR from the milestone and notify the PR owner and the product manager on Slack about it. The PR / issue owners are responsible for
communicating any issues which might be release relevant. Postponing the release should be considered as an edge case scenario.

#### Release day! 🚀

1. **At the start of the day, post a small message on slack announcing release day!**. Communicate you'll be handling
1. **At the start of the day, post a small message on slack announcing release day!** Communicate you'll be handling
the release, and the time you're aiming to start releasing (again, no later than 4pm, as issues may arise and
cause delays)
cause delays). This message should be posted early in the morning and before moving forward with any of the steps of the release,
in order to give enough time to people to check their PRs and issues. That way they can plan any remaining work. A template of the slack message can be found [here](https://rasa-hq.slack.com/archives/C36SS4N8M/p1613032208137500?thread_ts=1612876410.068400&cid=C36SS4N8M).
The release time should be communicated transparently so that others can plan potentially necessary steps accordingly. If there are bigger changes this should be communicated.
2. Make sure the milestone is empty (everything has been either merged or moved to the next milestone)
3. Once everything in the milestone is taken care of, post a small message on Slack communicating you are about to
start the release process (in case anything is missing).
Expand All @@ -333,7 +335,11 @@ Releasing a new version is quite simple, as the packages are build and distribut
1. Make sure all dependencies are up to date (**especially Rasa SDK**)
- For Rasa SDK that means first creating a [new Rasa SDK release](https://github.com/RasaHQ/rasa-sdk#steps-to-release-a-new-version) (make sure the version numbers between the new Rasa and Rasa SDK releases match)
- Once the tag with the new Rasa SDK release is pushed and the package appears on [pypi](https://pypi.org/project/rasa-sdk/), the dependency in the rasa repository can be resolved (see below).
2. Switch to the branch you want to cut the release from (`main` in case of a major / minor, the current feature branch for micro releases)
2. In case of a minor release, create a new branch that corresponds to the new release, e.g.
```bash
git checkout -b 2.4.x
```
3. Switch to the branch you want to cut the release from (`main` in case of a major, the `<major>.<minor>.x` branch for minors and micros)
- Update the `rasa-sdk` entry in `pyproject.toml` with the new release version and run `poetry update`. This creates a new `poetry.lock` file with all dependencies resolved.
- Commit the changes with `git commit -am "bump rasa-sdk dependency"` but do not push them. They will be automatically picked up by the following step.
3. Run `make release`
Expand All @@ -349,7 +355,14 @@ Releasing a new version is quite simple, as the packages are build and distribut
git checkout -b 1.2.x
git push origin 1.2.x
```

7. After all the steps are completed and if everything goes well then we should see a message automatically posted in the company's Slack (`product` channel) like this [one](https://rasa-hq.slack.com/archives/C7B08Q5FX/p1614354499046600)
8. If no message appears in the channel then you can do the following checks:
- Check the workflows in [Github Actions](https://github.com/RasaHQ/rasa/actions) and make sure that the merged PR of the current release is completed successfully. To easily find your PR you can use the filters `event: push` and `branch: <version number>` (example on release 2.4 you can see [here](https://github.com/RasaHQ/rasa/actions/runs/643344876))
- If the workflow is not completed, then try to re run the workflow in case that solves the problem
- If the problem persists, check also the log files and try to find the root cause of the issue
- If you still cannot resolve the error, contact the infrastructure team by providing any helpful information from your investigation
9. After the message is posted correctly in the `product` channel, check also in the `product-engineering-alerts` channel if there are any alerts related to the Rasa Open Source release like this [one](https://rasa-hq.slack.com/archives/C01585AN2NP/p1615486087001000)
### Cutting a Micro release
Micro releases are simpler to cut, since they are meant to contain only bugfixes.
Expand Down
2 changes: 2 additions & 0 deletions changelog/7640.bugfix.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
`rasa test`, `rasa test core` and `rasa test nlu` no longer show temporary paths
in case there are issues in the test files.
Loading

0 comments on commit 0b7bd61

Please sign in to comment.