Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge 2.3.x in main #8064

Merged
merged 43 commits into from
Mar 3, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
5106076
restrict cosine and inner for ted, add warnings to remove cosine if u…
dakshvar22 Feb 22, 2021
0889962
add changelog and fix migration guide
dakshvar22 Feb 22, 2021
18817e7
review
dakshvar22 Feb 22, 2021
3533785
reverted back auto-config
dakshvar22 Feb 23, 2021
67113b4
completely remove cosine as model confidence
dakshvar22 Feb 22, 2021
1b6fd56
reverted back auto-config
dakshvar22 Feb 23, 2021
5af02ae
use linear normalization for all model confidences
dakshvar22 Feb 23, 2021
c184d91
remove ranking length
dakshvar22 Feb 23, 2021
af70d22
normalized version of inner confidences
dakshvar22 Feb 24, 2021
fe8b005
add docs
dakshvar22 Feb 24, 2021
4310419
Apply suggestions from code review
dakshvar22 Feb 24, 2021
f38b16c
add dimension to reduce sum
dakshvar22 Feb 24, 2021
c2a50af
fix tests
dakshvar22 Feb 24, 2021
0d79b71
Merge branch '2.3.x' into restrict_model_confidences
dakshvar22 Feb 24, 2021
85ee43b
copy edits
dakshvar22 Feb 24, 2021
e9bfffc
change inner to linear_norm_inner and add some tests.'
dakshvar22 Feb 25, 2021
ed5a6a6
rename to linear_norm, add docs
dakshvar22 Feb 25, 2021
5a0d0e2
edit docs in components
dakshvar22 Feb 25, 2021
5f980ad
formatting
dakshvar22 Feb 25, 2021
8e17ed7
Update changelog/8014.bugfix.md
dakshvar22 Feb 25, 2021
60b8416
Merge branch '2.3.x' into restrict_model_confidences
dakshvar22 Feb 25, 2021
23edadf
bump version in docs
dakshvar22 Feb 25, 2021
37bd3d1
remove redundant test
dakshvar22 Feb 25, 2021
0ce7d52
Update GPU cluster
Feb 23, 2021
deae31e
Update zone
Feb 23, 2021
0e6687d
Update resource request
Feb 23, 2021
077b1c8
Add a missing secret
Feb 23, 2021
6d161a5
Apply suggestions from code review
dakshvar22 Feb 26, 2021
9f524e2
Update docs/docs/migration-guide.mdx
dakshvar22 Feb 26, 2021
a3621e4
more docs comments
dakshvar22 Feb 26, 2021
eecdb50
add warning
dakshvar22 Feb 26, 2021
85e8d36
final edits
dakshvar22 Feb 26, 2021
db50a37
format lists
dakshvar22 Feb 26, 2021
30da9ec
remove 'also' and fix typo CI'
dakshvar22 Feb 26, 2021
0df088c
Merge pull request #8014 from RasaHQ/restrict_model_confidences
dakshvar22 Feb 26, 2021
f51bba2
prepared release of version 2.3.4
dakshvar22 Feb 26, 2021
af140e5
Merge pull request #8062 from RasaHQ/prepare-release-2.3.4
dakshvar22 Feb 26, 2021
c14f4b8
Merge branch '2.3.x' into merge_2.3.x
dakshvar22 Feb 26, 2021
52b22f5
Update links to Sanic docs in our documentation
alwx Mar 2, 2021
3d3f172
Changelog entry
alwx Mar 2, 2021
535c195
Added 1.10.23 updates to 2.3.X Changelog (#8077)
b-quachtran Mar 2, 2021
590aa31
Merge branch 'main' into merge_2.3.x
m-vdb Mar 3, 2021
609bdd0
Merge branch '2.3.x' into merge_2.3.x
m-vdb Mar 3, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
more docs comments
  • Loading branch information
dakshvar22 committed Feb 26, 2021
commit a3621e4a993623e375b1b7f5b548729a01d3f8a6
2 changes: 1 addition & 1 deletion changelog/8014.bugfix.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Users are now encouraged to try `model_confidence=linear_norm` which will produc
model_confidence: linear_norm
constrain_similarities: True
```
This should ease up tuning fallback thresholds as confidences for wrong predictions are better distributed across the range `[0, 1]`.
This should ease up [tuning fallback thresholds](./fallback-handoff.mdx#fallbacks) as confidences for wrong predictions are better distributed across the range `[0, 1]`.

If an assistant was trained with `model_confidence=cosine` or `model_confidence=inner` setting using previous versions of Rasa Open Source, it should be re-trained by either removing the `model_confidence` option from the configuration or setting it to `linear_norm`

Expand Down
24 changes: 11 additions & 13 deletions docs/docs/migration-guide.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -26,20 +26,17 @@ Rasa Open Source `2.3.0` introduced the option of using cosine similarities for
labels. Since this is wrong, the option `model_confidence=cosine` has been removed and is no longer available.

Also, `model_confidence=inner` is deprecated as it produces an unbounded range of confidences which can break
the logic of assistants in various other places. Users are now encouraged to try `model_confidence=linear_norm`
which will produce a linearly normalized version of dot product similarities with
each value in the range `[0,1]`. Setting `model_confidence` to `linear_norm` will be helpful when trying to tune
fallback thresholds.
the logic of assistants in various other places.

If the assistant was trained with `model_confidence=cosine` or `model_confidence=inner` setting, it should be
re-trained by either removing the `model_confidence` option from the configuration or setting it to `linear_norm`, as:
We encourage you to try `model_confidence=linear_norm` which will produce a linearly normalized version of dot product similarities with each value in the range `[0,1]`. This can be done with the following config:
```
- name: DIETClassifier
model_confidence: linear_norm
constrain_similarities: True
...
model_confidence: linear_norm
constrain_similarities: True
```

If you trained a model with `model_confidence=cosine` or `model_confidence=inner` setting using previous versions of Rasa Open Source, please re-train by either removing the `model_confidence` option from the configuration or setting it to `linear_norm`.


## Rasa 2.2 to Rasa 2.3

Expand All @@ -51,10 +48,11 @@ components `DIETClassifier`, `ResponseSelector` and `TEDPolicy`. These include:
2. The default loss function (`loss_type=cross_entropy`) can add an optional sigmoid cross-entropy loss of all similarity values to constrain
them to an approximate range. You can turn on this option by setting `constrain_similarities=True`. This should help the models to perform better on real world test sets.

Also, a new option `model_confidence` has been added to each ML component. It affects how the model's confidence for each label is computed during inference. It can take one of two values:
Also, a new option `model_confidence` has been added to each ML component. It affects how the model's confidence for each label is computed during inference. It can take one of three values:
1. `softmax` - Dot product similarities between input and label embeddings are post-processed with a softmax function, as a result of which confidence for all labels sum up to 1.
2. `linear_norm` - Dot product similarities between input and label embeddings are post-processed with a linear normalization function. Confidence for each label will be in the range `[0,1]`.
The default value is `softmax`, but we recommend trying `linear_norm` as it should be easier to tune thresholds for fallback with it.
2. `cosine` - Cosine similarity between input and label embeddings. Confidence for each label will be in the range `[-1,1]`.
3. `linear_norm` - Dot product similarities between input and label embeddings are post-processed with a linear normalization function. Confidence for each label will be in the range `[0,1]`.
The default value is `softmax`, but we recommend trying `linear_norm`. This should make it easier to [tune thresholds for fallback](./fallback-handoff.mdx#fallbacks).
The value of this option does not affect how confidences are computed for entity predictions in `DIETClassifier`.

With both the above recommendations, users are encouraged to experiment by configuring their ML component, e.g. `DIETClassifier`, as:
Expand All @@ -64,7 +62,7 @@ With both the above recommendations, users are encouraged to experiment by confi
constrain_similarities: True
...
```
Once the assistant is re-trained with the above configuration, users should also tune fallback confidence thresholds.
Once the assistant is re-trained with the above configuration, users should also [tune fallback confidence thresholds](./fallback-handoff.mdx#fallbacks).


## Rasa 2.1 to Rasa 2.2
Expand Down