Merge branch 'main' into switch-to-mypy-part1

Dazu-io · Mar 12, 2021 · cbedef1 · cbedef1
2 parents c413f21 + 8692ea3
commit cbedef1
Show file tree

Hide file tree

Showing 50 changed files with 874 additions and 469 deletions.
diff --git a/CHANGELOG.mdx b/CHANGELOG.mdx
@@ -16,6 +16,113 @@ https://github.com/RasaHQ/rasa/tree/main/changelog/ . -->
 
 <!-- TOWNCRIER -->
 
+## [2.4.0] - 2021-03-11
+
+
+### Deprecations and Removals
+- [#6484](https://github.com/rasahq/rasa/issues/6484): NLG Server
+  - Changed request format to send `response` as well as `template` as a field. The `template` field will be removed in Rasa Open Source 3.0.0.
+
+  `rasa.core.agent`
+  - The terminology `template` is deprecated and replaced by `response`. Support for `template` from the NLG response will be removed in Rasa Open Source 3.0.0. Please see [here](nlg.mdx) for more details.
+
+  `rasa.core.nlg.generator`
+  - `generate()` now takes in  `utter_action` as a parameter.
+  - The terminology `template` is deprecated and replaced by `response`. Support for `template` in the `NaturalLanguageGenerator` will be removed in Rasa Open Source 3.0.0.
+
+  `rasa.shared.core.domain`
+  - The property `templates` is deprecated. Use `responses` instead. It will be removed in Rasa Open Source 3.0.0.
+  - `retrieval_intent_templates` will be removed in Rasa Open Source 3.0.0. Please use `retrieval_intent_responses` instead.
+  - `is_retrieval_intent_template` will be removed in Rasa Open Source 3.0.0. Please use `is_retrieval_intent_response` instead.
+  - `check_missing_templates` will be removed in Rasa Open Source 3.0.0. Please use `check_missing_responses` instead.
+
+  Response Selector
+  - The field `template_name` will be deprecated in Rasa Open Source 3.0.0. Please use `utter_action` instead. Please see [here](components.mdx#selectors) for more details.
+  - The field `response_templates` will be deprecated in Rasa Open Source 3.0.0. Please use `responses` instead. Please see [here](components.mdx#selectors) for more details.
+
+### Improvements
+- [#7022](https://github.com/rasahq/rasa/issues/7022): The following endpoints now require the existence of the conversation for the specified conversation ID, raising an exception and returning a 404 status code.
+
+  * `GET /conversations/<conversation_id:path>/story`
+
+  * `POST /conversations/<conversation_id:path>/execute`
+
+  * `POST /conversations/<conversation_id:path>/predict`
+- [#7438](https://github.com/rasahq/rasa/issues/7438): Simplify our training by overwriting `train_step` instead of `fit` for our custom models.
+
+  This allows us to use the build-in callbacks from Keras, such as the 
+  [Tensorboard Callback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard), 
+  which offers more functionality compared to what we had before. 
+
+    :::warning
+    If you want to use Tensorboard for `DIETClassifier`, `ResponseSelector`, or `TEDPolicy` and log metrics after 
+    every (mini)batch, please use 'batch' instead of 'minibatch' as 'tensorboard_log_level'.
+- [#7578](https://github.com/rasahq/rasa/issues/7578): When `TED` is configured to extract entities `rasa test` now evaluates them against the labels in the test stories. Results are saved in `/results` along with the results for the NLU components that extract entities.
+- [#7680](https://github.com/rasahq/rasa/issues/7680): We're now running integration tests for Rasa Open Source, with initial coverage for `SQLTrackerStore` (with PostgreSQL),
+  `RedisLockStore` (with Redis) and `PikaEventBroker` (with RabbitMQ). The integration tests are now part of our
+  CI, and can also be ran locally using `make test-integration`
+  (see [Rasa Open Source README](https://github.com/RasaHQ/rasa#running-the-integration-tests) for more information).
+- [#7763](https://github.com/rasahq/rasa/issues/7763): Allow tests to be located anywhere, not just in `tests` directory.
+- [#7893](https://github.com/rasahq/rasa/issues/7893): Model configuration files are now validated whether they match the expected schema.
+- [#7952](https://github.com/rasahq/rasa/issues/7952): Speed up `YAMLStoryReader.is_key_in_yaml` function by making it to check if key is in YAML without 
+  actually parsing the text file.
+- [#7953](https://github.com/rasahq/rasa/issues/7953): Speed up YAML parsing by reusing parsers, making the process of environment variable interpolation optional,
+  and by not adding duplicating implicit resolvers and YAML constructors to `ruamel.yaml`
+- [#7955](https://github.com/rasahq/rasa/issues/7955): Drastically improved finger printing time for large story graphs
+- [#8000](https://github.com/rasahq/rasa/issues/8000): Remove console logging of conversation level F1-score and precision since these calculations were not meaningful.
+
+  Add conversation level accuracy to core policy results logged to file in `story_report.json` after running `rasa test core` or `rasa test`.
+- [#8100](https://github.com/rasahq/rasa/issues/8100): Improved the [lock store](lock-stores.mdx) debug log message when the process has to
+  queue because other messages have to be processed before this item.
+
+### Bugfixes
+- [#4612](https://github.com/rasahq/rasa/issues/4612): Fixed the bug that OR statements in stories would break the check whether a model needs to be retrained
+- [#7063](https://github.com/rasahq/rasa/issues/7063): Update the spec of `POST /model/test/intents` and add tests for cases when JSON is provided.
+
+  Fix the incorrect temporary file extension for the data that gets extracted from the payload provided
+  in the body of `POST /model/test/intents` request.
+- [#7113](https://github.com/rasahq/rasa/issues/7113): Fix for the cli command `rasa data convert config` when migrating Mapping Policy and no rules.
+
+  Making `rasa data convert config` migrate correctly the Mapping Policy when no rules are available. It updates the `config.yml` file by removing the `MappingPolicy` and adding the `RulePolicy` instead. Also, it creates the `data/rules.yml` file even if empty in the case of no available rules.
+- [#7470](https://github.com/rasahq/rasa/issues/7470): Allow to have slots with values that result to a dictionary under the key `slot_was_set` (in `stories.yml` file).
+
+  An example would be to have the following story step in `stories.yml`:
+  ```yaml
+  - slot_was_set:
+      - some_slot:
+          some_key: 'some_value'
+          other_key: 'other_value'
+  ```
+  This would be allowed if the `some_slot` is also set accordingly in the `domain.yml` with type `any`.
+- [#7662](https://github.com/rasahq/rasa/issues/7662): Update the fingerprinting function to recognize changes in lookup files.
+- [#7932](https://github.com/rasahq/rasa/issues/7932): Fixed a bug when interpolating environment variables in YAML files which included `$` in their value. 
+  This led to the following stack trace:
+
+  ```
+  ValueError: Error when trying to expand the environment variables in '${PASSWORD}'. Please make sure to also set these environment variables: '['$qwerty']'.
+  (13 additional frame(s) were not displayed)
+  ...
+    File "rasa/utils/endpoints.py", line 26, in read_endpoint_config
+      content = rasa.shared.utils.io.read_config_file(filename)
+    File "rasa/shared/utils/io.py", line 527, in read_config_file
+      content = read_yaml_file(filename)
+    File "rasa/shared/utils/io.py", line 368, in read_yaml_file
+      return read_yaml(read_file(filename, DEFAULT_ENCODING))
+    File "rasa/shared/utils/io.py", line 349, in read_yaml
+      return yaml_parser.load(content) or {}
+    File "rasa/shared/utils/io.py", line 314, in env_var_constructor
+      " variables: '{}'.".format(value, not_expanded)
+  ```
+- [#7949](https://github.com/rasahq/rasa/issues/7949): The REQUESTED_SLOT always belongs to the currently active form.
+
+  Previously it was possible that after form switching, the REQUESTED_SLOT was for the previous form.
+- [#96](https://github.com/rasahq/rasa/issues/96): Update the `LanguageModelFeaturizer` tests to reflect new default model weights for `bert`, and skip all `bert` tests 
+  with default model weights on CI, run `bert` tests with `bert-base-uncased` on CI instead.
+
+### Miscellaneous internal changes
+- [#6484](https://github.com/rasahq/rasa/issues/6484), [#7737](https://github.com/rasahq/rasa/issues/7737), [#7879](https://github.com/rasahq/rasa/issues/7879)
+
+
 ## [2.3.4] - 2021-02-26
 
 

diff --git a/changelog/6484.misc.md b/changelog/6484.misc.md
diff --git a/changelog/6484.removal.md b/changelog/6484.removal.md
diff --git a/changelog/7022.improvement.md b/changelog/7022.improvement.md
diff --git a/changelog/7063.bugfix.md b/changelog/7063.bugfix.md
diff --git a/changelog/7113.bugfix.md b/changelog/7113.bugfix.md
diff --git a/changelog/7438.improvement.md b/changelog/7438.improvement.md
diff --git a/changelog/7470.bugfix.md b/changelog/7470.bugfix.md
diff --git a/changelog/7662.bugfix.md b/changelog/7662.bugfix.md
diff --git a/changelog/7680.improvement.md b/changelog/7680.improvement.md
diff --git a/changelog/7737.misc.md b/changelog/7737.misc.md
diff --git a/changelog/7879.misc.md b/changelog/7879.misc.md
diff --git a/changelog/7893.improvement.md b/changelog/7893.improvement.md
diff --git a/changelog/7932.bugfix.md b/changelog/7932.bugfix.md
diff --git a/changelog/7949.bugfix.md b/changelog/7949.bugfix.md
diff --git a/changelog/7952.improvement.md b/changelog/7952.improvement.md
diff --git a/changelog/7953.improvement.md b/changelog/7953.improvement.md
diff --git a/changelog/8000.improvement.md b/changelog/8000.improvement.md
diff --git a/changelog/96.bugfix.md b/changelog/96.bugfix.md
diff --git a/data/test_e2ebot/config.yml b/data/test_e2ebot/config.yml
@@ -0,0 +1,16 @@
+language: en
+pipeline:
+  - name: WhitespaceTokenizer
+    intent_tokenization_flag: True
+  - name: RegexFeaturizer
+  - name: LexicalSyntacticFeaturizer
+  - name: CountVectorsFeaturizer
+  - name: CountVectorsFeaturizer
+    analyzer: char_wb
+    min_ngram: 1
+    max_ngram: 4
+  - name: DIETClassifier
+    epochs: 1
+policies:
+- name: TEDPolicy
+  epochs: 5
diff --git a/data/test_e2ebot/data/nlu.yml b/data/test_e2ebot/data/nlu.yml
@@ -0,0 +1,18 @@
+version: "2.0"
+
+nlu:
+- intent: tell_name
+  examples: |
+    - my name is [joe](name)
+    - my name is [bob](name)
+    - I am [bob](name)
+    - I am [jim](name)
+    - I am called [bill](name)
+
+- intent: tell_mood
+  examples: |
+    - I am feeling [happy](mood)
+    - I am quite [happy](mood)
+    - I feel [sad](mood)
+    - I feel [happy](mood)
+    - I am so [sad](mood)
diff --git a/data/test_e2ebot/data/stories.yml b/data/test_e2ebot/data/stories.yml
@@ -0,0 +1,16 @@
+version: "2.0"
+
+stories:
+- story: Tell name and mood
+  steps:
+  - user: my name is [joe](name)
+  - bot: hi joe!
+  - user: I am feeling [happy](mood)
+  - bot: I'm glad
+
+- story: Tell name and mood 2
+  steps:
+  - user: my name is [bob](name)
+  - bot: hi bob!
+  - user: I am feeling [sad](mood)
+  - bot: oh dear...
diff --git a/data/test_e2ebot/domain.yml b/data/test_e2ebot/domain.yml
@@ -0,0 +1,13 @@
+version: "2.0"
+
+intents:
+ - tell_name
+ - tell_mood
+
+entities:
+ - name
+ - mood
+
+responses:
+  utter_greet:
+  - text: "hi!"
diff --git a/data/test_e2ebot/tests/test_stories.yml b/data/test_e2ebot/tests/test_stories.yml
@@ -0,0 +1,14 @@
+version: "2.0"
+
+stories:
+- story: Test with two easy entities
+  steps:
+    - user: my name is [joe](name)
+    - bot: hi joe!
+    - user: I am feeling [happy](mood)
+    - bot: I'm glad
+
+- story: Test with a hard entity
+  steps:
+    - user: today I was very [cranky](mood)
+    - bot: I'm sorry!
diff --git a/data/test_multiline_intent_examples_yaml/nlu.yml b/data/test_multiline_intent_examples_yaml/nlu.yml
@@ -0,0 +1,14 @@
+version: "2.0"
+nlu:
+- intent: greet
+  examples:
+  - text: |
+      Hello,
+      My name is Christof
+  - text: |
+      Hello,
+      You can call me Christof
+- intent: bye
+  examples: |
+    - bye, my friend
+    - cya
diff --git a/data/test_yaml_stories/story_with_two_equal_or_statements.yml b/data/test_yaml_stories/story_with_two_equal_or_statements.yml
@@ -0,0 +1,25 @@
+stories:
+- story: story_with_two_equal_or_statements
+  steps:
+  - intent: simple
+  - action: utter_default
+  - or:
+    - intent: affirm
+    - intent: thank_you
+  - action: utter_greet
+  - action: utter_default
+  - or:
+    - intent: affirm
+    - intent: thank_you
+  - action: utter_goodbye
+# same name again, same events, same intents, but with entities
+- story: story_with_two_equal_or_statements
+  steps:
+  - intent: simple
+  - action: utter_default
+  - or:
+    - intent: affirm
+      entities:
+        - name: peter
+    - intent: thank_you
+  - action: utter_greet
diff --git a/docs/docs/migration-guide.mdx b/docs/docs/migration-guide.mdx
@@ -11,6 +11,7 @@ This page contains information about changes between major versions and
 how you can migrate from one version to another.
 
 ## Rasa 2.3 to Rasa 2.4
+
 ### Deprecating `template` for `response`
 NLG Server
 - Changed request format to send `response` as well as `template` as a field. The `template` field will be removed in Rasa Open Source 3.0.0.
@@ -32,6 +33,7 @@ Response Selector
 - The field `template_name` will be deprecated in Rasa Open Source 3.0.0. Please use `utter_action` instead. Please see [here](./components.mdx#selectors) for more details.
 - The field `response_templates` will be deprecated in Rasa Open Source 3.0.0. Please use `responses` instead. Please see [here](./components.mdx#selectors) for more details.
 
+
 ## Rasa 2.3.3 to Rasa 2.3.4
 
 :::caution