-
Notifications
You must be signed in to change notification settings - Fork 6.7k
Closed as not planned
Labels
bugSomething isn't workingSomething isn't workingtriageIssue needs to be triaged/prioritizedIssue needs to be triaged/prioritized
Description
Bug Description
guideline evaluation is throwing error saying missing model
Version
latest
Steps to Reproduce
run the guideline evaluation example
Relevant Logs/Tracbacks
Traceback (most recent call last):
File "/home/rohit/.local/lib/python3.10/site-packages/llama_index/core/async_utils.py", line 33, in asyncio_run
return loop.run_until_complete(coro)
File "/home/rohit/.local/lib/python3.10/site-packages/nest_asyncio.py", line 98, in run_until_complete
return f.result()
File "/usr/lib/python3.10/asyncio/futures.py", line 201, in result
raise self._exception.with_traceback(self._exception_tb)
File "/usr/lib/python3.10/asyncio/tasks.py", line 232, in __step
result = coro.send(None)
File "/home/rohit/.local/lib/python3.10/site-packages/llama_index/core/evaluation/guideline.py", line 118, in aevaluate
eval_data = self._output_parser.parse(eval_response)
File "/home/rohit/.local/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py", line 230, in wrapper
result = func(*args, **kwargs)
File "/home/rohit/.local/lib/python3.10/site-packages/llama_index/core/output_parsers/pydantic.py", line 62, in parse
return self._output_cls.parse_raw(json_str)
File "/home/rohit/.local/lib/python3.10/site-packages/pydantic/v1/main.py", line 549, in parse_raw
return cls.parse_obj(obj)
File "/home/rohit/.local/lib/python3.10/site-packages/pydantic/v1/main.py", line 526, in parse_obj
return cls(**obj)
File "/home/rohit/.local/lib/python3.10/site-packages/pydantic/v1/main.py", line 341, in __init__
raise validation_error
pydantic.v1.error_wrappers.ValidationError: 2 validation errors for EvaluationData
passing
field required (type=value_error.missing)
feedback
field required (type=value_error.missing)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rohit/RagEval/src/main.py", line 86, in <module>
GuidelineData.get_score(query,reference_answer,reference_contexts,[guideline_index])
File "/home/rohit/RagEval/src/evaluation_metrics/guideline_score.py", line 25, in get_score
eval_result = evaluator.evaluate(
File "/home/rohit/.local/lib/python3.10/site-packages/llama_index/core/evaluation/base.py", line 62, in evaluate
return asyncio_run(
File "/home/rohit/.local/lib/python3.10/site-packages/llama_index/core/async_utils.py", line 47, in asyncio_run
raise type(e)(f"Error running coroutine: {e!s}") from e
TypeError: ValidationError.__init__() missing 1 required positional argument: 'model'Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingtriageIssue needs to be triaged/prioritizedIssue needs to be triaged/prioritized