Skip to content

[Bug]: guideline evaluation is throwing error #17012

@Rohith-Scalers

Description

@Rohith-Scalers

Bug Description

guideline evaluation is throwing error saying missing model

Version

latest

Steps to Reproduce

run the guideline evaluation example

Relevant Logs/Tracbacks

Traceback (most recent call last):
  File "/home/rohit/.local/lib/python3.10/site-packages/llama_index/core/async_utils.py", line 33, in asyncio_run
    return loop.run_until_complete(coro)
  File "/home/rohit/.local/lib/python3.10/site-packages/nest_asyncio.py", line 98, in run_until_complete
    return f.result()
  File "/usr/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/usr/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/home/rohit/.local/lib/python3.10/site-packages/llama_index/core/evaluation/guideline.py", line 118, in aevaluate
    eval_data = self._output_parser.parse(eval_response)
  File "/home/rohit/.local/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py", line 230, in wrapper
    result = func(*args, **kwargs)
  File "/home/rohit/.local/lib/python3.10/site-packages/llama_index/core/output_parsers/pydantic.py", line 62, in parse
    return self._output_cls.parse_raw(json_str)
  File "/home/rohit/.local/lib/python3.10/site-packages/pydantic/v1/main.py", line 549, in parse_raw
    return cls.parse_obj(obj)
  File "/home/rohit/.local/lib/python3.10/site-packages/pydantic/v1/main.py", line 526, in parse_obj
    return cls(**obj)
  File "/home/rohit/.local/lib/python3.10/site-packages/pydantic/v1/main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 2 validation errors for EvaluationData
passing
  field required (type=value_error.missing)
feedback
  field required (type=value_error.missing)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rohit/RagEval/src/main.py", line 86, in <module>
    GuidelineData.get_score(query,reference_answer,reference_contexts,[guideline_index])
  File "/home/rohit/RagEval/src/evaluation_metrics/guideline_score.py", line 25, in get_score
    eval_result = evaluator.evaluate(
  File "/home/rohit/.local/lib/python3.10/site-packages/llama_index/core/evaluation/base.py", line 62, in evaluate
    return asyncio_run(
  File "/home/rohit/.local/lib/python3.10/site-packages/llama_index/core/async_utils.py", line 47, in asyncio_run
    raise type(e)(f"Error running coroutine: {e!s}") from e
TypeError: ValidationError.__init__() missing 1 required positional argument: 'model'

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtriageIssue needs to be triaged/prioritized

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions