[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RagasMetric breaks when used with dataset #782

Open
haarisedhi102 opened this issue May 21, 2024 · 5 comments
Open

RagasMetric breaks when used with dataset #782

haarisedhi102 opened this issue May 21, 2024 · 5 comments

Comments

@haarisedhi102
Copy link
haarisedhi102 commented May 21, 2024

Describe the bug
RagasMetric breaks when used with dataset

To Reproduce
dataset.evaluate([RagasMetric])

Errors:
TypeError: RagasMetric.a_measure() got an unexpected keyword argument '_show_indicator'

and another

self.llm.set_run_config(run_config)
^^^^^^^^^^^^^^^^^^^^^^^

AttributeError: 'GPTModel' object has no attribute 'set_run_config'

Additional context
Add any other context about the problem here.

@penguine-ip
Copy link
Contributor

Hey @haarisedhi102 where are you finding this?

self.llm.set_run_config(run_config)

Also what version of deepeval are you on?

@lmpratti
Copy link
lmpratti commented Jul 2, 2024

@penguine-ip This happens on the latest version of deepeval too (0.21.62). I suspect it's due to the dependency on ragas library.
It fails even after adding the below set_run_config to the llm.

    def set_run_config(self, run_config: RunConfig = RunConfig()):
        self.run_config = run_config

@penguine-ip
Copy link
Contributor

@lmpratti You're probably using the DeepEvalBaseLLM wrapper in the ragas metric. Unforuntately the ragas metric only accept langhcain chat models, you can read the docs here: https://docs.confident-ai.com/docs/metrics-ragas#example

@lmpratti
Copy link
lmpratti commented Jul 3, 2024

@penguine-ip Thanks a ton! Is there a plan to support DeepEvalBaseLLM wrapped models for ragas metric?

@penguine-ip
Copy link
Contributor

@lmpratti Unfortunately no because ragas does not have their own ecosystem to support custom models, so out of our control. But we have our own similar implementation of the ragas metrics for this reason

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants