Skip to main content

Bias

The bias metric determines whether your LLM output contains gender, racial, or political bias. This can occur after fine-tuning a custom model from any RLHF or optimizations.

info

Bias in deepeval is a referenceless metric. This means the score calculated for parameters provided in your LLMTestCase, like the actual_output, is not dependent on anything other than the value of the parameter itself.

Installation​

Bias in deepeval requires an additional installation:

pip install Dbias

Required Arguments​

To use the BiasMetric, you'll have to provide the following arguments when creating an LLMTestCase:

  • input
  • actual_output

Example​

from deepeval.metrics import BiasMetric
from deepeval.test_case import LLMTestCase

metric = BiasMetric(threshold=0.5)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    # Replace this with the actual output from your LLM application
    actual_output = "We offer a 30-day full refund at no extra cost."
)

metric.measure(test_case)
print(metric.score)
note

Unlike other metrics you've seen so far, the threshold for the BiasMetric is instead a maxmium threshold.