View Source Rag.Evaluation (rag v0.2.1)

Functions to evaluate generations.

Summary

Types

provider()
response_function()

Functions

detect_hallucination(generation, provider)

Takes the values of query, response and context from generation, conproviders a prompt and passes it to response_function or provider to detect potential hallucinations. Then, puts a new hallucination evaluation in generation.evaluations.

evaluate_rag_triad(generation, provider)

Evaluates the response, query, and context according to the RAG triad.

Types

provider()

@type provider() :: struct()

response_function()

@type response_function() :: (String.t(), opts :: keyword() -> String.t())

Functions

detect_hallucination(generation, provider)

@spec detect_hallucination(Rag.Generation.t(), response_function() | provider()) ::
  Rag.Generation.t()

Takes the values of query, response and context from generation, conproviders a prompt and passes it to response_function or provider to detect potential hallucinations. Then, puts a new hallucination evaluation in generation.evaluations.

evaluate_rag_triad(generation, provider)

@spec evaluate_rag_triad(Rag.Generation.t(), response_function() | provider()) ::
  Rag.Generation.t()

Evaluates the response, query, and context according to the RAG triad.

  • context relevance: is the retrieved context relevant to the query?
  • groundedness: is the response supported by the context?
  • answer relevance: is the answer relevant to the query?

Prompts from https://github.com/truera/trulens/blob/main/src/feedback/trulens/feedback/prompts.py