Evaluation API Reference

All classes are importable from anchor.evaluation. For usage examples see the Evaluation Guide.

RetrievalMetrics

Immutable Pydantic model holding metrics for a single retrieval evaluation. All values are bounded in [0.0, 1.0].

from anchor.evaluation import RetrievalMetrics

RAGMetrics

Immutable Pydantic model for RAGAS-style RAG evaluation. All values bounded in [0.0, 1.0].

from anchor.evaluation import RAGMetrics

EvaluationResult

Combines retrieval and RAG metrics into a single evaluation result.

from anchor.evaluation import EvaluationResult

RetrievalMetricsCalculator

Computes standard IR metrics from ranked results and a known set of relevant document IDs. No LLM dependencies.

from anchor.evaluation import RetrievalMetricsCalculator

Constructor:

RetrievalMetricsCalculator(k: int = 10)

[!CAUTION] Raises ValueError if k < 1.evaluate(retrieved, relevant, k=None) -> RetrievalMetrics

def evaluate(
    self,
    retrieved: list[ContextItem],
    relevant: list[str],
    k: int | None = None,
) -> RetrievalMetrics

from anchor.evaluation import RetrievalMetricsCalculator
from anchor.models.context import ContextItem, SourceType

calc = RetrievalMetricsCalculator(k=5)
items = [ContextItem(id="a", content="x", source=SourceType.RETRIEVAL)]
metrics = calc.evaluate(items, relevant=["a", "b"], k=3)
print(metrics.precision_at_k)  # 1.0

LLMRAGEvaluator

RAGAS-style RAG evaluator driven by user-supplied callback functions. Each callback returns a float in [0.0, 1.0].

from anchor.evaluation import LLMRAGEvaluator

Constructor:

LLMRAGEvaluator(
    *,
    faithfulness_fn: Callable[[str, list[str]], float] | None = None,
    relevancy_fn: Callable[[str, str], float] | None = None,
    precision_fn: Callable[[str, list[str]], float] | None = None,
    recall_fn: Callable[[str, list[str], str], float] | None = None,
)

evaluate(query, answer, contexts, ground_truth=None) -> RAGMetrics

Dimensions without registered callbacks return 0.0.

PipelineEvaluator

Orchestrates retrieval and RAG evaluation into a single result.

from anchor.evaluation import PipelineEvaluator

Constructor:

PipelineEvaluator(
    *,
    retrieval_calculator: RetrievalMetricsCalculator | None = None,
    rag_evaluator: LLMRAGEvaluator | None = None,
)

evaluate_retrieval(retrieved, relevant, k=10) -> RetrievalMetrics -- evaluates retrieval only.

evaluate_rag(query, answer, contexts, ground_truth=None) -> RAGMetrics -- evaluates RAG only.

[!CAUTION] Raises ValueError if no rag_evaluator was configured.evaluate(...) -> EvaluationResult -- runs both evaluations.

def evaluate(
    self,
    query: str,
    answer: str,
    retrieved: list[ContextItem],
    relevant: list[str],
    contexts: list[str],
    ground_truth: str | None = None,
    k: int = 10,
    metadata: dict[str, Any] | None = None,
) -> EvaluationResult

EvaluationSample (A/B testing)

A single evaluation sample. Defined in anchor.evaluation.ab_testing.

from anchor.evaluation import EvaluationSample

EvaluationDataset (A/B testing)

A collection of EvaluationSample instances.

from anchor.evaluation import EvaluationDataset

AggregatedMetrics (A/B testing)

Aggregated retrieval metrics across multiple evaluation samples.

from anchor.evaluation import AggregatedMetrics

ABTestResult

Result of an A/B test comparing two retrievers.

from anchor.evaluation import ABTestResult

ABTestRunner

Runs an A/B test comparing two retrievers on a shared dataset.

from anchor.evaluation import ABTestRunner

Constructor:

ABTestRunner(evaluator: PipelineEvaluator, dataset: EvaluationDataset)

run(retriever_a, retriever_b, k=10, significance_level=0.05) -> ABTestResult

BatchEvaluator

Runs evaluation over an entire dataset and aggregates results. Importable from anchor.evaluation.batch.

from anchor.evaluation.batch import BatchEvaluator

Constructor:

BatchEvaluator(*, evaluator: PipelineEvaluator, retriever: Retriever, top_k: int = 10)

evaluate(dataset, k=10) -> AggregatedMetrics

Returns AggregatedMetrics (batch module variant) with count, mean_precision, mean_recall, mean_f1, mean_mrr, mean_ndcg, mean_hit_rate, p95_precision, p95_recall, min_precision, min_recall, and per_sample_results.

HumanJudgment

A single human relevance judgment for a query-document pair.

from anchor.evaluation import HumanJudgment

HumanEvaluationCollector

Collects human relevance judgments and computes inter-annotator agreement.

from anchor.evaluation import HumanEvaluationCollector

Constructor: HumanEvaluationCollector() -- no parameters.

Property: judgments -> list[HumanJudgment] -- copy of all collected judgments.

add_judgment(judgment: HumanJudgment) -> None -- add a single judgment.

add_judgments(judgments: list[HumanJudgment]) -> None -- add multiple judgments.

compute_agreement() -> float -- Cohen's kappa over (query, item_id) pairs judged by at least two annotators. Returns 0.0 if no overlapping judgments exist.

to_dataset(threshold: int = 2) -> EvaluationDataset -- converts judgments into an EvaluationDataset. Items with mean relevance at or above threshold are considered relevant.

compute_metrics() -> dict[str, float] -- returns mean_relevance, agreement, num_judgments, num_annotators, num_queries.