# LayaEval
> Open-source Python library with 42 deterministic evaluation checks for RAG pipelines and autonomous AI agents. Runs locally in milliseconds with zero opaque scoring.

LayaEval sits around your LLM, RAG application, or AI agent. It inspects what goes into the model and what comes out, providing instant pass/fail verdicts, normalized scores, and confidence levels.

## Quickstart
- Install: `pip install layaeval`
- GitHub Repository: https://github.com/VinayChaudhari1996/LayaEval
- Full Concatenated Context: https://www.layaeval.in/llms-full.txt

```python
from layaeval import Groundedness, ToolSelectionCorrectness

# 1. Did the RAG answer stick to the sources?
g = Groundedness(threshold=0.7)
res = g(context="Paris is the capital of France.", response="Paris is the capital of France.")
print(res.passed, res.value, res.confidence)

# 2. Did the AI agent select the correct tool?
t = ToolSelectionCorrectness(threshold=0.7)
res = t(task="Find a hotel in Paris", available_tools=["search_hotels", "send_email"], selected_tool="search_hotels")
print(res.passed, res.value, res.confidence)
```

## RAG Evaluation Checks (20 Checks)
- [Retrieval Relevance](https://www.layaeval.in/metrics/retrieval-relevance): Evaluates whether fetched documents match the user query.
- [Document Relevance](https://www.layaeval.in/metrics/document-relevance): Checks if an individual retrieved document is pertinent to the question.
- [Context Sufficiency](https://www.layaeval.in/metrics/context-sufficiency): Tests if retrieved sources contain enough information to formulate an answer without guessing.
- [Context Coverage](https://www.layaeval.in/metrics/context-coverage): Measures what proportion of necessary information is captured across the context.
- [Retrieval Completeness](https://www.layaeval.in/metrics/retrieval-completeness): Verifies whether all critical pieces of evidence were retrieved.
- [Groundedness](https://www.layaeval.in/metrics/groundedness): Determines if every sentence in the response is directly supported by the context.
- [Strict Groundedness](https://www.layaeval.in/metrics/strict-groundedness): Binary zero-tolerance check; fails immediately if even a single unproven claim exists.
- [Hallucination Risk](https://www.layaeval.in/metrics/hallucination-risk): Quantifies whether the model generated unsupported or fabricated statements.
- [Response Relevance](https://www.layaeval.in/metrics/response-relevance): Measures if the response directly addresses the user question.
- [Response Completeness](https://www.layaeval.in/metrics/response-completeness): Checks if every aspect of a multi-part prompt was addressed.
- [Factual Correctness](https://www.layaeval.in/metrics/factual-correctness): Compares the model output against an authoritative ground truth.
- [Answerability](https://www.layaeval.in/metrics/answerability): Determines if the question can be answered strictly from the supplied sources.
- [Abstention Correctness](https://www.layaeval.in/metrics/abstention-correctness): Validates that the model refrains from answering when source evidence is lacking.
- [Context Utilization](https://www.layaeval.in/metrics/context-utilization): Assesses whether the model actively used provided context rather than internal priors.
- [Citation Correctness](https://www.layaeval.in/metrics/citation-correctness): Verifies each citation marker references a passage that genuinely supports the assertion.
- [Citation Completeness](https://www.layaeval.in/metrics/citation-completeness): Ensures all factual claims are properly paired with a citation tag.
- [Citation Relevance](https://www.layaeval.in/metrics/citation-relevance): Evaluates whether cited sources are relevant to the claim being supported.
- [Claim Support](https://www.layaeval.in/metrics/claim-support): Checks whether an isolated statement is substantiated by specific evidence snippets.
- [Contradiction Detection](https://www.layaeval.in/metrics/contradiction-detection): Flags direct or implicit contradictions between model output and provided sources.
- [Context Conflict](https://www.layaeval.in/metrics/context-conflict): Identifies conflicting statements across different retrieved reference documents.

## Agent Evaluation Checks (22 Checks)
- [Intent Resolution](https://www.layaeval.in/metrics/intent-resolution): Checks if the agent categorized the user request correctly (action, query, troubleshooting).
- [Task Completion](https://www.layaeval.in/metrics/task-completion): Determines whether the agent carried the multi-step task through to completion.
- [Task Adherence](https://www.layaeval.in/metrics/task-adherence): Ensures the agent stayed within specified operational parameters and rules.
- [Goal Achievement](https://www.layaeval.in/metrics/goal-achievement): Evaluates whether the user's objective was achieved by the agent's actions.
- [Tool Selection](https://www.layaeval.in/metrics/tool-selection): Verifies whether the agent chose the optimal tool for the current step.
- [Tool Selection Correctness](https://www.layaeval.in/metrics/tool-selection-correctness): Checks whether the selected tool matches task requirements.
- [Tool Call Accuracy](https://www.layaeval.in/metrics/tool-call-accuracy): Checks the syntactic and semantic structure of the agent's tool invocation.
- [Tool Input Accuracy](https://www.layaeval.in/metrics/tool-input-accuracy): Confirms parameter types, values, and formats passed to tools conform to requirements.
- [Tool Output Use](https://www.layaeval.in/metrics/tool-output-use): Verifies tool execution responses were ingested into subsequent reasoning steps.
- [Tool Necessity](https://www.layaeval.in/metrics/tool-necessity): Checks if the tool call was necessary or if context already held the answer.
- [Tool Redundancy](https://www.layaeval.in/metrics/tool-redundancy): Flags redundant, duplicate tool calls that waste tokens and latency.
- [Tool Sequence](https://www.layaeval.in/metrics/tool-sequence): Evaluates whether tools were invoked in a logical order.
- [Navigation Efficiency](https://www.layaeval.in/metrics/navigation-efficiency): Tests if the agent navigated directly without cyclic reasoning or unnecessary steps.
- [Agent Groundedness](https://www.layaeval.in/metrics/agent-groundedness): Confirms final agent responses are grounded in tool outputs and environment context.
- [Agent Hallucination](https://www.layaeval.in/metrics/agent-hallucination): Detects agent claims about external state that never occurred in the trace.
- [Agent Response Relevance](https://www.layaeval.in/metrics/agent-response-relevance): Measures whether the agent's final response addresses the original request.
- [Answer Completeness](https://www.layaeval.in/metrics/answer-completeness): Verifies that all requested requirements were covered in the final reply.
- [Agent Correctness](https://www.layaeval.in/metrics/agent-correctness): Checks final results against expected ground truth or golden execution state.
- [Agent Abstention](https://www.layaeval.in/metrics/agent-abstention): Validates that the agent safely refrains from actions it lacks permissions or capability for.
- [Policy Compliance](https://www.layaeval.in/metrics/policy-compliance): Verifies agent execution adheres to corporate policies and safety boundaries.
- [Prompt Injection](https://www.layaeval.in/metrics/prompt-injection): Detects direct prompt injection attempts in user input.
- [Indirect Prompt Injection](https://www.layaeval.in/metrics/indirect-prompt-injection): Identifies hidden adversarial injection payloads inside fetched websites or tool outputs.

## Architecture & Integration
- Compatible with: LangChain, LlamaIndex, CrewAI, AutoGen, LangGraph, DSPy, or vanilla Python.
- Execution model: Local in-process evaluation with sub-10ms latency.
- Return type: `Result(passed: bool, value: float, confidence: float)`