DeepEval

Retrieve from GoodMem as a traced DeepEval retriever span and evaluate retrieval quality with DeepEval's RAG metrics.

Stable· Python

Overview

GoodMem is a memory layer for AI agents that handles embedding, vector search, reranking, and LLM-powered answering server-side. The DeepEval integration exposes that retrieval as a GoodMemRetriever: every search() call is traced as a DeepEval retriever span, and a retrieval turns directly into an LLMTestCase with retrieval_context populated via to_test_case() — which is what DeepEval's ContextualPrecisionMetric, ContextualRecallMetric, ContextualRelevancyMetric, and FaithfulnessMetric read. A companion GoodMemRetrievalHealthMetric scores the retrieval itself, so a degraded retrieval (for example a failed reranker) shows up as its own failure rather than as a mysterious drop in recall. The package also exports GoodMemConnection (connection settings) and GoodMemSpaceError (raised when a space name cannot be resolved unambiguously).

Installation

pip install goodmem-deepeval deepeval

The pip package is goodmem-deepeval; the import name is goodmem_deepeval.

Configuration

Variable	Description
`GOODMEM_BASE_URL`	Base URL of your GoodMem server (no `/v1` suffix), e.g. `https://your-goodmem-server.example.com`.
`GOODMEM_API_KEY`	API key used to authenticate requests (starts with `gm_`).

When these environment variables are set, the retriever needs no connection arguments. You can also pass base_url and api_key (plus verify_ssl, timeout, or an injected client) as constructor keywords. Either way, credentials are deliberately not pydantic model fields — they live on a private connection — so they can never reach a model_dump(), a repr(), or a trace payload.

Quick start

from deepeval import evaluate
from deepeval.metrics import ContextualRelevancyMetric, ContextualRecallMetric
from goodmem_deepeval import GoodMemRetriever, GoodMemRetrievalHealthMetric

retriever = GoodMemRetriever(space_name="docs", limit=5)   # credentials from the environment

result = retriever.search("how do I rotate an API key?")
answer = my_llm(result["hits"])                            # your generation step
expected = "..."                                           # your reference answer

case = retriever.to_test_case(result, actual_output=answer, expected_output=expected)

evaluate([case], [
    ContextualRelevancyMetric(),
    ContextualRecallMetric(),
    GoodMemRetrievalHealthMetric(),
])

ContextualRecallMetric and ContextualPrecisionMetric compare against a reference answer, so they need expected_output; ContextualRelevancyMetric and FaithfulnessMetric do not. The LLM-judge metrics need model credentials configured for your DeepEval setup; GoodMemRetrievalHealthMetric uses no LLM at all.

If you only need the retrieved context, retrieve() returns the chunk text directly:

context = retriever.retrieve("how do I rotate an API key?")   # list[str]

Local development. If you are running GoodMem locally with a self-signed certificate, pass verify_ssl=False (or a CA bundle path) as a constructor keyword. Keep TLS verification enabled for any deployed server.

What `search()` returns

search() reports the query, latency, top_k, and embedder on the retriever span, and returns a dictionary:

Key	What it is
`query`	The query, as passed.
`hits`	`chunk_id`, `chunk_text`, `memory_id`, `space_id`, `source`, `score`, `score_kind`, `metadata` — in the server's order.
`score_kind`	`"vector"` or `"reranker"`: what the server actually returned, not what was configured. The two scales differ and are never mixed or re-sorted.
`statuses`	Status entries indicating a real problem; `[]` when clean.
`partial`	`True` when the server reported a problem during this retrieval, with or without hits.
`abstract_reply`	The server-generated summary, only when `llm_id` is set.
`space_ids`	Which spaces were actually searched.

A retrieval that reported a problem and returned nothing comes back as empty hits with partial: True and a Python warning — never an exception, and always distinguishable from "no matches".

The health metric

GoodMemRetrievalHealthMetric(threshold=1.0, require_hits=False) reads the diagnostics that to_test_case() places in metadata["goodmem"] and scores the retrieval itself: 1.0 when the server reported nothing wrong, 0.0 when the retrieval was degraded, with the server's status codes in reason and score_breakdown. Set require_hits=True to also fail a case whose retrieval returned no chunks. A test case with no GoodMem metadata is skipped rather than failed.

Retrieval options

GoodMemRetriever takes the space to search (space_id, space_ids, or space_name — at least one is required) plus these optional constructor arguments:

Argument	Description
`limit`	Maximum number of results to return (default 5). Also overridable per `search()` call.
`fetch_k`	Candidates to retrieve before reranking; defaults to `limit`.
`reranker_id`	Reranker model that reorders matched chunks by relevance.
`min_score`	Client-side minimum score, applied only to reranker scores. Reranker scales are model-dependent, so calibrate it for the reranker in use; there is no default.
`filter`	A GoodMem filter expression applied to every space searched.
`metadata_filter`	Field/value pairs AND-ed into the filter; each value is compared as its own type (`str` as TEXT, `bool` as BOOLEAN, `int`/`float` as NUMERIC).
`llm_id`	LLM that generates a contextual answer (`abstract_reply`) alongside the chunks. Costs an LLM call per search.
`llm_temperature`	Sampling temperature for the LLM post-processor.
`embedder_id`	Used with `space_name` to check (or create) the space's embedder.
`create_space`	With `space_name`, create the space when it does not exist (default `False`).

Attach-by-name is idempotent reuse: an existing space whose embedder matches embedder_id is reused, while a different embedder — or an ambiguous name — raises GoodMemSpaceError, because retrieving across mismatched embedders returns plausible-looking nonsense.

Per-call filters compose with the constructor's:

retriever = GoodMemRetriever(space_name="docs", metadata_filter={"category": "billing"})
retriever.search("refunds", metadata_filter={"lang": "en", "archived": False})  # AND-ed per call

Learn more

The full README, examples, and API reference are available on the PyPI package page.

DeepEval

On this page