DeepEval

Expose the GoodMem RAG API as DeepEval-flavoured operation classes for use in evaluation scripts and test harnesses.

Stable· Python

Overview

GoodMem is a memory layer for AI agents that handles embedding, vector search, reranking, and LLM-powered answering server-side. The DeepEval integration wraps those operations — creating spaces, storing memories, and semantic retrieval with optional reranking and LLM summaries — as DeepEval-flavoured operation classes. Each class exposes a run(...) method that returns a JSON string, so you can call it from any DeepEval evaluation script, test harness, or scoring pipeline.

Installation

pip install deepeval-goodmem deepeval

The pip package is deepeval-goodmem; the import name is deepeval_goodmem. For local development, install with the dev extras via pip install -e ".[dev]".

Configuration

Variable	Description
`GOODMEM_BASE_URL`	Base URL of your GoodMem server (no `/v1` suffix), e.g. `https://your-goodmem-server.example.com`.
`GOODMEM_API_KEY`	API key sent as `X-API-Key` to authenticate requests.
`GOODMEM_VERIFY_SSL`	Set to `false` to skip TLS verification for self-signed dev certs (default: `true`).

When these environment variables are set, every operation constructor can be called with no arguments.

Quick start

import json
import os

os.environ["GOODMEM_BASE_URL"] = "https://your-goodmem-server.example.com"
os.environ["GOODMEM_API_KEY"] = "gm_xxxxxxxxxxxxxxxxxxxxxxxx"

from deepeval_goodmem import (
    GoodMemCreateMemory,
    GoodMemCreateSpace,
    GoodMemListEmbedders,
    GoodMemRetrieveMemories,
)

embedders = json.loads(GoodMemListEmbedders().run())
embedder_id = embedders["embedders"][0]["embedderId"]

space = json.loads(
    GoodMemCreateSpace().run(name="quickstart", embedder_id=embedder_id)
)
space_id = space["spaceId"]

GoodMemCreateMemory().run(
    space_id=space_id,
    text_content="The capital of France is Paris.",
)

results = json.loads(
    GoodMemRetrieveMemories().run(
        query="What is the capital of France?",
        space_ids=space_id,
        max_results=3,
    )
)
print(results)

Local development. The example above uses a TLS-verified production URL. If you are running GoodMem locally with a self-signed certificate, use https://localhost:8080 and disable TLS verification with GOODMEM_VERIFY_SSL=false. Keep TLS verification enabled for any deployed server.

Evaluate retrieval quality

The wrapper returns matched chunks under results. Feed their chunkText values into a DeepEval test case as retrieval_context, alongside the answer produced by your application:

from deepeval import evaluate
from deepeval.metrics import ContextualRelevancyMetric
from deepeval.test_case import LLMTestCase

query = "What is the capital of France?"
retrieval = json.loads(
    GoodMemRetrieveMemories().run(query=query, space_ids=space_id, max_results=3)
)

# Replace this with the answer produced by your RAG application.
answer = "Paris is the capital of France."
test_case = LLMTestCase(
    input=query,
    actual_output=answer,
    retrieval_context=[item["chunkText"] for item in retrieval["results"]],
)

evaluate(
    test_cases=[test_case],
    metrics=[ContextualRelevancyMetric(threshold=0.7)],
)

ContextualRelevancyMetric uses an LLM judge, so configure the model credentials required by your DeepEval setup before running the example.

Retrieval post-processing

GoodMem's retrieval call accepts these optional post-processing parameters (each framework exposes them under its own naming convention):

Parameter	Range	Description
`reranker_id`	UUID	Reranker model that reorders matched chunks by relevance.
`llm_id`	UUID	LLM that generates a contextual answer (`abstractReply`) alongside the chunks.
`relevance_threshold`	number	Minimum score for inclusion; scores are reranker-specific.
`llm_temperature`	0–2	Sampling temperature for the LLM post-processor.
`max_results`	integer	Maximum number of results to return.
`chronological_resort`	boolean	Re-sort the final results by creation time instead of relevance.

Learn more

The full README, examples, and API reference are available on the PyPI package page.