GoodMemTroubleshoot tracing
Verify local spans, collector receipt, and backend delivery before diagnosis.
Troubleshoot tracing
Use one known request to locate the export stage that fails. The tables below connect each symptom to its likely cause. A healthy server can still have an incorrect exporter endpoint, rejected credentials, or a sampler that excludes the request.
Verify each stage
The response goodmem-request-id header matches the span attribute goodmem.request.id, which is separate from the trace ID.
| Stage | Action | What it confirms |
|---|---|---|
| Local span output | Select logging on a test server. Find the request span in its output | GoodMem creates and samples the span |
| Collector receipt | Select otlp. Find the request ID in the collector debug output | The collector receives the span over OTLP |
| Backend receipt | Find the trace in the destination backend or its API | The backend stores the trace for that request |
For the local check, set these variables on a test server:
OTEL_TRACES_EXPORTER=logging
OTEL_TRACES_SAMPLER=always_on
GOODMEM_OTEL_EXPORT_PROFILE=neutral- Restart the server.
- Send the verification request.
- Find its
goodmem.request.idin the server output.
The local output contains span identifiers and metadata. Restore the previous exporter after this check.
For the collector check, use the debug exporter configuration.
- Set the GoodMem trace exporter to
otlp. - Select the collector endpoint and protocol.
- Restart the server or recreate its container.
- Send another verification request.
- Find its request ID and trace ID in the collector output.
For direct backend export, omit the collector check. At the backend, check the service name, request metadata, and expected parent and child spans. If the vendor converts trace identifiers, match the request through its metadata.
No traces appear
| Check | Action |
|---|---|
| Export is off | Set OTEL_TRACES_EXPORTER=otlp |
| SDK is off | Remove OTEL_SDK_DISABLED=true |
| Container lacks the variables | Add the variables to the service environment before container recreation |
| JVM property overrides an environment value | Check the Java launch properties |
| Startup configuration failed | Check goodmem_otel_initialization_failed: 1 means initialization failed; 0 confirms no startup failure, not delivery |
| Sampler excludes the request | For a controlled check, select always_on |
| External parent is unsampled | Check the parent decision and the configured sampler |
| Wrong backend view | Check the service, project, region, and time window |
Collector receipt fails
| Symptom | Check |
|---|---|
| Connection refused or timeout | Receiver address, port, network policy, and container network |
| Receiver works from the host only | Use the host gateway address instead of the server container's localhost |
| HTTP route error | Generic HTTP endpoints gain /v1/traces; trace-specific endpoints use the supplied path |
| Protocol error | Match grpc or http/protobuf to the receiver |
| Langfuse rejects export | Select http/protobuf; the SDK default is grpc |
| Authentication fails | Check the destination credentials and percent-encoded header values |
| TLS fails | Check the CA, hostname, mounted paths, and paired mTLS certificate/key |
Check both generic and trace-specific settings against the configuration reference. Keep credential headers out of diagnostic output.
Collector receipt succeeds but backend receipt fails
Check the destination exporter diagnostics, backend project, collector filters, and destination transforms. For LangSmith, check the parent policy.
A fan-out collector needs the neutral GoodMem profile and separate destination transforms.
A filter that retains only gen_ai.* spans removes API and retrieval ancestors from the trace.
The trace appears incomplete
| Symptom | Explanation |
|---|---|
| Input and output panels are empty | The trace data controls exclude content |
| Token usage is absent | The provider did not supply that counter, or the provider interface lacks it |
| Cost is absent | The backend needs usage data and its own price configuration |
| A job appears in another trace | Each background job attempt has its own trace; use the job ID to correlate attempts |
| Final spans are absent after shutdown | Check the termination grace period and exporter availability during shutdown |
Spans disappear under load
Check collector availability and export errors first. If export works, inspect the batch queue and export timeouts. A larger queue absorbs short bursts but uses more memory; sustained export failure still causes span loss. Use the batch controls to fit the deployment.