Building a Basic RAG Agent with GoodMem in JavaScript

Overview

This tutorial will guide you through building a complete Retrieval-Augmented Generation (RAG) system using GoodMem's vector memory capabilities with JavaScript. By the end of this guide, you'll have a functional Q&A system that can:

🔍 Semantically search through your documents
📝 Generate contextual answers using retrieved information
🏗️ Scale to handle large document collections

What is RAG?

RAG combines the power of retrieval (finding relevant information) with generation (creating natural language responses). This approach allows AI systems to provide accurate, context-aware answers by:

Retrieving relevant documents from a knowledge base
Augmenting the query with this context
Generating a comprehensive answer using both the query and retrieved information

Why GoodMem for RAG?

GoodMem provides enterprise-grade vector storage with:

Multiple embedder support for optimal retrieval accuracy
Streaming APIs for real-time responses
Advanced post-processing with reranking and summarization
Scalable architecture for production workloads

Prerequisites

Before starting, ensure you have:

✅ GoodMem server running (install with: curl -s https://get.goodmem.ai | bash)
✅ Node.js 14+ installed
✅ npm or yarn for package management
✅ API key for your GoodMem instance
✅ OpenAI API key (optional, for the complete RAG demo)

Installation & Setup

First, let's install the required packages:

#!javascript
// Install the GoodMem client library
// Run in your terminal: npm install @pairsystems/goodmem-client --save

console.log('📦 To install dependencies, run:');
console.log('   npm install @pairsystems/goodmem-client');
console.log('\n💡 Make sure Node.js 14+ is installed');

Authentication & Configuration

Let's configure our GoodMem client and test the connection:

const GoodMemClient = require('@pairsystems/goodmem-client');
const fs = require('fs').promises;
const path = require('path');

// Configuration - Update these values for your setup
const GOODMEM_HOST = process.env.GOODMEM_HOST || 'http://localhost:8080';
const GOODMEM_API_KEY = process.env.GOODMEM_API_KEY || 'your-api-key-here';

console.log('GoodMem Host:', GOODMEM_HOST);
console.log('API Key configured:', GOODMEM_API_KEY !== 'your-api-key-here' ? 'Yes' : 'No - Please update');

// Create and configure API client
const apiClient = new GoodMemClient.ApiClient();
apiClient.basePath = GOODMEM_HOST;
apiClient.defaultHeaders = {
    'X-API-Key': GOODMEM_API_KEY,
    'Content-Type': 'application/json',
    'Accept': 'application/json'
};

// Create API instances
const spacesApi = new GoodMemClient.SpacesApi(apiClient);
const memoriesApi = new GoodMemClient.MemoriesApi(apiClient);
const embeddersApi = new GoodMemClient.EmbeddersApi(apiClient);

// Create streaming client
let streamingClient = null;
try {
    streamingClient = new GoodMemClient.StreamingClient(apiClient);
    console.log('✅ StreamingClient initialized successfully');
} catch (error) {
    console.log('⚠️  StreamingClient not available:', error.message);
}

console.log('✅ GoodMem client configured successfully!');

Test Connection

Let's verify we can connect to the GoodMem server:

// Test connection by listing existing spaces
spacesApi.listSpaces()
    .then(response => {
        console.log('✅ Successfully connected to GoodMem!');
        const spaces = response.spaces || [];
        console.log(`   Found ${spaces.length} existing spaces`);
    })
    .catch(error => {
        console.log('❌ Error connecting to GoodMem:', error.message);
        console.log('   Please check your API key and host configuration');
        if (error.status) {
            console.log('   Response code:', error.status);
        }
    });

Creating Your First Space

In GoodMem, a Space is a logical container for organizing memories. Each space has:

Associated embedders for generating vector representations
Access controls (public/private)
Metadata labels for organization

Let's create a space for our RAG demo:

// First, let's see what embedders are available
let availableEmbedders = [];
let defaultEmbedder = null;

embeddersApi.listEmbedders()
    .then(embeddersResponse => {
        availableEmbedders = embeddersResponse.embedders || [];
        
        console.log(`📋 Available Embedders (${availableEmbedders.length}):`);
        availableEmbedders.forEach((embedder, i) => {
            console.log(`   ${i + 1}. ${embedder.displayName} - ${embedder.providerType}`);
            console.log(`      Model: ${embedder.modelIdentifier || 'N/A'}`);
            console.log(`      ID: ${embedder.embedderId}`);
            console.log();
        });
        
        if (availableEmbedders.length > 0) {
            defaultEmbedder = availableEmbedders[0];
            console.log(`🎯 Using embedder: ${defaultEmbedder.displayName}`);
        } else {
            console.log('⚠️  No embedders found. You may need to configure an embedder first.');
            console.log('   Refer to the documentation: https://docs.goodmem.ai/docs/reference/cli/goodmem_embedder_create/');
        }
    })
    .catch(error => {
        console.log('❌ Error listing embedders:', error.message);
    });

// Create a space for our RAG demo
const SPACE_NAME = 'RAG Demo Knowledge Base (JavaScript)';
let demoSpace = null;

// Define chunking configuration that we'll reuse throughout the tutorial
const DEMO_CHUNKING_CONFIG = {
    recursive: {
        chunkSize: 256,
        chunkOverlap: 25,
        separators: ['\n\n', '\n', '. ', ' ', ''],
        keepStrategy: 'KEEP_END',
        separatorIsRegex: false,
        lengthMeasurement: 'CHARACTER_COUNT'
    }
};

console.log('📋 Demo Chunking Configuration:');
console.log(`   Chunk Size: ${DEMO_CHUNKING_CONFIG.recursive.chunkSize} characters`);
console.log(`   Overlap: ${DEMO_CHUNKING_CONFIG.recursive.chunkOverlap} characters`);
console.log(`   Strategy: ${DEMO_CHUNKING_CONFIG.recursive.keepStrategy}`);
console.log('   💡 This chunking config will be reused for all memory creation!');
console.log();

spacesApi.listSpaces()
    .then(existingSpaces => {
        // Check if space already exists
        const spaces = existingSpaces.spaces || [];
        const existing = spaces.find(space => space.name === SPACE_NAME);
        
        if (existing) {
            console.log(`📁 Space '${SPACE_NAME}' already exists`);
            console.log(`   Space ID: ${existing.spaceId}`);
            console.log('   To remove existing space, see https://docs.goodmem.ai/docs/reference/cli/goodmem_space_delete/');
            demoSpace = existing;
            return existing;
        }
        
        // Create space if it doesn't exist
        const spaceEmbedders = [];
        if (defaultEmbedder) {
            spaceEmbedders.push({
                embedderId: defaultEmbedder.embedderId,
                defaultRetrievalWeight: 1.0
            });
        }
        
        const createRequest = {
            name: SPACE_NAME,
            labels: {
                purpose: 'rag-demo',
                environment: 'tutorial',
                'content-type': 'documentation',
                language: 'javascript'
            },
            spaceEmbedders: spaceEmbedders,
            publicRead: false,
            defaultChunkingConfig: DEMO_CHUNKING_CONFIG
        };
        
        return spacesApi.createSpace(createRequest);
    })
    .then(space => {
        if (!demoSpace) {
            demoSpace = space;
            console.log(`✅ Created space: ${space.name}`);
            console.log(`   Space ID: ${space.spaceId}`);
            console.log(`   Embedders: ${space.spaceEmbedders ? space.spaceEmbedders.length : 0}`);
            console.log(`   Labels: ${JSON.stringify(space.labels)}`);
            console.log(`   Chunking Config Saved: ${DEMO_CHUNKING_CONFIG.recursive.chunkSize} chars with ${DEMO_CHUNKING_CONFIG.recursive.chunkOverlap} overlap`);
        }
    })
    .catch(error => {
        console.log('❌ Error creating space:', error.message);
        if (error.status) {
            console.log('   Response code:', error.status);
        }
    });

// Verify our space configuration
if (demoSpace) {
    spacesApi.getSpace(demoSpace.spaceId)
        .then(spaceDetails => {
            console.log('🔍 Space Configuration:');
            console.log(`   Name: ${spaceDetails.name}`);
            console.log(`   Owner ID: ${spaceDetails.ownerId}`);
            console.log(`   Public Read: ${spaceDetails.publicRead}`);
            console.log(`   Created: ${new Date(spaceDetails.createdAt).toISOString()}`);
            console.log(`   Labels: ${JSON.stringify(spaceDetails.labels)}`);
            
            console.log('\n🤖 Associated Embedders:');
            if (spaceDetails.spaceEmbedders && spaceDetails.spaceEmbedders.length > 0) {
                spaceDetails.spaceEmbedders.forEach(embedderAssoc => {
                    console.log(`   Embedder ID: ${embedderAssoc.embedderId}`);
                    console.log(`   Retrieval Weight: ${embedderAssoc.defaultRetrievalWeight}`);
                });
            } else {
                console.log('   No embedders configured');
            }
        })
        .catch(error => {
            console.log('❌ Error getting space details:', error.message);
        });
} else {
    console.log('⚠️  No space available for the demo');
}

Adding Documents to Memory

Now let's add some sample documents to our space. GoodMem will automatically:

Chunk the documents into optimal sizes
Generate embeddings using the configured embedders
Index the content for fast retrieval

We'll use sample company documents that represent common business use cases:

// Load our sample documents
async function loadSampleDocuments() {
    const documents = [];
    const sampleDir = 'sample_documents';
    
    const docFiles = {
        'company_handbook.txt': 'Employee handbook with policies and procedures',
        'technical_documentation.txt': 'API documentation and technical guides',
        'product_faq.txt': 'Frequently asked questions about products',
        'security_policy.txt': 'Information security policies and procedures'
    };
    
    for (const [filename, description] of Object.entries(docFiles)) {
        const filepath = path.join(sampleDir, filename);
        
        try {
            const content = await fs.readFile(filepath, 'utf-8');
            documents.push({ filename, description, content });
            console.log(`📄 Loaded: ${filename} (${content.length.toLocaleString()} characters)`);
        } catch (error) {
            console.log(`⚠️  File not found: ${filepath}`);
        }
    }
    
    return documents;
}

// Load the documents
let sampleDocs = [];
loadSampleDocuments()
    .then(docs => {
        sampleDocs = docs;
        console.log(`\n📚 Total documents loaded: ${docs.length}`);
    })
    .catch(error => {
        console.log('❌ Error loading documents:', error.message);
    });

// Create the first memory individually to demonstrate single memory creation
function createSingleMemory(spaceId, document) {
    const memoryRequest = {
        spaceId: spaceId,
        originalContent: document.content,
        contentType: 'text/plain',
        chunkingConfig: DEMO_CHUNKING_CONFIG,
        metadata: {
            filename: document.filename,
            description: document.description,
            source: 'sample_documents',
            document_type: document.filename.split('_')[0],
            ingestion_method: 'single'
        }
    };
    
    return memoriesApi.createMemory(memoryRequest)
        .then(memory => {
            console.log(`✅ Created single memory: ${document.filename}`);
            console.log(`   Memory ID: ${memory.memoryId}`);
            console.log(`   Status: ${memory.processingStatus}`);
            console.log(`   Content Length: ${document.content.length} characters`);
            console.log();
            return memory;
        })
        .catch(error => {
            console.log(`❌ Error creating memory for ${document.filename}:`, error.message);
            return null;
        });
}

let singleMemory = null;
if (demoSpace && sampleDocs.length > 0) {
    const firstDoc = sampleDocs[0];
    console.log('📝 Creating first document using CreateMemory API:');
    console.log(`   Document: ${firstDoc.filename}`);
    console.log('   Method: Individual memory creation');
    console.log();
    
    createSingleMemory(demoSpace.spaceId, firstDoc)
        .then(memory => {
            singleMemory = memory;
            if (memory) {
                console.log('🎯 Single memory creation completed successfully!');
            } else {
                console.log('⚠️  Single memory creation failed');
            }
        });
} else {
    console.log('⚠️  Cannot create memory: missing space or documents');
}

// Demonstrate retrieving a memory by ID using getMemory
if (singleMemory) {
    console.log('📖 Retrieving memory details using getMemory API:');
    console.log(`   Memory ID: ${singleMemory.memoryId}`);
    console.log();
    
    // Retrieve the memory without content
    memoriesApi.getMemory(singleMemory.memoryId, { includeContent: false })
        .then(retrievedMemory => {
            console.log('✅ Successfully retrieved memory:');
            console.log(`   Memory ID: ${retrievedMemory.memoryId}`);
            console.log(`   Space ID: ${retrievedMemory.spaceId}`);
            console.log(`   Status: ${retrievedMemory.processingStatus}`);
            console.log(`   Content Type: ${retrievedMemory.contentType}`);
            console.log(`   Created At: ${new Date(retrievedMemory.createdAt).toISOString()}`);
            console.log(`   Updated At: ${new Date(retrievedMemory.updatedAt).toISOString()}`);
            
            if (retrievedMemory.metadata) {
                console.log('\n   📋 Metadata:');
                for (const [key, value] of Object.entries(retrievedMemory.metadata)) {
                    console.log(`      ${key}: ${value}`);
                }
            }
            
            // Now retrieve with content included
            console.log('\n📖 Retrieving memory with content:');
            return memoriesApi.getMemory(singleMemory.memoryId, { includeContent: true });
        })
        .then(retrievedWithContent => {
            if (retrievedWithContent.originalContent) {
                // Decode the base64 encoded content
                const decodedContent = Buffer.from(retrievedWithContent.originalContent, 'base64').toString('utf-8');
                
                console.log('✅ Content retrieved and decoded:');
                console.log(`   Content length: ${decodedContent.length} characters`);
                const preview = decodedContent.length > 200 ? decodedContent.substring(0, 200) + '...' : decodedContent;
                console.log(`   First 200 chars: ${preview}`);
            } else {
                console.log('⚠️  No content available');
            }
        })
        .catch(error => {
            console.log('❌ Error retrieving memory:', error.message);
            if (error.status) {
                console.log('   Status code:', error.status);
            }
        });
} else {
    console.log('⚠️  No memory available to retrieve');
}

// Create the remaining documents using batch memory creation
function createBatchMemories(spaceId, documents) {
    const memoryRequests = documents.map(doc => ({
        spaceId: spaceId,
        originalContent: doc.content,
        contentType: 'text/plain',
        chunkingConfig: DEMO_CHUNKING_CONFIG,
        metadata: {
            filename: doc.filename,
            description: doc.description,
            source: 'sample_documents',
            document_type: doc.filename.split('_')[0],
            ingestion_method: 'batch'
        }
    }));
    
    const batchRequest = {
        requests: memoryRequests
    };
    
    console.log(`📦 Creating ${memoryRequests.length} memories using BatchCreateMemory API:`);
    
    return memoriesApi.batchCreateMemory(batchRequest)
        .then(() => {
            console.log('✅ Batch creation request submitted successfully');
        })
        .catch(error => {
            console.log('❌ Error during batch creation:', error.message);
            if (error.status) {
                console.log('   Response code:', error.status);
            }
        });
}

if (demoSpace && sampleDocs.length > 1) {
    const remainingDocs = sampleDocs.slice(1);
    createBatchMemories(demoSpace.spaceId, remainingDocs)
        .then(() => {
            console.log('\n📋 Total Memory Creation Summary:');
            console.log('   📄 Single CreateMemory: 1 document');
            console.log(`   📦 Batch CreateMemory: ${remainingDocs.length} documents submitted`);
            console.log('   ⏳ Check processing status in the next cell');
        });
} else {
    console.log('⚠️  Cannot create batch memories: insufficient documents or missing space');
}

// List all memories in our space to verify they're ready
if (demoSpace) {
    memoriesApi.listMemories(demoSpace.spaceId)
        .then(memoriesResponse => {
            const memories = memoriesResponse.memories || [];
            
            console.log(`📚 Memories in space '${demoSpace.name}':`);
            console.log(`   Total memories: ${memories.length}`);
            console.log();
            
            memories.forEach((memory, i) => {
                const metadata = memory.metadata || {};
                const filename = metadata.filename || 'Unknown';
                const description = metadata.description || 'No description';
                
                console.log(`   ${i + 1}. ${filename}`);
                console.log(`      Status: ${memory.processingStatus}`);
                console.log(`      Description: ${description}`);
                console.log(`      Created: ${new Date(memory.createdAt).toISOString()}`);
                console.log();
            });
        })
        .catch(error => {
            console.log('❌ Error listing memories:', error.message);
        });
}

// Monitor processing status for all created memories
async function waitForProcessingCompletion(spaceId, maxWaitSeconds = 120) {
    console.log('⏳ Waiting for document processing to complete...');
    console.log('   💡 Note: Batch memories are processed asynchronously, so we check by listing all memories in the space');
    console.log();
    
    const startTime = Date.now();
    const maxWaitMs = maxWaitSeconds * 1000;
    
    while (Date.now() - startTime < maxWaitMs) {
        try {
            const memoriesResponse = await memoriesApi.listMemories(spaceId);
            const memories = memoriesResponse.memories || [];
            
            // Check processing status
            const statusCounts = {};
            memories.forEach(memory => {
                const status = memory.processingStatus;
                statusCounts[status] = (statusCounts[status] || 0) + 1;
            });
            
            console.log(`📊 Processing status: ${JSON.stringify(statusCounts)} (Total: ${memories.length} memories)`);
            
            // Check if all are completed
            const allCompleted = memories.every(memory => memory.processingStatus === 'COMPLETED');
            if (allCompleted) {
                console.log('✅ All documents processed successfully!');
                return true;
            }
            
            // Check for any failures
            const failedCount = statusCounts['FAILED'] || 0;
            if (failedCount > 0) {
                console.log(`❌ ${failedCount} memories failed processing`);
                return false;
            }
            
            await new Promise(resolve => setTimeout(resolve, 5000)); // Wait 5 seconds
        } catch (error) {
            console.log('❌ Error checking processing status:', error.message);
            return false;
        }
    }
    
    console.log(`⏰ Timeout waiting for processing (waited ${maxWaitSeconds}s)`);
    return false;
}

if (demoSpace) {
    waitForProcessingCompletion(demoSpace.spaceId)
        .then(processingComplete => {
            if (processingComplete) {
                console.log('🎉 Ready for semantic search and retrieval!');
                console.log('📈 Batch API benefit: Multiple documents submitted in a single API call');
                console.log('🔧 Consistent chunking: All memories use DEMO_CHUNKING_CONFIG');
            } else {
                console.log('⚠️  Some documents may still be processing. You can continue with the tutorial.');
            }
        });
} else {
    console.log('⚠️  Skipping processing check - no space available');
}

Semantic Search & Retrieval

Now comes the exciting part! Let's perform semantic search using GoodMem's streaming API. This will:

Find relevant chunks based on semantic similarity
Stream results in real-time
Include relevance scores for ranking
Return structured data for easy processing

// Perform semantic search using GoodMem's streaming API
async function semanticSearchStreaming(query, spaceId, maxResults = 5) {
    console.log(`🔍 Streaming search for: '${query}'`);
    console.log(`📁 Space ID: ${spaceId}`);
    console.log(`📊 Max results: ${maxResults}`);
    console.log('-'.repeat(50));
    
    try {
        const controller = new AbortController();
        const timeoutId = setTimeout(() => {
            console.log('⏰ Streaming timeout after 30 seconds');
            controller.abort();
        }, 30000);
        
        // Create request object with proper structure
        const request = {
            message: query,
            spaceIds: [spaceId],
            requestedSize: maxResults,
            fetchMemory: true,
            fetchMemoryContent: false,
            format: GoodMemClient.StreamingClient.StreamingFormat.NDJSON
        };
        
        const stream = await streamingClient.retrieveMemoryStream(
            controller.signal,
            request
        );
        
        const retrievedChunks = [];
        let eventCount = 0;
        
        for await (const event of stream) {
            eventCount++;
            
            if (event.retrievedItem && event.retrievedItem.chunk) {
                const chunkRef = event.retrievedItem.chunk;
                const chunkData = chunkRef.chunk || {};
                const chunkText = chunkData.chunkText || '';
                
                retrievedChunks.push({
                    chunkText: chunkText,
                    relevanceScore: chunkRef.relevanceScore,
                    memoryIndex: chunkRef.memoryIndex,
                    resultSetId: chunkRef.resultSetId,
                    chunkSequence: chunkData.chunkSequenceNumber || 0
                });
                
                console.log(`📄 Chunk ${retrievedChunks.length}:`);
                console.log(`   Relevance: ${chunkRef.relevanceScore.toFixed(3)}`);
                const preview = chunkText.length > 100 ? chunkText.substring(0, 100) + '...' : chunkText;
                console.log(`   Text: ${preview}`);
                console.log();
            } else if (event.resultSetBoundary) {
                console.log(`🔄 ${event.resultSetBoundary.kind}: ${event.resultSetBoundary.stageName}`);
            }
        }
        
        clearTimeout(timeoutId);
        
        console.log(`✅ Streaming search completed: ${retrievedChunks.length} chunks found, ${eventCount} events processed`);
        return retrievedChunks;
    } catch (error) {
        if (error.name === 'AbortError' || error.message.includes('aborted')) {
            console.log('Stream completed or cancelled');
        } else {
            console.log('❌ Streaming error:', error.message);
        }
        return [];
    }
}

// Test semantic search with a sample query
if (demoSpace && streamingClient) {
    const sampleQuery = 'What is the vacation policy for employees?';
    semanticSearchStreaming(sampleQuery, demoSpace.spaceId)
        .then(searchResults => {
            console.log(`\n✅ Search completed with ${searchResults.length} results`);
        });
} else if (!streamingClient) {
    console.log('⚠️  StreamingClient not available for search');
} else {
    console.log('⚠️  No space available for search');
}

// Let's try a few different queries to see how streaming semantic search works
async function testMultipleStreamingQueries(spaceId) {
    const testQueries = [
        'How do I reset my password?',
        'What are the security requirements for remote work?',
        'API authentication and rate limits',
        'Employee benefits and health insurance',
        'How much does the software cost?'
    ];
    
    for (let i = 0; i < testQueries.length; i++) {
        const query = testQueries[i];
        console.log(`\n🔍 Test Query ${i + 1}: ${query}`);
        console.log('='.repeat(60));
        
        await semanticSearchStreaming(query, spaceId, 3);
        
        console.log('\n' + '-'.repeat(60));
    }
}

if (demoSpace && streamingClient) {
    testMultipleStreamingQueries(demoSpace.spaceId)
        .then(() => console.log('\n✅ All queries completed'));
} else if (!streamingClient) {
    console.log('⚠️  StreamingClient not available for testing multiple queries');
} else {
    console.log('⚠️  No space available for testing multiple streaming queries');
}

Next Steps & Advanced Features

Congratulations! 🎉 You've successfully built a semantic search system using GoodMem. Here's what you've accomplished:

✅ What You Built

Document ingestion pipeline with automatic chunking and embedding
Semantic search system with relevance scoring
Streaming retrieval using GoodMem's real-time API

🚀 Next Steps for Advanced Implementation

1. Multiple Embedders & Reranking

Coming Soon

2. Integration with Popular Frameworks

Coming Soon

3. Advanced Post-Processing

Coming Soon

📚 Additional Resources

GoodMem Documentation:

Advanced Configuration Guide

JavaScript

On this page