What is ManasDB Core?

ManasDB Core is the Node.js-native memory layer for AI applications.

It is designed to be an alternative to cloud-dependent services like Mem0, providing intelligent data retrieval for RAG (Retrieval-Augmented Generation) systems natively within your own infrastructure.

Traditional RAG stacks require you to manually piece together multiple components (vector databases, embedding APIs, caching layers, deduplication, reranking). ManasDB Core consolidates this complexity into a single SDK that handles:

• Zero-Config Onboarding — Boots into MemoryProvider with zero database setup.
• Local Embeddings — Use Ollama or Transformers; no mandatory API keys.
• Data Privacy — Data stays on your server; no cloud lock-in.
• Semantic Caching — Up to 29x faster repeated queries.
• Governance & Budgeting — Hard USD spending caps for enterprise safety.

⚡ 10-Second Quick Start

Try @manasdb/core in your terminal right now with zero configuration. This uses the ephemeral MemoryProvider.

import { ManasDB } from "@manasdb/core";

const memory = new ManasDB(); // Zero config!
await memory.init();

await memory.absorb("ManasDB Core is a Node.js-native memory layer.");
const results = await memory.recall("What is ManasDB Core?");

console.log(results[0].text);

Why @manasdb/core vs Traditional Stacks

Most vector databases provide approximate nearest-neighbor (ANN) search and nothing else. To build a robust AI product, developers must manually build deduplication, PII redaction, hybrid search scoring (fusing sparse keyword + dense vector), and caching.

@manasdb/core pushes all of this retrieval intelligence directly into the storage layer. Unlike traditional stacks which require you to ship your proprietary user text to managed cloud APIs or manually glue together local models, @manasdb/core executes entirely inside your Node.js backend. You get enterprise-grade retrieval capabilities while remaining 100% air-gapped from third-party vendor clouds.

Quick Comparison Table

Is it right for me? (Decision Tree)

ManasDB Core shines for developers building production Node.js applications that require precision, cost-control, and privacy.

✓ You SHOULD use ManasDB Core if:

• You are building a Node.js monorepo or standard backend (Express, Next.js, Fastify).
• You use MongoDB Atlas (Vector Search) or PostgreSQL (`pgvector`).
• You have between 10K and 10M vectors.
• Data privacy is paramount and your data cannot be sent to a 3rd-party memory startup.

✕ You should NOT use ManasDB Core if:

• You are at billion-scale: Use dedicated systems like Pinecone or Milvus.
• You require GPU-accelerated ANN: It relies on CPU limits from Atlas and pgvector.
• You are building knowledge graphs: Use Neo4j; ManasDB Core is document-oriented.

Common Use Cases

• Customer Support: AI chatbot that accurately cites product docs and past ticket history.
• Developer Tools: Semantic search across API references and codebase changelogs.
• Legal / Compliance: Retrieving specific clauses from massive contracts with the PII Shield auto-redacting sensitive names and IDs.
• Healthcare: Secure patient-record Q&A where strict PII Shielding prevents HIPAA violations from hitting cloud LLMs.
• Enterprise Knowledge: Internal wikis that understand complex contextual reasoning rather than just performing keyword matches.

Can I use it in production?

Yes. ManasDB Core is built for enterprise-readiness and designed to scale predictably:

• Volume: Vector indexing scalability is offloaded to MongoDB Atlas $vectorSearch and Postgres pgvector, allowing comfortable scaling up to ~10M records.
• Concurrency Safety: The absorb() pipeline utilizes atomic $setOnInsert (Mongo) and ON CONFLICT (Postgres) upserts, preventing race conditions. Read paths are entirely stateless.
• Strict Mode Protection: ManasDB Core will fail-fast at runtime if a developer accidentally requests a database operation without a valid provider URI loaded.

Getting help & community

Since ManasDB Core is built to empower developers without requiring a support subscription, you are encouraged to join the direct community spaces:

• GitHub Repository: View the source code, open issues, and submit PRs at github.com/manasdb/manasdb
• Discussions: Ask questions or provide architecture feedback in the GitHub Discussions tab.
• NPM Module: Install locally via npm install @manasdb/core without any cloud keys.

Installation & Setup

1. Step-by-Step Manual Installation

If you prefer to install @manasdb/core into an existing architecture, follow these steps.

A. Install the Core SDK

The core SDK contains the engine, chunkers, and scoring logic. It doesn't include any database dependencies by default to keep your node_modules lean.

npm install @manasdb/core

B. Install Database Drivers

Install only the drivers for the databases you intend to use. ManasDB Core lazy-loads these dependencies to ensure 100% crash-free zero bloat.

npm install mongodb # For MongoDB Atlas
npm install pg # For PostgreSQL
npm install ioredis # Optional: For Tier 1 Redis Semantic Caching

3. Database Setup Guides

MongoDB Atlas Setup

1. Create a free or dedicated cluster on MongoDB Atlas.
2. Go to Database Access and create a user with read/write permissions.
3. Go to Network Access and whitelist your IP (or 0.0.0.0/0 for universal development access).
4. No need to manually create the Vector Search Index; ManasDB Core will attempt to provision it automatically upon first absorb().

PostgreSQL (pgvector) Setup

ManasDB Core requires the pgvector extension to perform native dense vector searches. If you are using a managed provider like Supabase, Neon, or AWS RDS, pgvector is already supported.

CREATE EXTENSION IF NOT EXISTS vector;

ManasDB Core will automatically create the required schemas and tables (like _manas_documents and _manas_vectors) on startup.

4. Docker Setup (docker-compose.yml)

If you want to run your databases locally without relying on cloud providers, you can use Docker. Below is a complete docker-compose.yml file that provisions PostgreSQL (with the pgvector extension pre-installed) and Redis for Tier 1 caching.

version: '3.8'
services:
  postgres-ai:
    image: pgvector/pgvector:pg16
    container_name: manasdb-postgres
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: password123
      POSTGRES_DB: vectors
    ports:
      - "5432:5432"
    volumes:
      - manasdb-pg-data:/var/lib/postgresql/data
    restart: unless-stopped

  redis-cache:
    image: redis:7-alpine
    container_name: manasdb-redis
    ports:
      - "6379:6379"
    restart: unless-stopped

volumes:
  manasdb-pg-data:

Run docker-compose up -d to start the infrastructure.

5. Environment Configuration

Create a .env file at the root of your Node.js project. You only need to define URIs for the services you are actively using.

MONGODB_URI=mongodb+srv://<user>:<pwd>@cluster.mongodb.net/
POSTGRES_URI=postgresql://admin:password123@localhost:5432/vectors
REDIS_URI=redis://localhost:6379

6. First Run Verification

Once installed and configured, verify that @manasdb/core can connect to your databases and that the necessary indices and extensions are active. The SDK includes a built-in health check CLI utility:

npx manas health

7. Troubleshooting Installation Issues

Configuration

1. Configuration Overview

ManasDB Core provides a highly flexible configuration object that controls which databases are used, how embeddings are generated, and whether advanced features like Tier-1 caching and Tree Reasoning are active.

import { ManasDB } from "@manasdb/core";

const memory = new ManasDB({
  // 1. Database Routing
  uri: process.env.DATABASE_URI, // Auto-detects MongoDB or Postgres
  dbName: "my_database",         // Required for MongoDB 

  // 2. Project Isolation
  projectName: "customer_support", // Namespaces your vectors

  // 3. Embedding Model Selection
  modelConfig: { source: "transformers" },

  // 4. Advanced Features (Optional)
  cache: { provider: "redis", uri: process.env.REDIS_URI },
  reasoning: { enabled: true },
  piiShield: { enabled: true },
  telemetry: true,
  debug: false
});

await memory.init();

2. Quick Config Templates

Here are the most common copy-paste ready configurations.

Standard MongoDB Setup (Free Local Embeddings)

const memory = new ManasDB({
  uri: process.env.MONGODB_URI,
  dbName: "ai_memory",
  projectName: "default_project",
  modelConfig: { source: "transformers" }
});

PostgreSQL Setup with OpenAI

const memory = new ManasDB({
  uri: process.env.POSTGRES_URI,
  projectName: "default_project",
  modelConfig: { 
    source: "openai", 
    model: "text-embedding-3-small" 
  }
});

High-Performance Polyglot Mode

const memory = new ManasDB({
  databases: [
    { type: "mongodb", uri: process.env.MONGODB_URI, dbName: "ai_memory" },
    { type: "postgres", splitType: "pg", uri: process.env.POSTGRES_URI },
  ],
  cache: { provider: "redis", uri: process.env.REDIS_URI },
  projectName: "enterprise_rag",
  modelConfig: { source: "transformers" },
  reasoning: { enabled: true }
});

3. Environment Variables Reference

4. Security & Strict Mode Protection

To prevent silent failures in production, ManasDB Core implements a Strict Mode.

• If you accidentally initialize @manasdb/core without providing a valid uri or databases array, await memory.init() will securely boot, but it will issue a console warning.
• If your application then attempts to call absorb() or recall() without a running database, ManasDB Core will Fail Fast and throw a descriptive error (such as MANASDB_ERROR: Cannot absorb(). No valid database providers...) rather than silently hanging or corrupting memory state.

5. Embedding Model Configuration

Controlled via the modelConfig object.

Using an Air-Gapped Custom Driver

If your enterprise uses a proprietary or internal embedding model API, you can seamlessly integrate it:

class InternalDriver {
  getModelKey() { return "corp-embed-v2"; }
  async embed(text) {
    const vector = await fetchInternalAPI(text);
    return { vector, dims: vector.length, model: this.getModelKey() };
  }
}

const memory = new ManasDB({
  uri: process.env.POSTGRES_URI,
  modelConfig: { source: "custom", driver: new InternalDriver() }
});

6. Caching Configuration

To enable massive 29x speedups on repeated complex queries, enable Tier-1 Redis caching.

cache: {
  provider: "redis",
  uri: process.env.REDIS_URI,
  semanticThreshold: 0.95, // Cosine similarity required to trigger a cache hit (default: 0.95)
  ttl: 3600 // Expiration time in seconds (default: 1 hour)
}

7. Advanced Configuration Options

• reasoning.enabled: Set to true to build Hierarchical Tree Index layouts. This is required if you want to use the powerful reasoningRecall() method later.
• piiShield.enabled: Set to true to scrub generic PII (Emails, Phones, SSNs) before data hits the DB.
• piiShield.customRules: Pass an array of RegExp objects (e.g. [/MY_REGEX/g]) to redact custom corporate patterns.
• debug: Defaults to false. Prints granular routing, chunking, and memory profiling logs to the console.

Core Concepts

1. Architecture Overview

ManasDB Core sits securely inside your Node.js application, bridging the gap between raw unstructured data and your underlying PostgreSQL/MongoDB databases.

ABSORB PIPELINE
───────────────
Raw Text
  │
  ├─ PII Shield (optional auto-redaction)
  │
  ├─ Token-Aware Chunker
  │     ├─ Section-boundary detection (### headers)
  │     └─ Token-budget sliding window (default 100)
  │
  ├─ Chunk Embedding + Deduplication (SHA256)
  /* ├─ Quantization guardrail (Planned) */
  │
  └─ Provider Broadcast → Inserts into _manas_documents & _manas_vectors

2. Vector Embeddings Explained

Embeddings are arrays of floating-point numbers that represent the underlying "meaning" or "semantics" of a piece of text. For example, "The weather is cold" and "It is freezing outside" will output vectors that exist very close together in space, even though they share no exact keyword matches.

3. Chunking Strategy

Large documents cannot be embedded as a single vector without diluting the semantic meaning. ManasDB Core uses a Token-Aware Sliding Window Chunker.

• Max Tokens: Breaks text into chunks of roughly 100 tokens (approx. 2-3 sentences).
• Overlap Tokens: Retains 20 tokens from the previous chunk to maintain context continuity.
• Structural Awareness: Respects Markdown headers and double-newlines to avoid breaking sentences mid-thought.

4. Semantic Similarity

Retrieval relies on calculating the Cosine Similarity between the user's Query Vector and the Stored Vectors. A score of `1.0` means an identical match, while `0.0` signifies entirely orthogonal concepts.

5. Hybrid Retrieval

Relying purely on Dense Vectors can cause failures when users search for exact product IDs, acronyms, or names. ManasDB Core automatically balances Dense Similarity (Vector Search) with Sparse Similarity (BM25 or Text Search) via Reciprocal Rank Fusion (RRF).

6. Deduplication Process

ManasDB Core protects your database from ballooning in size by generating a cryptographic SHA-256 hash of the content before embedding it. If you attempt to `absorb()` an identical document, the engine ensures zero duplicate vectors are generated.

You can also manually trigger semantic deduplication to clean up "near-duplicates" using the dedup() method:

// Remove memories that are 95% similar to each other
const report = await memory.dedup({ minSimilarity: 0.95 });
console.log(`Removed ${report.purgedCount} duplicates.`);

7. Tree Reasoning Architecture

When reasoning: { enabled: true } is active, ManasDB Core maps standard flat text into a Document → Section → Leaf hierarchy. During retrieval, reasoningRecall() ranks the highest-level Section first, and then retrieves all of its contributing Leaf nodes for deep context extraction without noise.

8. Maximal Marginal Relevance (MMR)

Standard similarity search can return results that are redundant (different chunks saying the same thing). ManasDB Core implements MMR to re-rank results, promoting diversity by penalizing candidates that are semantically too similar to results already selected for the top list.

• Lambda (λ): A balancing factor. Set to 1.0 for pure relevance, or lower (e.g., 0.5) to introduce variety.

API Reference & Usage

1. API Methods Overview

2. absorb(text, options?)

Transforms raw unstructured text into structured indexed search data.

// 1. Pre-flight Estimate
const cost = memory.estimateAbsorbCost(largeText);

// 2. Ingest with Budget Safety
await memory.absorb(largeText, {
  projectName: "project_x",
  metadata: { source: "manual" }
});

3. recall(query, options)

Retrieves the most semantically relevant memories. Supports MMR for diversification and Governance Tracing.

const results = await memory.recall("How do I use budget caps?", {
  limit: 5,
  lambda: 0.5, // Diversify results (MMR)
  mode: 'qa'   // Enable _trace output
});

console.log(results._trace.costUSD); // Track cost of this specific query

4. reasoningRecall(query, options?)

A hierarchical retrieval strategy returning parent sections instead of scattered raw chunks. (Requires reasoning: { enabled: true }).

const result = await memory.reasoningRecall("Summarize Q3 Goals", {
  topSections: 5,   // Analyze the best 5 document sections
  topSection: 0     // Return the absolute best one
});
// Returns: { section: "Q3 Strategy Board Meeting", score: 0.94, leaves: [...] }

5. list(limit?)

Retrives a list of documents stored in the system, sorted by their creation date.

const docs = await memory.list(10);
console.log(docs.map(d => d.text.substring(0, 50)));

6. forget(id) & forgetMany(query)

Compliance erasure tools to remove documents by specific ID or bulk metadata tags.

// 1. Precise deletion
await memory.forget("doc_123");

// 2. Bulk GDPR deletion
const report = await memory.forgetMany({ userId: "user_99" });
/* Returns: { 
    deletedTotal: 8, 
    providers: [{ provider: 'mongo', deleted: 3 }, ...], 
    timestamp: "2026-03-17T..." 
} */

7. Maintenance: health(), close(), & clearAll()

Methods for monitoring and managing the lifecycle of your @manasdb/core instance.

// Check if databases are connected
const status = await memory.health();

// Cleanup and Wipe (Dangerous!)
await memory.clearAll();

// Shutdown connections
await memory.close();

8. Common Patterns & Best Practices

• Fire & Forget: If latency is critical, you can run absorb() asynchronously without `await` blockages.
• Batch Caching: Group long-form QA context and apply absorb() with Promise.all() to chunk parallel workloads fast.

9. Error Handling

The library attempts to never throw unhandled promise rejections. Invalid URIs or API timeouts from external inference platforms return structured error logs instead of dying ({ error: true, code: "EMBEDDING_FAILED" }) allowing graceful fallback strategies inside your AI.

Embedding Models

1. Model Comparison Table

2. Decision Tree: Which Model?

Follow these rules when provisioning your RAG pipeline:

• Is your data legally regulated (HIPAA, SOC2, GDPR)? Do not use OpenAI/Gemini. Use `transformers` or `ollama`.
• Are you deploying on Vercel Edge / Serverless? Native `transformers` might exceed bundle limits. Use `openai` or self-hosted `ollama`.
• Are you aiming for > 1,000,000 vectors? Consider dimension costs. OpenAI's 1536d vectors take 4x more RAM on Postgres than MiniLM's 384d.

3. Per-Model Setup Guides

Transformers (Native Local)

ManasDB Core uses `@xenova/transformers` internally. Models download once securely on the first run and cache inside your `node_modules`. No setup required beyond providing source: "transformers".

Ollama

Ensure Ollama is running (`ollama serve`) and you have pulled the model (`ollama pull nomic-embed-text`).

new ManasDB({
  modelConfig: { source: "ollama", model: "nomic-embed-text" }
})

OpenAI & Gemini

Provide the respective API keys. ManasDB automatically batches text arrays into the maximum allowed payload limits for each provider to prevent rate-limiting.

// Set process.env.OPENAI_API_KEY
new ManasDB({
  modelConfig: { source: "openai", model: "text-embedding-3-small" }
})

4. Custom Embedding Drivers

You are not locked into our providers. Provide any class that implements an embed(textArray) method returning arrays of floats.

class CohereDriver {
  getModelKey() { return "cohere-v3"; }
  async embed(text) {
    // ... custom HTTP fetch logic here ...
    return { vector: rawResponse, dims: 1024, model: this.getModelKey() };
  }
}

5. Model Switching & Pruning Guide

Warning: You cannot mix vectors from different models in the same space. If you start with `transformers` (384d) and switch to `openai` (1536d) a week later, similarity searches will panic.

ManasDB Core maintains separate schema indices for each model gracefully.

1. If you change the `source`, ManasDB Core will automatically create a new index space.
2. Your old data will still exist in the database, but it will be ignored by the new model's search.
3. To clean up dead index tables, run the CLI: npx manas index-prune which will interactively ask if you want to drop the old vectors.

Storage Providers

1. Database Comparison

2. MongoDB Atlas Guide

ManasDB Core leverages the native Atlas Vector Search integration, utilizing powerful HNSW indices.

• Free Tier: The M0 sandbox cluster supports `$vectorSearch` natively. Meaning you can test @manasdb/core in the cloud for absolutely zero cost.
• Index Creation: Previously, Mongo required manually clicking through their UI to build the Search Index JSON. ManasDB Core automates this. It watches for new collections and automatically provisions the HNSW Search Index mapping via Atlas Admin API bindings during `absorb()`.

3. PostgreSQL Guide

@manasdb/core binds heavily to the pgvector extension.

• Setup: You must connect to your database as a superuser once and execute CREATE EXTENSION IF NOT EXISTS vector;. After that, ManasDB Core handles all the schema provisioning.
• Types: Vectors are stored natively as the vector(dims) data type. Text metadata uses JSONB constraints for flexible querying.
• Indices: ManasDB Core builds hybrid HNSW indexes with `m=16` and `ef_construction=64`. Once your dataset eclipses 5,000 records, the engine calculates the optimal query parameters dynamically to avoid massive brute-force sequential scans.

4. Backup & Restore Strategies

Since ManasDB Core embeds its knowledge into standard SQL/NoSQL tables, it completely inherits your existing disaster recovery protocols.

You do not need separate backup tools.

• If you dump your PostgreSQL database using `pg_dump`, your embeddings go with it.
• If you rely on MongoDB Atlas point-in-time recovery to rewind your database to yesterday at 4 PM, your AI context correctly rewinds with it without breaking links to user profiles.

Polyglot Mode

1. What is Polyglot Mode?

Polyglot Mode is a powerful ManasDB Core architecture that allows you to simultaneously connect to, write to, and read from multiple dissimilar database engines at the exact same time (e.g. MongoDB Atlas + PostgreSQL).

ABSORB() → [ManasDB Core Engine] 
                │
                ├─→ MongoDB Atlas ($vectorSearch)
                └─→ PostgreSQL (pgvector)

2. Use Cases for Polyglot

• Zero-Downtime Migrations: Syncing data from Mongo to Postgres without bringing down the production application.
• Redundancy & Failover: If the primary cloud vector database goes offline, ManasDB Core instantly falls back to the secondary database.
• Separation of Concerns: Keeping high-throughput chat RAG in managed MongoDB Atlas while shipping compliance/archival RAG vectors to an internal air-gapped Postgres instance.

3. Setup Guide

Pass an array of database configurations into the databases property.

const memory = new ManasDB({
  databases: [
    { type: "mongodb", uri: process.env.MONGODB_URI, dbName: "ai_memory" },
    { type: "postgres", uri: process.env.POSTGRES_URI }
  ],
  projectName: "hybrid_app"
});

4. Data Synchronization

When Polyglot is enabled, every call to absorb() fires parallel, non-blocking asynchronous inserts to all connected databases. If one database fails to ingest, ManasDB Core gracefully queues the operation, ensuring eventual consistency. Read operations (e.g., recall()) automatically query the fastest/healthiest provider.

5. Failover & Recovery

ManasDB Core includes a built-in circuit breaker. If MongoDB Atlas experiences a network partition, ManasDB Core will seamlessly reroute all recall() queries to your PostgreSQL replica. Once MongoDB reconnects, the engine will heal the missing vectors.

6. Score Merging across Providers

Because `$vectorSearch` (HNSW) and `pgvector` (`<=>` cosine operators) calculate math slightly differently at the hardware level, ManasDB Core unifies all scores in the application layer. When querying across both, it normalizes the cosine outputs so that a `0.95` from Mongo means the exact same real-world relevance as a `0.95` from Postgres.

7. Cost & Performance Implications

• Cost: Storage costs double (since data lives in two places). However, embedding API costs (e.g. OpenAI) remain exactly the same, because ManasDB Core calculates the expensive vector first, and then broadcasts the payload.
• Performance: Because connections are broadcast via Promise.all(), write latency is capped at the speed of your slowest provider. However, read latency drops, because ManasDB Core can race the identical queries against both providers and resolve immediately with whichever database responds first.

8. Global Compliance Deletion (ForgetMany)

ManasDB Core ensures that "The Right to be Forgotten" is respected across your entire database cluster. When you call forgetMany({ userId: "..." }), the engine broadcasts the deletion request to every connected database driver in parallel.

• Atomic Cleanup: Deletes the primary document, its constituent chunks, and the high-dimensional vectors across all platforms.
• Audit Trail: Returns a unified JSON report detailing how many items were scrubbed from Mongo vs Postgres, with a cryptographic-safe ISO timestamp for your compliance logs.

Caching & Performance

1. Caching Architecture

ManasDB Core features a dual-layer caching system designed to drop query latency from ~400ms down to < 5ms for repeated similar questions.

QUERY → [Tier 2: Node.js LRU Memory] → (Hit? Return instantly)
                 ↓ Miss
        [Tier 1: Redis Semantic Cache] → (Hit? Return in <10ms)
                 ↓ Miss
        [Vector Database Search] → Calculate, Rerank, Store in Cache, Return

2. Tier 1 vs Tier 2 Cache

• Tier 2 (LRU Cache): Runs directly on your Node.js server RAM. It holds extremely hot, recent semantic queries. Zero network latency.
• Tier 1 (Redis Cache): A shared networked cache state across all your Node.js instances/pods. If User A asks a complex question, User B receives the cached answer instantly from Redis without hitting the expensive vector DB.

3. Redis Setup & Configuration

You can use any standard Redis server (Upstash, ElastiCache, or Docker \`redis:7-alpine\`).

cache: {
  provider: "redis",
  uri: process.env.REDIS_URI, // e.g. redis://user:pass@host:port
  semanticThreshold: 0.96,    // Only return cached answers if the new query is 96% similar
  ttl: 86400                  // Cache valid for 24 hours
}

4. Optimizing Cache Hit Rates

Tuning the \`semanticThreshold\` is the key to ROI:

• Threshold = 0.98: Strict mode. Only almost identical typos ("what is @manasdb/core" vs "whats @manasdb/core") will hit the cache.
• Threshold = 0.90: Loose mode. Conceptually similar but unique queries might hit the cache. Saves massive DB costs, but risks returning slightly off-topic answers for nuanced differences.
• Default = 0.95: The sweet spot for general Q&A bots.

5. Common Bottlenecks & Tuning

If your AI is responding slowly, check the following:

Lazy-Loading & Dynamic Providers

1. Agnostic Dependency Architecture

ManasDB Core is designed to be lean. Most RAG libraries force you to install pg, mongodb, and openai at the same time, even if you only use one. ManasDB Core uses a **Provider Factory** that only imports dependencies at runtime when they are actually called.

2. The dynamic import() Flow

When you call await memory.init(), the engine analyzes your configuration and executes an asynchronous dynamic import for only the required drivers:

// If config has MONGODB_URI:
const MongoProvider = (await import('./providers/mongodb.js')).default;
this.drivers.push(new MongoProvider(...));

3. Verifying Dependencies

Because dependencies are optional, you must ensure you've installed the specific driver for your database:

• PostgreSQL: npm install pg
• MongoDB: npm install mongodb
• Redis: npm install ioredis

If you attempt to initialize a provider without its package, ManasDB Core will catch the ERR_MODULE_NOT_FOUND and provide a helpful console instruction on exactly which command to run.

4. Singleton Driver Management

The Factory Registry ensures that even if you have multiple `absorb()` calls happening simultaneously, only a single instance of each database driver is initialized per process, protecting your connection pool and memory overhead.

Tier-1 Redis Semantic Caching

1. Why Semantic Caching?

Vector database searches are mathematically expensive and DB-intensive. However, users often ask semantically identical questions (e.g., "What is the price?" vs "How much does it cost?"). By using a semantic cache, we can bypass the database entirely for redundant queries.

2. The Dual-Tier Architecture

ManasDB Core implements a deterministic two-stage lookup:

1. Tier 2 (Memory): An in-process LRU cache for 1ms response times on the physical server.
2. Tier 1 (Redis): A distributed semantic cache shared across your entire fleet of Node.js pods.

3. Performance Impact

4. Intelligent Short-Query Bypass

During benchmarking, we discovered that extremely short queries (under 3 words) are faster to resolve in the database than fetching from a networked Redis store. ManasDB Core automatically detects these "tiny queries" and bypasses Redis to prevent latency regression from TCP hops.

Compliance & Diversification

1. Global Compliance Deletion (ForgetMany)

In production RAG systems, respecting "The Right to be Forgotten" is non-negotiable. ManasDB Core provides a unified API to erase all traces of a user or session across all active database providers.

// Irreversibly delete all data for a specific user
const report = await memory.forgetMany({ userId: "user_123" });

console.log(report);
/*
{
  query: { userId: "user_123" },
  deletedTotal: 15,
  providers: [
    { provider: "mongo", deleted: 5 },
    { provider: "postgres", deleted: 10 }
  ],
  timestamp: "2026-03-17T..."
}
*/

2. Automated Lifecycle Expiration

Many compliance frameworks require that data is only retained for a specific period. You can use expireOlderThan() to automate the cleanup of old memories.

// Delete all memories older than 30 days
const result = await memory.expireOlderThan("30d");

// Delete memories older than a specific date
const result2 = await memory.expireOlderThan(new Date('2024-01-01'));

3. The "Forget" Design Philosophy

While standard databases use `delete`, ManasDB Core uses forget() as its primary verb to align with AI memory lifecycles. forget(id) is an alias for delete(id) that cascades through primary document storage, chunked text segments, and high-dimensional vector embeddings simultaneously.

4. Maximal Marginal Relevance (MMR)

Recall can often suffer from "Semantic Redundancy," where all top results are nearly identical. ManasDB Core implements MMR to provide a more diverse set of results.

By adjusting the lambda parameter in recall(), you can tune the balance between accuracy and variety:

• Lambda = 1.0 (Default): Pure relevance. The highest scoring vectors are returned regardless of similarity to each other.
• Lambda = 0.5: Balanced. System surfaces relevant results but actively penalizes items that are semantically redundant.
• Lambda = 0.1: Maximum Diversity. Surfaces a wide array of conceptual matches, useful for brainstorming or broad summarization.

5. Efficiency & Optimization

Calculating MMR requires comparing candidate vectors against each other. To avoid performance degradation, ManasDB Core uses Conditional Vector Projection:

1. If lambda = 1.0, vectors are not fetched from the database, saving massive network bandwidth.
2. If lambda < 1.0, the engine performs a "Broad Retrieval" (fetching limit * 2 candidates), projects their vectors, and runs the MMR re-ranking algorithm in-memory.

This ensures that you only pay the "vector bandwidth tax" when you explicitly request diversified results.

Production & Scaling

1. Scaling Considerations (Decision Tree)

Before launching your AI feature to production, analyze your expected load:

• < 50,000 documents: Default configurations are fine. Deploy on a single Node.js instance with a managed Postgres/Mongo backend.
• 50K - 1M documents: Enable Tier 1 Redis Caching. Ensure your database is hosted in the same cloud region as your compute nodes to minimize inference latency.
• > 1M documents: Consider Polyglot Mode to split ingestion load across two databases.

2. Performance Benchmarks

Here are internal load-tested benchmarks running @manasdb/core against a dedicated AWS RDS `db.m6g.xlarge` instance (4 vCPU, 16GB RAM):

3. Database Sizing Guide

To calculate your PostgreSQL/Mongo RAM requirements, use this formula:

Required RAM = (Number of Vectors) * (Dimensions) * 4 bytes / 1,000,000,000

• For 1 Million vectors using OpenAI (1536d): 1M * 1536 * 4 = ~6.1 GB RAM. You will need at least a 16GB RAM database instance to ensure indices stay completely resident in memory without swapping to disk.
• For 1 Million vectors using Transformers (384d): 1M * 384 * 4 = ~1.5 GB RAM. You can use a significantly cheaper $15/mo database.

4. Kubernetes Deployment

ManasDB Core is completely stateless. It stores absolutely nothing in the local filesystem. You can scale your Node.js pods horizontally (HPA) to infinity behind a Load Balancer.

Warning: Ensure you configure a REDIS_URI so all Kubernetes pods share the same Tier 1 memory cache.

5. Governance & Budgeting

Never wake up to a surprise API bill. ManasDB Core allows you to set hard monthly caps that are enforced pre-flight before any embedding or retrieval tokens are consumed.

const memory = new ManasDB({
  uri: process.env.MONGODB_URI,
  retry: {
    budget: { 
      monthlyLimit: 50.00 // $50 USD/month cap
    }
  }
});

• Pre-Flight Blocking: If a request would push you over the limit, it is rejected immediately.
• Cost Tracking: Every recall result includes a _trace with the actual USD cost of that specific query.
• Estimation API: Use estimateAbsorbCost(text) to predict costs for massive backfills before committing.

6. Programmatic Observability

ManasDB Core is built for teams that monitor their AI. Use onTrace to pipe internal retrieval decisions to your own dashboard (Datadog, Grafana, etc.).

memory.onTrace((trace) => {
  console.log(`[Observability] Query: ${trace.tokens} tokens | Cost: $${trace.costUSD}`);
});

Security & Privacy

1. Security Overview Checklist

ManasDB Core is built to operate in highly regulated environments (FinTech, Healthcare, Enterprise SaaS). Before going live:

• [ ] Ensure your Node.js environment connects to your DB using TLS/SSL string queries (e.g. ?sslmode=require).
• [ ] If using external embedding APIs, inject keys via secure secrets managers, not hardcoded `.env` files.
• [ ] Enable the built-in PII Shield if you are ingesting raw customer support tickets.

2. Encryption at Rest & in Transit

ManasDB Core relies entirely on the encryption logic of your chosen storage provider. Because @manasdb/core sits on top of MongoDB Atlas and PostgreSQL, it automatically inherits their SOC2 and HIPAA compliant AES-256 transparent data encryption at rest.

3. PII Shield Deep-Dive

ManasDB Core includes a high-speed, deterministic middleware scanner that scrubs Personally Identifiable Information before it is sent to OpenAI or embedded into your database.

const memory = new ManasDB({
  uri: process.env.DATABASE_URI,
  piiShield: { 
    enabled: true, 
    customRules: [
      /\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b/g, // Credit Cards
      /\b(?i)confidential\b/g // Corporate keywords
    ] 
  }
});

4. GDPR & Compliance Guide

To comply with GDPR "Right to be Forgotten" requests, you must trace vector identities back to the user. When a deletion request arrives, you can wipe them completely from the RAG pipeline with a single call:

const report = await memory.forgetMany({ userId: "user_123" });
/* 
Returns: { 
  deletedTotal: 12, 
  timestamp: "2026-03-17...", 
  providers: [...] 
} 
*/

Audit Trail: The resulting JSON report serves as critical proof of erasure for compliance officers, documenting exactly how many records were scrubbed across all polyglot providers.

5. Multi-Tenancy & ProjectRegistry

For SaaS applications, ManasDB Core provides two layers of isolation. Use projectName for strict database-level namespaces, or ProjectRegistry for managing hundreds of tenants lazily.

import { ProjectRegistry } from "@manasdb/core";

const registry = new ProjectRegistry();

// Lazily gets or creates an isolated ManasDB instance
const clientMemory = await registry.get("client_123", { 
  uri: process.env.DATABASE_URI 
});

6. Model Dimension Lock

One of the most common "silent failures" in vector databases is querying a collection with the wrong embedding model (e.g. querying a 1536-dim collection with a 384-dim model). ManasDB Core prevents this by locking the project to its initial dimensions.

• On init(), ManasDB Core verifies the model dimensions against the stored manifest.
• Mismatches throw a hard error, preventing corrupted data ingestion.
• To override (e.g. during a migration), set allowModelChange: true.

6. Data Residency

Unlike third-party vector databases (Pinecone, Weaviate Cloud), ManasDB Core operates 100% locally. If your enterprise requires that German healthcare data never leaves the EU, use @manasdb/core with Local Transformers or self-hosted EU Postgres. Zero bits will cross the ocean.

7. Credential Management

ManasDB connects to external API providers via the SDK. Do not pass credentials to ManasDB during the client build step. Always initialize ManasDB on the Server-Side (Node.js/Next.js Route Handlers/Server Actions) bridging process.env securely.

Failure Modes & Recovery

Building production-grade AI applications requires understanding how your system behaves when infrastructure fails. ManasDB Core is architected for "Graceful Degradation."

1. Storage Provider Outages

If your primary database goes offline, ManasDB Core behavior depends on your configuration:

• Single DB: absorb() and recall() will throw a hard error. We recommend wrapping these in standard try/catch blocks.
• Polyglot Mode: ManasDB Core utilizes a "Best Effort" broadcast. If one database is down but another is up, the operation succeeds. This prioritizes availability over immediate consistency.
• Retry Logic: As of v0.4.1, you can configure retry: { attempts: 3, backoff: 1000 } in the constructor to handle transient network blips automatically.

2. Embedding API Failures

Because semantic search requires vectors, an outage in OpenAI, Gemini, or Ollama is a critical failure.

3. Redis Cache Outages

Redis is treated as a non-critical optimization.

• If Redis goes down, ManasDB Core silently bypasses the cache and routes all traffic to the underlying databases.
• Your application will remain functional, though you will observe a temporary increase in search latency.
• The system self-heals: once Redis is back online, ManasDB Core resumes filling the cache automatically.

4. Telemetry Resilience

Telemetry writes are "fire-and-forget." If the telemetry table is full or the connection is interrupted, ManasDB Core catches the error internally, ensuring that data loss in your audit logs never crashes your main application logic.

5. Monitoring & Alert Systems

For high-availability clusters, we recommend alerting on these specific signals:

• MANASDB_INSERT_FAILURE: Triggered when all providers fail.
• Budget Exceeded: Triggered when a request violates the monthly USD cap.
• Partial Failures: Logged as warnings when at least one provider is down but others are functional.
• Memory Limits: Watch for ManasDBWarning if utilizing the zero-config MemoryProvider in production environments.