---
sidebar_position: 4
title: "Data plane"
slug: /architecture/data-plane
question: Where does each kind of ABI data live?

---

# Data plane

ABI keeps each kind of data behind an engine service, and each service writes to the backend its adapter names in `config.yaml`. The same project keeps its data in local files and embedded stores on a laptop and in shared servers under Docker Compose, with no code change: only the adapters in `config.yaml` differ. Every kind of data has an open-source backend, so you can run the whole plane on infrastructure you own.

Each table below lists the backends the engine ships for that service and what each `abi new project` preset runs: `config.yaml` for `abi dev`, `config.local.yaml` for Docker Compose, `config.remote.yaml` for a remote deploy. "Engine default" means the template leaves the service out and the engine's default applies. Beyond the shipped backends, a service can take your own class (`custom`) or another engine's service (`nats_rpc`, with exceptions); [Services and adapters](/architecture/services-and-adapters#another-engines-service-nats_rpc) says which services accept which, with each adapter's settings and dependencies.

---

## Any data

### Graph

The **[triple store](/architecture/services-and-adapters#triple-store)** holds ontologies and the knowledge graph as RDF triples and answers SPARQL. Everything agents know about your domain is modelled there as typed entities and relationships. Every insert or delete is also broadcast on the bus, so modules can react to graph changes.

| Backend | `adapter:` | abi dev | Docker Compose | Remote |
|---|---|---|---|---|
| Oxigraph server (HTTP) | `oxigraph` | yes | — | — |
| Apache Jena Fuseki (TDB2) | `apache_jena_tdb2` | — | yes | yes |
| Amazon Neptune through an SSH bastion | `aws_neptune_sshtunnel` | — | — | — |
| Amazon Neptune | `aws_neptune` | — | — | — |
| RDF files on local disk | `fs` | — | — | — |
| Oxigraph embedded in the engine process | `oxigraph_embedded` | — | — | — |
| RDF files in an object storage service | `object_storage` | — | — | — |

### Vectors

The **[vector store](/architecture/services-and-adapters#vector-store)** keeps embeddings in collections for similarity search and agent memory retrieval. Pipelines that process unstructured content (documents, emails, transcripts) embed it here and keep its provenance in the graph.

| Backend | `adapter:` | abi dev | Docker Compose | Remote |
|---|---|---|---|---|
| Qdrant server | `qdrant` | — | yes | yes |
| Qdrant embedded in the engine process | `qdrant_in_memory` | — | — | — |
| SQLite with the sqlite-vec extension | `sqlite_vec` | yes | — | — |

### Documents and events

The **[document service](/architecture/services-and-adapters#document)** stores documents in namespace-bound collections, one document per atomic write.

| Backend | `adapter:` | abi dev | Docker Compose | Remote |
|---|---|---|---|---|
| SQLite | `sqlite` | yes | — | — |
| PostgreSQL | `postgresql` | — | yes | yes |

The **[event service](/architecture/services-and-adapters#event)** keeps the durable event log and broadcasts each published event on the bus. With PostgreSQL in NATS mode, an hourly job moves events older than `archive_after_days` to the dataset service.

| Backend | `adapter:` | abi dev | Docker Compose | Remote |
|---|---|---|---|---|
| SQLite | `sqlite` | engine default | engine default | engine default |
| PostgreSQL | `postgresql` | — | — | — |

The **[activity log](/architecture/services-and-adapters#activity-log)** records who did what. A failed record never breaks the call that produced it. Its `document` adapter writes through the document service, so it lives wherever the documents do.

| Backend | `adapter:` | abi dev | Docker Compose | Remote |
|---|---|---|---|---|
| SQLite | `sqlite` | without NATS, engine default | without NATS, engine default | without NATS, engine default |
| The engine's Document service | `document` | engine default | engine default | engine default |

The Nexus app keeps its own tables (users, workspaces, conversations) in PostgreSQL through SQLAlchemy, outside the engine services.

### Datasets

The **[dataset service](/architecture/services-and-adapters#dataset)** holds named tables with a partition spec, queried with SQL. DuckLake keeps its catalog in SQLite or PostgreSQL and its data files under `storage/datasets/` by default.

| Backend | `adapter:` | abi dev | Docker Compose | Remote |
|---|---|---|---|---|
| DuckLake (DuckDB) with a SQLite or PostgreSQL catalog | `ducklake` | yes | yes | engine default |

### Objects

**[Object storage](/architecture/services-and-adapters#object-storage)** holds files and binary objects under prefixes: raw files from integrations (PDFs, CSVs, images, audio) before pipelines process them, and artifacts afterwards. Every write is also recorded as an event.

| Backend | `adapter:` | abi dev | Docker Compose | Remote |
|---|---|---|---|---|
| Local disk | `fs` | yes | — | — |
| Amazon S3 or an S3-compatible store (MinIO) | `s3` | — | yes | yes |
| Naas workspace storage | `naas` | — | — | — |
| Cloudflare R2 | `r2` | — | — | — |

### Key-value and cache

The **[key-value service](/architecture/services-and-adapters#key-value)** stores keys and values with expiry: short-lived state that belongs in no durable store.

| Backend | `adapter:` | abi dev | Docker Compose | Remote |
|---|---|---|---|---|
| Redis | `redis` | — | yes | yes |
| In-process dictionary, optionally persisted to SQLite | `python` | yes | — | — |

The **[cache](/architecture/services-and-adapters#cache)** is tiered: a list of adapters, hot before cold. Unless a template sets it, a preset runs the engine's default tiers; the table shows what each one runs.

| Backend | `adapter:` | abi dev | Docker Compose | Remote |
|---|---|---|---|---|
| Local disk | `fs` | — | — | — |
| Redis | `redis` | hot tier, engine default | hot tier, engine default | hot tier, engine default |
| The engine's Object storage service | `object_storage` | cold tier, engine default | cold tier, engine default | cold tier, engine default |
| The engine's Key-value service | `keyvalue` | — | — | — |

### Messages

The **[bus](/architecture/services-and-adapters#bus)** publishes and delivers messages on topics between services and modules. In NATS mode (the default) it is not a choice: the engine runs it on JetStream at `nats.nats_url`, and any other bus adapter is refused. Projects created with `--without-nats` (legacy) name a bus adapter, as the "without NATS" cells show.

| Backend | `adapter:` | abi dev | Docker Compose | Remote |
|---|---|---|---|---|
| RabbitMQ (legacy, projects without NATS) | `rabbitmq` | — | without NATS | without NATS |
| In-process queue, optionally persisted to SQLite | `python_queue` | without NATS | — | — |
| NATS JetStream | `nats_jetstream` | NATS mode, no block | NATS mode, no block | NATS mode, no block |

Module jobs and module discovery also run on JetStream in NATS mode, but they talk to the broker directly: they are not messages on the `bus` service. See [Real-time fabric](/architecture/real-time-fabric).

Secrets (the [secret service](/architecture/services-and-adapters#secret)) and outgoing email (the [email service](/architecture/services-and-adapters#email)) are services too, with their own adapters.

---

## Any compute

ABI provides three compute modes for moving data through the system:

**Batch compute** runs as [module jobs](/architecture/remote-modules#module-jobs). A module declares its jobs with a schedule (`Cron`, `Every`); JetStream schedules and delivers the triggers, with no central scheduler. The engine hosts the jobs of its own modules, and a remote module hosts its own. Jobs run only in NATS mode. Integrations pull data from external APIs on schedules or on demand, and pipelines turn raw API responses into RDF triples or embeddings and write them through the services above. Pipeline code is plain Python with no proprietary DSL. (Dagster remains available as a legacy extra for projects without NATS.)

**Event-driven compute** runs on NATS: a job can run on an event (`OnEvent`, on an ABI event type or a NATS subject, with a filter on the event's fields). A new object lands in storage, its `ObjectPut` event triggers a pipeline, the pipeline updates the knowledge graph, an agent is notified.

**Interactive compute** runs inline within agents. When a user or system sends a request, the agent picks SPARQL queries, Python workflows and external integrations as tools and runs them within the conversation. Latency is bounded by the tool calls, not by a separate compute tier.

The three modes are composable: a scheduled pipeline can publish an event, that event can trigger a workflow, and that workflow can invoke an agent.

---

## Any model

Agents do not hardcode a provider. Models come from provider modules (`naas-abi-marketplace`: OpenAI, Anthropic, Gemini, Mistral, OpenRouter, Ollama and others), which register them in the engine's model registry. Switching providers is a configuration change.

**OpenRouter** is the default of a new project: one API key serves chat, the agents and embeddings, across many providers' models.

**Ollama** serves models locally for air-gapped or privacy-sensitive deployments. The `ai.ollama` module serves Qwen2.5 3B and `nomic-embed-text` with no key. No inference request leaves your network.

**Custom endpoints**: any service that implements the OpenAI-compatible chat completions API can back a model: fine-tuned models, on-premises inference servers, private endpoints.

The default chat and embedding models are set in `config.yaml` (`services.model_registry`), and agents with no explicit model use them.

---

## Anywhere

The full stack runs in four topologies without code changes. The engine reads `config.{ENV}.yaml` when the `ENV` variable names a file that exists, and `config.yaml` otherwise; the adapters in that file decide where the data goes.

| Topology | Infrastructure | Notes |
|---|---|---|
| Local development | `abi dev up` on your machine, or Docker Compose (`abi start`) | `abi dev up` reads `config.yaml`, which names Oxigraph (triple store), SQLite + sqlite-vec (vector store), SQLite (document), DuckLake (dataset), Filesystem (object storage), Python (key-value), dotenv (secret), and starts the Oxigraph server itself. Compose sets `ENV=local` and reads `config.local.yaml`, which names Apache Jena (triple store), Qdrant (vector store), PostgreSQL (document), DuckLake (dataset), S3 (object storage), Redis (key-value), dotenv (secret), Filesystem (email); the stores run in containers |
| On-premises | Your Docker hosts, or Dokploy (`abi deploy dokploy`) | The shared backends of your choice: Dokploy sets `ENV=dokploy` and reads the `config.dokploy.yaml` it writes |
| Air-gapped | Same as on-premises, Ollama for models | Zero external API calls |
| Managed | naas.ai-hosted infrastructure | MIT core, managed operations tier |

There are no proprietary runtimes, no ABI-specific storage formats, and no services that phone home. The data plane is yours.

---

→ [Interoperability](./interoperability)