---
sidebar_position: 2.1
title: "Real-time fabric"
slug: /architecture/real-time-fabric
slide: /diagrams/abi-realtime-fabric.svg
slide_alt: The serving engine, the NATS broker with JetStream, and remote modules. Kernel services answer on abi.svc subjects, discovery on abi.discovery subjects. Each service's adapter, chosen in config.yaml, reaches its backend.
question: How do engine services and remote modules communicate?

---

# Real-time fabric

The real-time communication fabric connects distributed agents and applications.
It unifies messaging, streaming, and state into one system: kernel services,
remote modules, and jobs share one connection. That is what Naas.AI (AI Networks
as a Service) is at the core, with modules and ontological grounding.

The engine uses [NATS](https://nats.io) for that fabric. Kernel services (object
storage, triple store, datasets, documents, secrets and the rest) are
request/reply endpoints on the broker. Scheduled work runs as
[module jobs](/architecture/remote-modules#module-jobs) on JetStream. Modules can
run in their own process and find each other through **discovery**. A module
needs one connection and nothing else: no extra port, hostname, or database
credential.

The fabric is not the [bus service](/architecture/services-and-adapters#bus). The
bus is one engine service among the others, with its own adapters; in NATS mode
the engine runs it on the broker's JetStream, but service calls, jobs and
discovery do not go through it. Behind each kernel service, the adapter
`config.yaml` names still reaches its own backend:
[Services and adapters](/architecture/services-and-adapters) lists them.

A project is in **NATS mode** when its config has a top-level `nats:` block.
`abi new project` writes one unless you pass `--without-nats`. Without it, the
engine keeps the in-process wiring of 2.x (see
[Migrate from 2.x to 3.0](/user-guide/upgrade/migrate-to-3)).

## The `nats:` block

```yaml
nats:
  # The broker, logged in as the user `abi` (the engine's user).
  nats_url: "nats://abi:{{ secret.NATS_ABI_PASSWORD }}@nats:4222"
  # Signs every service token. Use 32 bytes or more; never given to modules.
  jwt_secret: "{{ secret.NATS_JWT_SECRET }}"
  # The module registry, hosted by the serving engine.
  discovery:
    project: "my_project"
    lease_seconds: 180
  # The broker's HTTP monitoring port, read by the Nexus System app.
  monitoring_url: "http://nats:8222"
```

It sits at the top level, next to `api:`, not under `services:`. Only
`jwt_secret` is required; the other keys have working defaults:

| Key | Default | Meaning |
|---|---|---|
| `nats_url` | `nats://127.0.0.1:4222` | The broker. |
| `client_timeout_seconds` | 10 | How long the engine's own service calls wait for a reply. |
| `max_concurrent_requests` | 64 (1 to 4096) | Calls each kernel service, and discovery, handles at once. |
| `discovery` | none | The module registry. Without this block the engine hosts no discovery, and remote modules cannot register. |
| `discovery.project`, `discovery.lease_seconds` | `default`, 20 (1 to 300) | The discovery project modules register under, and a module instance's lease. `abi new project` writes 180, so modules outlive an engine restart. |
| `object_storage_streaming.*` | 64 KiB chunks, 32 sessions, 60 s idle | Chunked object transfers. |
| `models.streaming.*`, `models.generation_timeout_seconds` | 16 MiB per request, no deadline | Remote model calls. |
| `rpc_overflow.*` | 256 MiB per value, 1 GiB parked per process | [Payloads above the broker limit](/architecture/remote-apps#large-payloads). |
| `jobs.enabled`, `jobs.retention.*`, `jobs.interrupt_grace_seconds` | on, 7 days, 5 s | [Module jobs](/architecture/remote-modules#module-jobs) hosted by this engine. |
| `engine.role`, `engine.rollout_id`, `engine.lease_seconds`, `engine.standby_timeout_seconds`, `engine.drain_seconds` | `serve`, empty, 20, 900, 60 | [One serving engine](#one-serving-engine). |
| `monitoring_url` | none | The broker's monitoring endpoint (no auth: keep it private), for the [System app](/developers/architecture/observability). |

## Kernel services over NATS

Every kernel service but the bus has a versioned protobuf contract in
[`naas-abi-proto`](https://github.com/jupyter-naas/abi/tree/main/libs/naas-abi-proto): activity
log, cache, coding environment, dataset, document, email, event, key-value,
model registry, object storage, secret, source control, triple store and vector
store. The serving engine exposes each one as a NATS micro service on
`abi.svc.<service>.v1.<method>`, on a queue group.

In NATS mode the boundary holds **even inside one process**. Services and
modules loaded by the engine do not get each other's Python objects: they get
facades that send requests over NATS. So a service that runs in its own process
later needs no code change, and there is no local fallback: when the broker is
down, cross-service calls fail.

Calls follow the same rules everywhere:

- **Deadline.** A call waits 10 seconds by default (`client_timeout_seconds`,
  or the SDK's `timeout`), covering connect and request. Long dataset
  operations (`compact`, `flush`, `query`) take a deadline per call.
- **At most once.** A call is attempted once. A timeout or a lost connection
  may hide a write that completed: reconcile before you retry.
- **No responders.** When nobody serves an engine subject, the client sends
  again with a short backoff for up to 5 seconds, within the call's deadline.
  Nobody received the first request, so this is always safe; it hides the gap
  of an engine handover.
- **Errors** come back as typed codes (`RPCError.code` in the SDK). A service's
  error reply carries its code in the `Abi-Error-Code` header.
- **Authentication.** Each request carries a short-lived service token in its
  headers. See [Security model](/developers/architecture/security).

Large values do not fail at the broker's 8 MiB message limit (the `max_payload`
ABI's generated `nats.conf` sets; the NATS server's own default is 1 MB): object
and model transfers stream in chunks, other replies and requests overflow into
chunked sessions, and large reads stream. See
[Large payloads](/architecture/remote-apps#large-payloads).

### From another process: `nats_rpc`

A process that does not serve, such as a worker or a second engine, reaches the
serving engine's services by setting a service's adapter to `nats_rpc`, with its
own `nats_url`, `jwt_secret` and `service_identity`. It keeps no data for that
service. Every service accepts it except the bus (JetStream on the broker
already) and the model registry (the engine that loads the provider modules
serves the models). See
[Another engine's service](/architecture/services-and-adapters#another-engines-service-nats_rpc).

## JetStream

JetStream, the broker's persistence layer, holds what must outlive a process:

| Stream or bucket | What |
|---|---|
| Bus streams (`naas-abi-<hash>`) | The bus service (`services.bus`), always JetStream in NATS mode. |
| `ABI_JOBS_<project>` | Module job triggers and schedules (JetStream message schedules, NATS 2.14 or later). |
| `abi_claim_checks` (object store) | Published messages above the broker limit. |
| Key-value bucket `ABI_ENGINE` | The serving engine's lease. |
| Key-value bucket `ABI_DISCOVERY_<project>` | The module registry of each project. |

The engine that hosts discovery creates the jobs stream and the claim-check
store at start: modules may not create streams.

## One serving engine

Exactly one engine serves the kernel services per NATS account. It holds a
lease, the key `owner` in the key-value bucket `ABI_ENGINE`, renewed every
quarter of `nats.engine.lease_seconds` (20 s by default). A second engine that
tries to serve waits up to one lease period for the holder to go, then fails
with `EngineAlreadyServing`, naming the holder's host, pid, instance, version,
rollout and start time. This is what stops a second replica or a second
`abi dev up` from splitting requests between two copies of every service.

An engine has one of three roles (`nats.engine.role`, or `ABI_ENGINE_ROLE`):

- **`serve`** (default): takes the lease, serves the kernel subjects and hosts
  the jobs.
- **`client`**: never serves, opens no backend and hosts no jobs; every service
  is a NATS client of the serving engine. For scripts and notebooks next to a
  running stack.
- **`auto`**: serves when no engine does, and is a client otherwise; it never
  stands by. `abi chat`, `abi run` and `abi agent list` use it.

### Deploys without downtime

Set `ABI_ROLLOUT_ID` (for example the image tag) on each deploy. A new engine
whose rollout id differs from the holder's **stands by**: it loads, waits, and
takes the lease when the old engine releases it. A standby that waits longer
than `nats.engine.standby_timeout_seconds` (900 s) stops. The old engine stops
its jobs, releases the lease, stops taking new calls, and finishes the calls and
transfers it already holds for up to `nats.engine.drain_seconds` (60 s). For a
moment both serve, rather than neither.

A serving engine with a rollout id refuses to start when a service it owns keeps
its data on this host (SQLite, embedded Oxigraph, in-memory Qdrant, sqlite-vec,
`fs`): the next engine would answer from another copy. Each adapter's "Keeps its
data" column on [Services and adapters](/architecture/services-and-adapters#services)
gives the engine's answer. Development and single-host setups keep those
backends and deploy stop-then-start.

A holder that cannot renew for half a lease stops serving at once (fencing), and
serves again if a later renewal succeeds. A crashed engine is replaced after one
lease period. Under an orchestrator that starts the new container before it
stops the old one, give each deploy its own `ABI_ROLLOUT_ID` and keep the stop
grace period above `drain_seconds`.

### When the broker is gone

nats-py reconnects for about two minutes (60 attempts, 2 s apart). After that an
SDK module stops, and so does the API process when `api.reload` is `false`
(SIGTERM, then exit status 1 after 20 s), so a supervisor (Docker
`restart: unless-stopped`, for example) starts a fresh one. `api.reload`
defaults to `true` and the generated templates leave it there: the loss is then
only logged, because the reloader would not restart a worker that exits. Set
`api.reload: false` in production.

## Discovery

Discovery is the registry of the modules that run in their own process. It runs
in the serving engine, for the project named in `nats.discovery.project`, on
`abi.discovery.<project>.v1.*`. A module instance registers its
**descriptor**: its module id, version, contract major, the modules it depends
on, and the agents, jobs, models and apps it offers. The services it calls are
not in the descriptor: the module declares them in code, and the SDK refuses
calls to any other.

- An instance holds a **lease** and renews it about every 5 seconds. A clean
  stop unregisters at once; a crash is cleaned up when the lease expires.
- Its **status** is `STARTING` until its dependencies are ready and it has
  initialized, then `READY`. `DEGRADED` means a dependency went away; `STAGED`
  means a newer rollout waits for its whole cohort; `DRAINING` means it is
  stopping. Callers only use `READY` instances.
- The registry holds at most **256 live instances and 512 KiB** per project.
- **Rollouts.** Instances started with the same `ABI_ROLLOUT_ID` (and
  `ABI_ROLLOUT_MODULES`) cut over together: the previous generation stays
  ready until every module of the new one is up, then drains.
- A module registers **only as itself**: its token must assert its module id
  (see [Module identities](/developers/architecture/security#module-identities)).

Nexus reads discovery to list remote agents next to the engine's, and remote
apps on the Apps page. The System app's **Modules** tab shows every instance.

## The module lifecycle

Engine modules (the ones under `modules:` in the config) load inside the
engine, as in 2.x. In NATS mode the serving engine starts a job host for each
module that declares jobs, once the module is initialized (unless
`nats.jobs.enabled` is `false`).

Remote modules (built with `naas-abi-sdk`) run under `run_module` and follow
the same hooks:

![Remote module lifecycle: register STARTING, wait for dependencies, initialize, renew READY, then unregister on SIGTERM.](/diagrams/abi-nats-lifecycle.svg)

A module declares the services it calls (`ModuleDependencies(services=...)`)
and the modules it depends on (`modules=...`); undeclared calls are refused.
It binds its agents, models, apps and job handlers in `on_initialized`, before
it becomes `READY`. On `SIGTERM` it stops taking new work, finishes the agent
runs and jobs it started, and unregisters. Read on in
[Remote modules](/architecture/remote-modules).

→ [Remote modules](/architecture/remote-modules)