---
question: How do I write a pipeline class in a module?
kind: task
via_web: na
via_cli: available
via_python: available
via_api: na
via_config: na
ref_cli: abi new pipeline
ref_python: Pipeline
prereq: You need a module that declares `triple_store` and `document` in its `ModuleDependencies` (a module that `abi new module` created declares both). See [Creating a Module](/capabilities/build/modules/creating-a-module).
next:
  - /capabilities/build/pipelines/run-a-pipeline
  - /capabilities/build/pipelines/schedule-a-pipeline
---

# Create a pipeline

Pipelines transform raw data from integrations into OWL/RDF triples for the knowledge graph. They are the semantic ingestion layer: a pipeline's `async def run()` returns an `rdflib.Graph`. The class is `naas_abi_sdk.pipeline.Pipeline`, which needs the extras `naas-abi-sdk[workflows,rdf]`.

The module builds each pipeline class it finds under its `pipelines/` folder. It offers the class to its own agents as a tool (`as_tools()`), and publishes it as the durable workflow `pipeline.<slug>` that Nexus, scripts and other modules start by name.

## Role in the stack

```text
Integration (raw API data)  →  Pipeline (transforms to RDF)  →  Triple Store
```

A pipeline:
- Calls an integration to fetch raw data.
- Maps that data to ontology classes and properties.
- Builds an RDF graph and returns it, writing it to the triple store when it is meant to persist it.
- Offers itself to the module's agents as a tool through `as_tools()`, and to other modules and Nexus as the durable workflow `pipeline.<slug>`.

If your output is not RDF, write a [workflow](/capabilities/build/agents/creating-a-workflow) instead.

## Web UI

Not applicable. Nexus has no screen that writes a pipeline: it is a Python class in a module. Use the CLI or Python on this page. Once the module runs, System › Jobs starts `pipeline.<slug>`: see [Run a pipeline](/capabilities/build/pipelines/run-a-pipeline?via=web).

## CLI

### Scaffold the pipeline

From the project's root, name the module's `pipelines/` folder:

```bash
abi new pipeline Items src/catalog/pipelines
```

The arguments are `<name> [path]`. The name is converted to PascalCase and `Pipeline` is appended. Run from inside the module's package (`src/catalog/`), the path defaults to its `pipelines/` folder, the only place the module looks for pipelines: a path outside it gets a warning. The command is [`abi new pipeline`](/user-guide/reference/cli#abi-new-pipeline).

It writes two files:

- `ItemsPipeline.py`: a configuration dataclass, a parameters model and a `Pipeline` subclass whose `async def run` describes one individual as RDF. Replace `run` with your own.
- `ItemsPipeline_test.py`: a test that runs the pipeline and checks its name.

The command prints the pipeline's name (`items_pipeline`) and its durable name (`pipeline.items_pipeline`).

## Python

### Write the class

The class reads services through the module, which `New(module)` receives:

```python title="src/catalog/pipelines/ItemsPipeline.py"
from dataclasses import dataclass

from naas_abi_sdk.pipeline import Pipeline, PipelineConfiguration, PipelineParameters
from pydantic import Field
from rdflib import RDF, RDFS, Graph, Literal, Namespace, URIRef

MY = Namespace("http://ontology.naas.ai/catalog/")  # your module's ontology namespace
GRAPH = "http://ontology.naas.ai/graph/catalog"


@dataclass
class ItemsPipelineConfiguration(PipelineConfiguration):
    """Configuration for ItemsPipeline."""


class ItemsPipelineParameters(PipelineParameters):
    filter: str | None = Field(default=None, description="Optional filter for the items to ingest")


class ItemsPipeline(Pipeline[ItemsPipelineParameters]):
    """Ingest the catalog's items into the knowledge graph."""

    def __init__(self, configuration: ItemsPipelineConfiguration, module=None):
        super().__init__(configuration)
        self.module = module

    @classmethod
    def New(cls, module) -> "ItemsPipeline":
        return cls(ItemsPipelineConfiguration(), module)

    async def run(self, parameters: ItemsPipelineParameters) -> Graph:
        # 1. Fetch raw data (your code, usually an integration)
        items = await self.module.list_items(parameters.filter)

        # 2. Build an RDF graph
        g = Graph()
        for item in items:
            uri = URIRef(f"http://ontology.naas.ai/abi/item/{item['id']}")
            g.add((uri, RDF.type, MY.Item))  # a class of your ontology
            g.add((uri, RDFS.label, Literal(item["name"])))

        # 3. Store it, and return it
        await self.module.engine.services.triple_store.insert(g, GRAPH)
        return g
```

### Let the module build it

- **Discovery.** `BaseModule` finds the `Pipeline` subclasses defined under the package's `pipelines/` folder (test files, abstract classes and classes that are only imported aside), or the ones listed in `pipelines = (...)` on the module class.
- **Build.** After `on_initialized`, the module builds each one: `New(module)` (sync or async) if the class defines it, else the class with an empty `<Name>PipelineConfiguration()`. A class that needs more is bound in `on_initialized` with `self.expose_pipeline("items_pipeline", instance)`. A class that is neither buildable nor bound stops the module from starting, with a message naming it.
- **Name.** The class name in snake case, `items_pipeline`. Its durable workflow is `pipeline.items_pipeline`. Callers get the graph as `{"format": "text/turtle", "triples": n, "data": "<turtle>"}`, without `data` above 60 KiB: write a large graph to the triple store from the pipeline itself, as above.
- **Tools.** `self.get_pipeline_instance("items_pipeline").as_tools()` gives an agent one tool, which returns the graph as Turtle. An agent built in `on_initialized` needs `await self.build_processes()` first.
- **Dependencies.** The module declares `triple_store` (the services the class calls) and `document`, and runs with discovery. Importing the class needs `naas-abi-sdk[workflows,rdf]`, and `as_tools()` needs `[agents]`.

## REST API

Not applicable. There is no HTTP call that writes the pipeline class. Once the module runs, Nexus starts the pipeline as `pipeline.items_pipeline` through `/api/workflows`: see [Run a pipeline](/capabilities/build/pipelines/run-a-pipeline?via=api).

## Config

Not applicable. The class is not a `config.yaml` block. Its schedule, when it has one, is `job = JobSettings(...)` on the class: see [Schedule a pipeline](/capabilities/build/pipelines/schedule-a-pipeline).

## Scheduling a pipeline

A pipeline sets its triggers on the class, with `job = JobSettings(triggers=(...))`. The steps are on [Schedule a pipeline](/capabilities/build/pipelines/schedule-a-pipeline), and each run shows in the System app's Jobs tab ([Monitor runs](/capabilities/build/pipelines/monitor-runs)).

---

## Best practices

- **Idempotent**: running the same pipeline twice should not create duplicate triples. Use `owl:hasKey` in your ontology for entity deduplication, or check for existing triples before inserting.
- **Schema-first**: define (or reuse) an ontology class for every entity type you ingest before writing the pipeline.
- **Small batches**: for large datasets, paginate the integration and insert triples in batches to avoid memory pressure.
- **Log progress**: use the standard `logging` module; avoid `print()`.
- **Test `run()` directly**: build the class and `await` its `run` with a parameters instance, in `<Name>Pipeline_test.py` next to the file, as the scaffolded test does.
