Canonical page: https://docs.singlebase.io/data

# Data

Store JSON documents in collections and work with them through `data.*` operations. The same records power your application and give your agents context.

## Overview

**DocumentDB** gives your applications and agents three ways to work with the same data: structured JSON queries, similarity search by meaning, and full-text search for words and phrases. Eligible document content is vectorized into embeddings for VectorDB similarity search, while the database's FTS5 index supports keyword search. Embeddings power similarity search; FTS5 uses its own text index.

Your app creates and updates records; your agents read the ones you allow. Store your data once, then retrieve the context that fits the task through configured collections and [docsets](/chat.md#docsets). Vector indexing and full-text search depend on your project configuration.

**One database, three ways in:**
- **Structured queries** - [Filters](/queries.md), sorting, pagination, and aggregation
- **FTS5 full-text search** - Match names, phrases, and keywords with [`search`](#full-text-search)
- **VectorDB embeddings and vector search** - Point a vector [docset](/chat.md#docsets) at a collection for similarity search

**Hybrid retrieval** combines these search paths when grounding chat answers. A vector docset searches indexed collections and Smart Bucket file vectors, and uses full-text search for its other collections. You can also combine keyword and vector docsets in one request. See [Chat search and hybrid retrieval](/chat.md#search-and-hybrid-retrieval).

**Key features:**
- Schemaless JSON documents identified by `_id`
- Single or bulk inserts and updates
- [Data operators](#data-operators) such as `views:$incr` and `tags:$push`
- [Templates](/templates.md) that compute values on write
- Per-user ownership with `_user_id`
- Per-agent, per-collection access with [agent keys](/agents.md)
- Soft deletes with configurable retention

**Use cases:** Product catalogs, knowledge records, orders, user content, agent memory and notes, app state

---

## Operations

Every `data.*` operation requires `payload.collection`.

| Operation | Description | Required payload |
|---|---|---|
| `data.query` | Retrieve documents | `collection` |
| `data.count` | Count matching documents | `collection` |
| `data.aggregate` | Compute aggregate values | `collection`, `compute` |
| `data.insert` | Insert one or more documents | `collection`, `data` |
| `data.update` | Update documents by `_id` or filter | `collection`, `data` |
| `data.upsert` | Update matches, or insert when nothing matches | `collection`, `filter`, `insert_data` |
| `data.archive` | Archive documents with optional retention | `collection`, one selector |
| `data.delete` | Archive documents with seven-day retention | `collection`, one selector |

---

## Document Fields

Singlebase sets these fields. You can read these fields. Use `_user_id` at the payload root to select an owner; other fields can be used in filters and sorting. Updates and upserts ignore `_created_at` and `_modified_at`.

| Field | Description |
|---|---|
| `_id` | Document ID |
| `_user_id` | Owner's user ID, when set. See [Ownership](#ownership) |
| `_created_at` | Creation time |
| `_modified_at` | Last update time |

---

## Ownership

`_user_id` links a document to a user. How it behaves depends on the caller.

**Signed-in users:**
- Reads (`data.query`, `data.count`, `data.aggregate`) and mutations (`data.update`, `data.upsert`, `data.archive`, `data.delete`) are limited to the signed-in user's documents.
- `data.insert` automatically assigns the signed-in user's ID, even when `_user_id` is omitted. A caller-supplied owner is replaced with that ID.
- A payload-level `_user_id` on scoped reads or mutations must equal the signed-in user's ID; another value returns `USER_ID_MISMATCH`.
- Changing ownership through `data.update` is refused (`USER_ID_UPDATE_REQUIRES_ADMIN`).
- Reading, updating, upserting, archiving, or deleting without a bearer token requires a privileged server or agent key (`AUTHENTICATION_REQUIRED` otherwise).

**Server and agent keys without a bearer token:**
- Read and mutate across owners, subject to the agent key's collection permissions.
- Scope a request to one existing project user by sending `_user_id` at the **payload root**, alongside `filter`. This selects an owner; it does not reassign ownership.
- Set an owner on insert with payload-level `_user_id` or `data._user_id`.
- Reassign an owner only with a single update-by-ID object containing `data._user_id`. The new user must exist. Filter-based or bulk owner changes return `OWNER_CHANGE_REQUIRES_SINGLE_DOCUMENT`.

**Ownership selectors:**
Never put `_user_id` inside `filter`, including nested filters or `_user_id:$in`; this returns `INVALID_USER_ID_FILTER`. Use the payload root instead:

```json
{
  "operation": "data.query",
  "payload": {
    "collection": "notes",
    "_user_id": "<EXISTING_USER_ID>",
    "filter": { "status": "active" },
    "per_page": 20
  }
}
```

For scoped mutations, selecting more matching documents than the scope limit allows returns `SCOPE_TOO_LARGE`. A requested update ID outside the selected owner returns `DOCUMENT_NOT_FOUND`.

---

## data.query

Retrieve documents from a collection.

```json
{
  "operation": "data.query",
  "payload": {
    "collection": "notes",
    "filter": { "status": "active" },
    "sort": "_created_at desc",
    "page": 1,
    "per_page": 20
  }
}
```

**Parameters:**

| Field | Description |
|---|---|
| `collection` | Collection name (required) |
| `filter` | Match conditions. See [Query Syntax](/queries.md) |
| `search` | Full-text search string |
| `sort` | Sort expression. Default `_created_at desc` |
| `page`, `per_page` | Page-based pagination. Defaults `1` and `20` |
| `limit`, `offset` | Offset-based pagination |
| `compute` | Computed fields |
| `lookups` | Lookups into related collections |
| `archive_only` | Return archived documents only. Default `false` |
| `_user_id` | Scope the request to one user; send at the payload root |

**Response:**
```json
{
  "data": [
    {
      "_id": "note_123",
      "_user_id": "user_123",
      "title": "First note",
      "status": "active",
      "_created_at": "2026-10-03T12:00:00+00:00",
      "_modified_at": "2026-10-03T12:00:00+00:00"
    }
  ],
  "meta": {
    "pagination": {
      "total_items": 84,
      "count": 20,
      "per_page": 20,
      "page": 1,
      "total_pages": 5,
      "next_page": 2,
      "prev_page": null
    }
  },
  "exec_time": 0.018
}
```

**Example - Get one document:**
```json
{
  "operation": "data.query",
  "payload": {
    "collection": "notes",
    "filter": { "_id": "note_123" },
    "limit": 1
  }
}
```

The result is still an array.

---

## data.count

Count documents that match a filter.

```json
{
  "operation": "data.count",
  "payload": {
    "collection": "notes",
    "filter": { "status": "active" }
  }
}
```

**Response:**
```json
{
  "data": { "count": 42 }
}
```

`filter` and full-text `search` select the records to count. The [ownership](#ownership) rules of `data.query` apply.

---

## data.aggregate

Compute values such as totals and averages.

```json
{
  "operation": "data.aggregate",
  "payload": {
    "collection": "orders",
    "filter": { "status": "paid" },
    "compute": {
      "orders": { "$count": "*" },
      "revenue": { "$sum": "total" },
      "average_order": { "$avg": "total" }
    }
  }
}
```

**Parameters:**

| Field | Description |
|---|---|
| `collection` | Collection name (required) |
| `compute` | Named aggregate definitions (required) |
| `filter` | Match conditions |
| `archive_only` | Aggregate archived documents only |
| `include_archive` | Include archived documents |
| `cache` | Cache the result for this many seconds |

The aggregate result is returned in `data`. An empty result returns `data: {}`.

::: warning
`data.aggregate` is scoped to the signed-in user. With a server or agent key, use payload-level `_user_id` for totals belonging to one user. Never put ownership inside `filter`.
:::

---

## data.insert

Insert one document or an array of documents.

```json
{
  "operation": "data.insert",
  "payload": {
    "collection": "notes",
    "data": {
      "title": "First note",
      "status": "active",
      "views": 0,
      "tags": ["draft"]
    }
  }
}
```

**Parameters:**

| Field | Description |
|---|---|
| `collection` | Collection name (required) |
| `data` | A document object, or an array of documents (required) |
| `_user_id` | Owner for every inserted document. See [Ownership](#ownership) |
| `unique_fields` | Fields that must be unique, such as `["email"]` |
| `on_conflict` | What to do when a unique field conflicts. Default `"error"` |
| `ttl_seconds` | Expire documents after this many seconds |
| `metadata` | Hidden document metadata, stored in `_metadata` |

With a bearer token, this insert automatically belongs to the signed-in user. With a server or agent key and no bearer token, send an existing user ID in payload-level `_user_id` to assign an owner; omit it for an unowned document.

**Response:**
```json
{
  "data": [
    {
      "_id": "note_123",
      "_user_id": "user_123",
      "title": "First note",
      "status": "active",
      "views": 0,
      "tags": ["draft"]
    }
  ]
}
```

**Example - Bulk insert with a unique field:**
```json
{
  "operation": "data.insert",
  "payload": {
    "collection": "subscribers",
    "data": [
      { "email": "ada@example.com", "plan": "free" },
      { "email": "grace@example.com", "plan": "pro" }
    ],
    "unique_fields": ["email"]
  }
}
```

---

## data.update

Update documents by `_id`, or patch every document that matches a filter.

**Example - Update by ID:**
```json
{
  "operation": "data.update",
  "payload": {
    "collection": "notes",
    "data": {
      "_id": "note_123",
      "title": "Updated note",
      "views:$incr": 1
    }
  }
}
```

**Example - Update several documents by ID:**
```json
{
  "operation": "data.update",
  "payload": {
    "collection": "notes",
    "data": [
      { "_id": "note_123", "status": "active" },
      { "_id": "note_456", "status": "archived" }
    ]
  }
}
```

**Example - Update by filter:**
```json
{
  "operation": "data.update",
  "payload": {
    "collection": "notes",
    "filter": { "status": "draft" },
    "data": { "status": "active" },
    "max_docs": 100
  }
}
```

**Parameters:**

| Field | Description |
|---|---|
| `collection` | Collection name (required) |
| `data` | A patch object, or an array of patches (required) |
| `filter` | Select documents to patch. When set, use one patch object |
| `max_docs` | Maximum number of documents to change |
| `_user_id` | Select documents owned by this user; send at the payload root |

**Rules:**
- Without `filter`, every patch needs a non-empty `_id` (`MISSING_DOCUMENT_ID`).
- An empty `filter: {}` counts as a filter and can match many documents. Use a narrow filter and `max_docs`.
- Plain nested objects merge with existing values. Use `:$replace` to replace an object.

Returns the updated documents in `data`.

---

## data.upsert

Update matching documents, or insert a new one when nothing matches.

```json
{
  "operation": "data.upsert",
  "payload": {
    "collection": "page_views",
    "filter": { "slug": "release-notes" },
    "insert_data": {
      "slug": "release-notes",
      "views": 1
    },
    "update_data": {
      "views:$incr": 1
    },
    "max_docs": 1
  }
}
```

**Parameters:**

| Field | Description |
|---|---|
| `collection` | Collection name (required) |
| `filter` | Match conditions (required) |
| `insert_data` | Document to insert when nothing matches (required) |
| `update_data` | Patch applied to matches |
| `max_docs` | Maximum number of documents to update |

Upsert scopes matches to the signed-in user and assigns that user to a new document. Server or agent callers can scope the request with payload-level `_user_id`. Ownership conditions inside `filter` are rejected. Upsert cannot reassign an existing owner.

---

## data.archive

Archive documents. Archived documents are hidden from normal queries.

```json
{
  "operation": "data.archive",
  "payload": {
    "collection": "notes",
    "ids": ["note_123", "note_456"],
    "ttl_seconds": 2592000
  }
}
```

**Parameters:**

| Field | Description |
|---|---|
| `collection` | Collection name (required) |
| `id`, `ids`, or `filter` | Exactly one selector (required) |
| `ttl_seconds` | Keep archived documents this many seconds before removal |
| `max_docs` | Maximum number of documents to archive |

**Response:**
```json
{
  "data": {
    "count": 2,
    "deleted_count": 2,
    "items": [],
    "txn_id": "txn_123"
  }
}
```

Read archived documents with `data.query` and `archive_only: true`. There is no unarchive operation.

---

## data.delete

Delete documents. `data.delete` archives them with a fixed seven-day retention (604800 seconds), then they are removed.

```json
{
  "operation": "data.delete",
  "payload": {
    "collection": "notes",
    "id": "note_123"
  }
}
```

Requires exactly one of `id`, `ids`, or `filter` (`EXACTLY_ONE_ARCHIVE_SELECTOR_REQUIRED`). `max_docs` is optional; `ttl_seconds` is ignored. The response matches `data.archive`.

---

## Full-text Search

Find documents by the words in them. Add `search` to `data.query`, and combine it with `filter` to narrow the results.

```json
{
  "operation": "data.query",
  "payload": {
    "collection": "product_catalog",
    "search": "laptop backpack",
    "filter": { "category": "bags" },
    "per_page": 10
  }
}
```

Full-text search uses SQLite FTS5 and depends on the collection's search configuration. Agent keys need the `search` permission on the collection.

**VectorDB embeddings and vector search:** Collection indexing creates embeddings from document content and stores chunks in the project vector index. Vector retrieval depends on enabled indexing and available embeddings. To find records that match the intent of a request, even when the words differ, create a [docset](/chat.md#docsets) with vector search over the collection and use it from `chat.send` or `chat.ask`. Docsets can also use full-text search, and an answer can draw on several docsets.

---

## Data Operators

Change a field in place with `field:$operator` keys in `data`, `update_data`, or `insert_data`. Use dot notation for nested fields.

```json
{
  "operation": "data.update",
  "payload": {
    "collection": "posts",
    "data": {
      "_id": "post_123",
      "stats.views:$incr": 1,
      "tags:$push": "featured",
      "draft_notes:$unset": true
    }
  }
}
```

| Operator | Effect | Example |
|---|---|---|
| `field` or `field:$set` | Set a value | `"status": "active"` |
| `field:$replace` | Replace a value, including a whole object | `"settings:$replace": {"theme": "dark"}` |
| `field:$unset` | Remove the field | `"obsolete:$unset": true` |
| `field:$incr` | Add a number (alias `$inc`) | `"views:$incr": 1` |
| `field:$decr` | Subtract a number (alias `$dec`) | `"stock:$decr": 2` |
| `field:$push` | Append one item to an array | `"tags:$push": "reviewed"` |
| `field:$lpush` | Prepend one item to an array | `"tags:$lpush": "pinned"` |
| `field:$extend` | Append several items | `"tags:$extend": ["api", "docs"]` |
| `field:$lextend` | Prepend several items | `"tags:$lextend": ["new", "hot"]` |
| `field:$pop` | Remove the last array item | `"tags:$pop": true` |
| `field:$remove` | Remove matching array items | `"tags:$remove": "draft"` |
| `field:$clear` | Empty an array | `"tags:$clear": true` |
| `field:$rename` | Move the value to another path | `"old_title:$rename": "title"` |
| `field:$copy` | Copy the value to another path | `"title:$copy": "original_title"` |

**Merge behavior:**
```javascript
// Existing: { "settings": { "theme": "light", "lang": "en" } }

{ "settings": { "theme": "dark" } }
// Result:   { "settings": { "theme": "dark", "lang": "en" } }

{ "settings:$replace": { "theme": "dark" } }
// Result:   { "settings": { "theme": "dark" } }
```

::: warning Not atomic
Writes that use operators or templates read the current documents, apply the changes, then save them. A concurrent write can land in between, so `$incr` is not an atomic counter.
:::

---

## Templates

Compute values from the document itself with [templates](/templates.md). Templates work in `data.insert`, `data.update`, and `data.upsert`.

```json
{
  "operation": "data.insert",
  "payload": {
    "collection": "orders",
    "data": {
      "product": "Notebook",
      "price": 12,
      "quantity": 3,
      "label": "{{ product }} x {{ quantity }}",
      "total": "&:{{ price * quantity }}",
      "slug": "{{ $slugify(product) }}",
      "created_on": "{{ $now() | date_format('YYYY-MM-DD') }}"
    }
  }
}
```

**Result:**
```json
{
  "product": "Notebook",
  "price": 12,
  "quantity": 3,
  "label": "Notebook x 3",
  "total": 36,
  "slug": "notebook",
  "created_on": "2026-10-03"
}
```

Plain values and operators are applied first, then templates are rendered against the result. See [Templates](/templates.md) for the full syntax and helper list.

---

## Automations

When enabled for the project and configured for a collection, automations can copy fields, generate slugs, or run AI tasks. Eligible local actions on insert run before persistence and can affect the returned document. Deferred actions, including AI work, run in the background after the write. Later queries can see their results.

---

## Common Errors

| Message | Meaning |
|---|---|
| `MISSING_COLLECTION_NAME` | `payload.collection` is missing or blank |
| `AUTHENTICATION_REQUIRED` | A scoped read or mutation has neither a signed-in user nor a privileged key |
| `USER_NOT_FOUND` | A payload-level owner or reassignment names an unknown project user |
| `USER_ID_UPDATE_REQUIRES_ADMIN` | A signed-in user tried to change `_user_id` |
| `MISSING_DOCUMENT_ID` | An update without `filter` has a patch with no `_id` |
| `MISSING_UPSERT_INSERT_DATA` | `insert_data` is missing |
| `USER_ID_MISMATCH` | Requested scope differs from the signed-in user |
| `INVALID_USER_ID_FILTER` | Ownership was placed inside a filter |
| `OWNER_CHANGE_REQUIRES_SINGLE_DOCUMENT` | Owner reassignment was attempted with a filter, bulk update, or upsert |
| `SCOPE_TOO_LARGE` | Scoped mutation selection exceeds its limit |
| `DOCUMENT_NOT_FOUND` | A selected document is not owned by the scoped user |
| `EXACTLY_ONE_ARCHIVE_SELECTOR_REQUIRED` | Archive or delete needs exactly one of `id`, `ids`, `filter` |

Empty results are not errors: queries and writes return `data: []` and aggregate returns `data: {}`, with `message: "NO_DATA"`.

---

## Next Steps

- [Query Syntax](/queries.md) - Filter operators, sorting, pagination, and search
- [Templates](/templates.md) - Expressions and helper functions
- [Authentication](/auth.md) - Get the ID token that scopes data to a user
- [Agents](/agents.md) - Grant collection access to AI agents
- [Conversation](/chat.md#docsets) - Search by meaning with docsets