Skip to content

ENG-2119 Document the mapping between CrossApp types and the Concept table #1290

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
maparent wants to merge 4 commits into
base: main
Choose a base branch
from
+222 −0
Open

ENG-2119 Document the mapping between CrossApp types and the Concept table #1290

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6c633fb
eng-2119 document crossapp-db mapping
maparent Aug 11, 2026
a20b72b
Add another special case
maparent Aug 12, 2026
325824a
Multiple corrections
maparent Aug 13, 2026
3c2f705
More corrections
maparent Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
222 changes: 222 additions & 0 deletions packages/database/doc/concept_design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
# Concept design

This document aims to explain the rationale of the Concept table, and how it relates to Discourse Graph data structures.

## DiscourseGraph structures
Comment thread
maparent marked this conversation as resolved.

@mdroidian mdroidian Aug 13, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's please change this to Discourse Graph


DiscourseGraph assumes a basic graph data model: All knowledge objects are Nodes (with literal properties and a text description) or Relations between Nodes (without attributes other than its source or destination.)

@mdroidian mdroidian Aug 13, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's please change this to Discourse Graph


Both Nodes and Relations are typed, and thus we have Node schemas describing node types.
Relations schemas allow for constraints on source and destination node types.
The [Roam Research implementation](https://github.com/DiscourseGraphs/discourse-graph/tree/main/apps/roam) of relation schemas contains a specific pair of constraints, forming the triple: (source node type, relation label, destination node type).
Many triples with a common label are implicitly connected.

The [Obsidian implementation](https://github.com/DiscourseGraphs/discourse-graph/tree/main/apps/obsidian) made that connection explicit, and further defined a RelationType schema, with the label information, while the RelationTriple schema refers to the type constraint triple. Both this schema (with triple and type) and the roam-style schema that combines triple and type information are modeled with the RelationTripleSchema type.

## Generalization

When designing the database format, I (Marc-Antoine) thought that those five kinds (Node and Relation instances, Node schemas, RelationType schemas, RelationTriple schemas) were really special cases of a single underlying data object type, which I called Concept.

### Underlying rationale

Note: Implementors can skip this section.

This generalization is rooted in prior experience with RDF, TopicMaps, semantic Frames, and David Spivak's [algebraic database](https://arxiv.org/abs/1602.03501) representation. In all cases, any type is defined by the attributes it can have. RDF calls them properties, Minsky's frames uses the term slots, etc. RDF/OWL distinguishes literal attributes (`owl: DatatypeProperty`, stored in the `literal_content` column) from attributes which are references to other objects (`owl:ObjectProperty` stored in the `reference_content` column). We adopted the term roles for ObjectProperties from TopicMaps. Spivak's work gives this distinction a categorical grounding.

Unifying relations and nodes in particular allows relations to both have extra attributes (as in a PropertyGraph vs. a strict RDF graph) and be referred to (either as the source/destination of another Relation, or as a target of an `ObjectProperty`.) Cliff Joslyn described this recursive mathematical structure as a [übergraph](https://arxiv.org/abs/1704.05547v1). It can also be expressed as reified relations in RDF-\*.

### Use cases

Having ObjectProperties makes it possible to represent certain special cases of DiscourseGraphs, such as the necessary relation between Evidence and Source.

@mdroidian mdroidian Aug 13, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's please change this to Discourse Graph


Having referenceable relations allows arguments to be made about relations.

However, it does mean that there is an underlying tension between what should be modeled as an ObjectProperty of a type vs a reified Relation. In general, ObjectProperties should be used whenever the relation is constitutive of the source object. This is akin to the composition vs aggregation distinction in UML.

### Mapping between CrossApp types and the database.

The five object kinds are not represented directly in the database, the mapping needs explaining. I will use the CrossApp types as a basis to make the distinction.

#### CrossApp type fields (taken from `src/crossAppContract.ts`)

```yaml
Base:
localId: LocalId
rid?: string
createdAt: Date
modifiedAt?: Date
authorId: LocalId

CrossAppNodeSchema:
# Base and...
metadata?: Json
label: string
template?: string
templateTitle?: string

CrossAppRelationTypeSchema:
# Base and...
metadata?: Json
label: string
complement: string

CrossAppRelationTripleSchema (Roam):
# Base and...
metadata?: Json
label: string
complement: string
sourceType: LocalId
destinationType: LocalId

CrossAppRelationTripleSchema (Obsidian):
# Base and...
metadata?: Json
relation: LocalId
sourceType: LocalId
destinationType: LocalId

CrossAppNode:
# Base and...
nodeType: LocalId
content.direct: InlineCrossAppContent
content.full?: InlineCrossAppTypedContent

CrossAppRelation:
# Base and...
relationType: LocalId # Refers to CrossAppRelationTripleSchema in Roam, CrossAppRelationTypeSchema in Obsidian
source: LocalId | Rid
destination: LocalId | Rid
```

#### Matching of common fields between CrossApp and Concept

| CrossApp field | | Concept column |
| -------------- | --- | -------------------------------- |
| - | | `id` |
| - | | `epistemic_status` |
| - | | `description` |
| `localId` | <-> | `source_local_id` |
| `rid` | <~> | `space_id` |
| `createdAt` | <-> | `created` |
| `modifiedAt` | <-> | `last_modified` |
| `authorId` | <-> | `author_local_id` => `author_id` |
| `metadata` | <- | `literal_content` |

Note on metadata: It is read from the database, but currently not written.
When some specific keys of `literal_content` are mapped to a CrossApp field, those key-value pairs are not included again in the `metadata`. Thus `metadata` is a grab-bag for residual data. (Eg color for now.)

Residual (not otherwise accounted for) keys in Obsidian frontmatter are mapped to `literal_content->source_data`. (Not through CrossApp.)

#### Matching of CrossAppNodeSchema and Concept

| CrossAppNodeSchema | Concept | value |
| ------------------ | ----------------------------------- | ----- |
| `label` | `name` | {} |
| `template` | `literal_content->template_content` | |
| `templateTitle` | `literal_content->template` | |
| - | `is_schema` | true |
| - | `schema_id` | null |
| - | `arity` | 0 |
| - | `reference_content` | {} |

Note that this does not yet allow for ObjectProperties to be defined; this would require to define `literal_content->roles`, corresponding `reference_content`s, and change the `arity`. See below.

Also note: In Obsidian, where we do not go through CrossAppNodeSchema, the label is also assigned to `literal_content->label`.

Query filter: `.eq("arity",0).eq("is_schema", true)`

#### Matching of CrossAppNode and Concept

| CrossAppNode | Concept | value |
| ----------------------- | ----------------------------------------------- | ------------------ |
| `content->direct->text` | `name` | false |

@graphite-app graphite-app Bot Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incorrect value in the mapping table. The name column should contain the text from content->direct->text, not the boolean value false. The value false appears to be misaligned and actually belongs to the is_schema field in the next row (line 133).

| `content->direct->text` | `name`                                          |                    |
| -                       | `is_schema`                                     | false              |

This documentation error could mislead developers implementing the mapping between CrossAppNode and Concept.

Suggested change
| `content->direct->text` | `name` | false |
| `content->direct->text` | `name` | |
| - | `is_schema` | false |

Spotted by Graphite

Fix in Graphite

Is this helpful? React 👍 or 👎 to let us know.

@mdroidian mdroidian Aug 13, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maparent

| - | `is_schema` | false |
| - | `arity` | 0 |
| - | `reference_content` | {} |
| `nodeType` | `schema_represented_by_local_id` => `schema_id` | ref to Node schema |

Query filter: `.eq("arity",0).eq("is_schema", false)`

#### Matching of CrossAppRelationTypeSchema and Concept (Obsidian only)

| CrossAppRelationTypeSchema | Concept | value |
| -------------------------- | -------------------------------- | --------------------------- |
| - | `literal_content->roles` | `["source", "destination"]` |
| - | `is_schema` | true |
| - | `schema_id` | null |
| - | `arity` | 2 |
| - | `reference_content` | {} |
| `label` | `name`, `literal_content->label` | |
| `complement` | `literal_content->complement` | |

Query filter: `.eq("arity",2).eq("is_schema", true).is("reference_content->source", "null")`

#### Matching of Obsidian CrossAppRelationTripleSchema and Concept

| CrossAppRelationTripleSchema | Concept | value |
| ---------------------------- | ---------------------------------- | ------------------------------ |
| - | `name` | composite |
| - | `literal_content->label` | taken from RelationType schema |
| - | `literal_content->complement` | taken from RelationType schema |
| - | `literal_content->roles` | `["source", "destination"]` |
| - | `is_schema` | true |
| - | `schema_id` | null |
| - | `arity` | 2 |
| `sourceType` | `reference_content->source` | ref to Node schema |
| `destinationType` | `reference_content->destination` | ref to Node schema |
| `relation` | `reference_content->relation_type` | ref to RelationType schema |

Note that putting the relationType in `reference_content->relation_type` without a corresponding role was a hackish shortcut, and should be revisited (see below.)

Query filter: `.eq("arity",2).eq("is_schema", true).not("reference_content->relation_type", "is", "null")`

#### Matching of Roam CrossAppRelationTripleSchema and Concept

| CrossAppRelationTripleSchema | Concept | value |
| ---------------------------- | -------------------------------- | --------------------------- |
| `id` | `name` | |
| - | `literal_content->roles` | `["source", "destination"]` |
| - | `is_schema` | true |
| - | `schema_id` | null |
| - | `arity` | 2 |
| `label` | `literal_content->label` | |
| `complement` | `literal_content->complement` | |
| `sourceType` | `reference_content->source` | ref to Node schema |
| `destinationType` | `reference_content->destination` | ref to Node schema |

Query filter: `.eq("arity",2).eq("is_schema", true).not("reference_content->source", "is", "null").is("reference_content->relation_type", "null")`

In most cases, you would want both Roam and Obsidian RelationTripleSchemas, hence you would simply use:

Combined query filter: `.eq("arity",2).eq("is_schema", true).not("reference_content->source", "is", "null")`

#### Matching of Obsidian CrossAppRelation and Concept

| CrossAppRelation | Concept | value |
| ---------------- | -------------------------------- | ---------------------------- |
| - | `name` | composite |
| - | `is_schema` | false |
| - | `arity` | 2 |
| `relationType` | `schema_id` (in Obsidian) | ref to RelationType schema |
| `relationType` | `schema_id` (in Roam) | ref to RelationTriple schema |
| `source` | `reference_content->source` | ref to Node |
| `destination` | `reference_content->destination` | ref to Node |

Query filter: `.eq("arity",2).eq("is_schema", false)`

### Design considerations and future changes

#### Arity and ObjectProperties

The arity on either a schema or instance is based on the size of the `literal_content->roles` array in the schema.
The `reference_content` allows either single or multiple values (`Record<string, number|number[]>`). The `reference_content` values are collated in a computed column `refs`, whose index allows for efficient filters on sql queries before digging into the `reference_content` jsonb.

As a first approximation, we distinguished relations from nodes using `arity==2`, but this precludes using ObjectProperties.

To remedy this, we propose adding a computed column `is_relation`, which would check whether (source, destination) are both in the schema's roles. The query filters using `arity` would be redefined to use `is_relation`.

Internal node references such as the Evidence's Source can then be expressed using roles and internal relations.

In `RelationTripleSchema`, we refer to the `RelationTypeSchema` with a `relation_type` entry in the `reference_content` column. This, unusually, is not backed by an entry in the `roles`. This is a deviation from the mental model, but in the current situation, adding that role would break the `arity=2` checks. Introducing the `is_relation` column will also allow this to be part of the roles.

Note that we probably won't add a range constraint in that case; the constraint should require the `relation_type` to be any `RelationTypeSchema`, but there is no row materializing this meta-class.