Query Blocks and Databases

Problem

Our query blocks are quite limiting right now. We don't have enough extensibility for users to build their own systems, like other products allow (Notion, Tana, Obsidian, Roam Research, TiddlyWiki, etc).

Seed documents already have open-ended attributes: a document can carry metadata such as title, publish date, layout, tags, status, author, or any other field a user invents (though not exposed in the UI). Query blocks also already exist, but today they are closer to "show me the children of this document" than a real database view.

We also want to have different representations for the query results — tables, boards, lists, cards, etc.

We also want to have different kind of grouping, filtering, and sorting options.

The technical challenge is that open-ended metadata is easy to store but expensive to query. If rich query blocks filter client-side after fetching a whole directory, they will be slow, leaky, and hard to paginate correctly. Query execution needs to move into the backend, backed by generic database indexes over arbitrary document attributes.

Solution

Initial idea was to index all the attributes on the backend and expose an API to query documents based on those attributes. This is still what I'm suggesting, but with a few extra details.

What I'm proposing is basically to implement something like Notion Databases or Obsidian Bases. More something like Obsidian.

We expose a way for users to define attributes on documents. Like Markdown front-matter, but in our UI.

Users can define arbitrary attributes. We'd autocomplete attributes that we've already indexed.

We then add richness to our Query blocks, allowing to sort, compare, filter, and group by those attributes.

We add more representations to the query block, like boards, lists, etc.

We implement a backend API to resolve and paginate query blocks efficiently.

Schemas vs. Schemaless

I love schemas. But I think in a flexible system like ours they might be too rigid and restricting at this stage we're in. Documents in Seed are closer to open-ended knowledge objects than rows in a database table. A document might be a task, a note, an essay, a project, an event, or several of those at once.

Most schema-enforcement attempts have failed in the past: schema.org, Semantic Web, the sheer zoo of RDF vocabularies, and tons of different attempts at standardizing things. People just could never agree on standards.

Schemas are very useful in a more local context though. So eventually I do believe we should add them. But I have lots of innate concerns about formalizing schemas on the general-purpose level in our system right now. As with everything we do, I think we should start small. We still could have most of user-facing benefits of schemas, but treating them more as read-time lenses, than write-time declarations.

I like Rich Hickey's approach to this, which could be summarized as something like: "Put facts in maps, put rules at the edges, and don’t confuse today’s view with tomorrow’s truth.". I think this is what we should do.

So what I'm suggesting is that the rules should be defined on the query side, not on the document side. The query would define which attribute does it care about, filter any directories if needed, add the necessary predicates and map document attributes to tabular-style data, which we could then display as tables, lists, boards, etc.