Skip to content

Commit

Permalink
use ssb-meta-feeds with v1 tree structure (#10)
Browse files Browse the repository at this point in the history 
* use ssb-meta-feeds with v1 tree structure

* improve README with arj's feedback

* improve Template._matchShard() readability

* update ssb-meta-feeds
  • Loading branch information
@staltz
staltz authored Oct 25, 2022
1 parent 92451d7 commit d88fde4
Show file tree
Hide file tree
Showing 6 changed files with 288 additions and 398 deletions.
194 changes: 85 additions & 109 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<!--
SPDX-FileCopyrightText: 2021 Andre 'Staltz' Medeiros
SPDX-FileCopyrightText: 2021-2022 Andre 'Staltz' Medeiros
SPDX-License-Identifier: CC0-1.0
-->
Expand All @@ -17,7 +17,7 @@ Depends on ssb-friends APIs, and calls ssb-ebt APIs.
- Requires **Node.js 10** or higher
- Requires **ssb-db** or **ssb-db2**
- Requires [**ssb-friends**](https://github.com/ssbc/ssb-friends) version **5.0** or higher
- Requires [**ssb-ebt**](https://github.com/ssbc/ssb-ebt) version **7.0** or higher
- Requires [**ssb-ebt**](https://github.com/ssbc/ssb-ebt) version **9.0** or higher

```
npm install --save ssb-replication-scheduler
Expand Down Expand Up @@ -160,34 +160,30 @@ the template is `null` or a falsy value, then it means that for that hops level
we don't do partial replication and we **will** do **full** replication (which
means pre-2022 SSB replication of the peer's `main` feed).

When the template is a JSON tree of objects and arrays, where the root of the
tree is always the _root meta feed_. The template describes which **keys** in
the metafeeds and subfeeds must match exactly the **values** given. So that if
we write `feedpurposes: 'indexes'`, it means we are interested in matching the
metafeed that has the field `feedpurposes` exactly matching the value "indexes".
All specified fields must match, but omitted fields are allowed to be any value.
When the template is a JSON array, it means we want to replicate only some leaf
feeds in the "metafeed tree", where the root of the tree is always the
_root meta feed_. The structure of the tree is assumed to follow the
["tree structure v1"](https://github.com/ssbc/ssb-meta-feeds-spec#v1), which
means we're only concerned about the leaf feeds.

The field `subfeeds` is not matching an actual field, instead, it is assumes we
are dealing with a meta feed and this is describing its subfeeds that we would
like to replicate.
Each item in the template should be an object describing which **keys** in a
leaf feed must match exactly the **values** given for that leaf to be
replicated. So that if we write `{purpose: 'git-ssb'}`, it means we are
interested in matching the leaf feed that has the field `purpose` exactly
matching the value "git-ssb". All *specified* fields must match, but *omitted*
fields are allowed to be any value. If you omit all the fields, i.e. if you pass
the empty object `{}`, then this means "replicate **ALL** leaf feeds".

#### Special variables

Some keys and some values are special, in the sense that they are not taken
literally, but are going to be substituted by other context-relative values.
These special variables are always prefixed with **`$`**.
Some values are special, in the sense that they are not taken literally, but are
going to be substituted by other context-relative values. These special
variables are always prefixed with **`$`**.

- Special keys
- `$format`
- Special values
- `$main`
- `$root`

The field _key_ `$format` refers to [ssb-ebt](https://github.com/ssbc/ssb-ebt)
"replication formats" and can be included in a template to specify which
replication format to use in ssb-ebt. The value of this field should be the
format's name as a string.

If the value of a field, e.g. in ssb-ql-0 queries, are the special strings
`"$main"` or `"$root"`, then they respectively refer to the IDs of the _main
feed_ and of the _root meta feed_.
Expand All @@ -202,97 +198,77 @@ In the example below, we set up partial replication with the meaning:

```js
partialReplication: {
0: {
subfeeds: [
{ feedpurpose: 'coolgame' },
{ feedpurpose: 'git-ssb' },
{
feedpurpose: 'indexes',
subfeeds: [
{
feedpurpose: 'index',
$format: 'indexed',
},
],
0: [
{ purpose: 'main' },
{ purpose: 'coolgame' },
{ purpose: 'git-ssb' },
{ purpose: 'index' }
],

1: [
{
purpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: null, private: true },
},
},
{
purpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'post', private: false },
},
},
{
purpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'vote', private: false },
},
},
{
purpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'about', private: false },
},
},
{
purpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'contact', private: false },
},
},
],

2: [
{
purpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'about', private: false },
},
],
},

1: {
subfeeds: [
{
feedpurpose: 'indexes',
subfeeds: [
{
feedpurpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: null, private: true },
},
$format: 'indexed',
},
{
feedpurpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'post', private: false },
},
$format: 'indexed',
},
{
feedpurpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'vote', private: false },
},
$format: 'indexed',
},
{
feedpurpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'about', private: false },
},
$format: 'indexed',
},
{
feedpurpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'contact', private: false },
},
$format: 'indexed',
},
],
},
{
purpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'contact', private: false },
},
],
},

2: {
subfeeds: [
{
feedpurpose: 'indexes',
subfeeds: [
{
feedpurpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'about', private: false },
},
$format: 'indexed',
},
{
feedpurpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'contact', private: false },
},
$format: 'indexed',
},
],
},
],

3: [
{
purpose: 'index',
metadata: {
querylang: 'ssb-ql-0',
query: { author: '$main', type: 'about', private: false },
},
],
},
},
],
}
```

Expand Down
6 changes: 3 additions & 3 deletions metafeed-finder.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,11 @@ module.exports = class MetafeedFinder {
const metaFeedId = this._map.get(mainFeedId)
cb(null, metaFeedId)
} else if (mainFeedId === this._ssb.id) {
this._ssb.metafeeds.getRoot((err, rootMF) => {
this._ssb.metafeeds.findOrCreate((err, rootMF) => {
if (err) cb(err)
else if (!rootMF) cb(null, null)
else {
const metaFeedId = rootMF.keys.id
const metaFeedId = rootMF.id
this._map.set(mainFeedId, metaFeedId)
this._inverseMap.set(metaFeedId, mainFeedId)
cb(null, metaFeedId)
Expand Down Expand Up @@ -223,7 +223,7 @@ module.exports = class MetafeedFinder {
this._requestsByMainfeedId.clear()

await this._forEachNeighborPeer((rpc, goToNextNeighbor) => {
debug('"getSubset" on peer %s for metafeed/announce messages', rpc.id)
debug('"getSubset" for peer %s for metafeed/announce messages', rpc.id)
pull(
rpc.getSubset(this._makeQL1(requests), { querylang: 'ssb-ql-1' }),
pull.filter((value) => this._validateMetafeedAnnounce({ value })),
Expand Down
8 changes: 4 additions & 4 deletions package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@
"pull-pushable": "^2.2.0",
"pull-stream": "^3.6.0",
"ssb-db2": ">=3.0.0 <=6",
"ssb-ebt": "~9.0.0",
"ssb-meta-feeds": "~0.29.0",
"ssb-ebt": "^9.1.2",
"ssb-meta-feeds": "~0.35.2",
"ssb-network-errors": "^1.0.1",
"ssb-subset-ql": "^1.0.0"
},
Expand All @@ -44,11 +44,11 @@
"ssb-bendy-butt": "^1.0.1",
"ssb-caps": "^1.1.0",
"ssb-db": "^20.4.0",
"ssb-db2": "^6.2.2",
"ssb-db2": "^6.2.4",
"ssb-conn": "^6.0.4",
"ssb-fixtures": "^3.0.4",
"ssb-friends": "^5.1.7",
"ssb-index-feed-writer": "~0.8.0",
"ssb-index-feeds": "0.10.2",
"ssb-keys": "^8.5.0",
"ssb-subset-rpc": "~0.3.2",
"tap-arc": "~0.3.5",
Expand Down
Loading

0 comments on commit d88fde4

Please sign in to comment.