Extract RDF Interfaces to avoid conversions

[filip26]

Let's collaborate on a new repository with RDF dataset interfaces that would allow seamless integration between Jena & Titanium and maybe other libs, like URDNA2015, N-QUADS Reader/Writer (bundled with Titanium now) etc.

The interface(s) should be minimal and single purpose, to exchange RDF dataset, i.e. reading. It is hard, and there are existing attempts, but something like micro-jars, containing only minimum necessary, single purpose, interfaces could be a way to deal with limitations/features coming from strongly typed language and dependencies linking enforcing strong consistency. In comparison to the JavaScript world where compatibility is declared.

[rvesse]

Well there's Commons RDF but that's essentially a dead project because none of the contributing projects could agree on anything 🤣

Turning this around could we think about inverting the control flow here?

Part of the overhead right now AFAIK is that we have to hand full control over to Titanium and then convert its final output data model back into Jena's data model. If instead Titanium had an output interface in the vein of Jena's StreamRDF then we could hand a Jena specific implementation of that interface to Titanium. Then Titanium call the relevant methods on its interface as it produces triples which Jena can then immediately convert to its own types in a streaming fashion.

I don't know if this is feasible within the JSON-LD processing model but perhaps worth considering. WDYT?

[rvesse]

Ok, not sure what aspect of StreamRDF you consider non-minimal, maybe you could expand on that? I realise that example is obviously tied to Jena's types for the RDF model but it's a pretty straightforward output interface.

Note that the lack of ability to stream process JSON-LD doesn't mean that that style of output interface isn't a possibility to consider, ultimately JSON-LD, like any RDF serialisation should just emit a list of triples when parsed.

If you have some idea of what this hypothethical interface looks like in your head could you perhaps sketch it out as psuedo-code or equivalent so we aren't talking at cross purposes?

[filip26]

My goal for now, is to collect ideas, level of an interest, and requirements, then we can sketch the interface. Having an interface that allows you to pull out quads in a stream-like way is one option. I'm not saying it's a bad idea.

The first question is, do we commit to it? Second, what are the requirements? And third, what should it look like?

[afs]

Retro-fitting interface to existing code runs up against the fact that then there two ways to doing things. i.e. it is a significant impact on the existing system.

Expanding on Rob's point -

Having the parser stream out would be very good - no whole-parse temporary copy.

Could the parser take functions (with generics?) to call to deliver the parts of URIs, literals, bnodes? Then Titanium user can create the RDF terms using native objects and stream as quads/triples? StreamRDF-ish.

It's easier to work in "generic RDF" - a triple is 3 RDF terms, no restrictions on predicate must be URIs.

This may be a part of why CommonsRDF does not get traction. At lower levels of systems, the asymmetric subject/predicate/object is a nuisence.

[hmottestad]

I've made a fork of Titanium where I use generics. I'll try to see if I can post some code to show how this helps with my use in RDF4J.

[hmottestad]

Here are some of the changes I've made to Titanium:

https://github.com/HASMAC-AS/hasmac-json-ld/blob/1adf430c225eda33477c74327c029b56740c3ebb/src/main/java/no/hasmac/jsonld/processor/ToRdfProcessor.java#L72

When converting JSON-LD to RDF I've added generics, a consumer and a value factory.

Here is how I use it from RDF4J:

https://github.com/eclipse-rdf4j/rdf4j/blob/763305f1a636d29a23930804cc0fc64892310cca/core/rio/jsonld/src/main/java/org/eclipse/rdf4j/rio/jsonld/JSONLDParser.java#L166

When writing data from RDF4J to JSON-LD I pass in my own dataset adapter:

https://github.com/eclipse-rdf4j/rdf4j/blob/763305f1a636d29a23930804cc0fc64892310cca/core/rio/jsonld/src/main/java/org/eclipse/rdf4j/rio/jsonld/JSONLDWriter.java#L160

And I've made adapter classes for converting between the RDF4J representation and the Titanium representation of triples/quads/iris/etc.

https://github.com/eclipse-rdf4j/rdf4j/blob/763305f1a636d29a23930804cc0fc64892310cca/core/rio/jsonld/src/main/java/org/eclipse/rdf4j/rio/jsonld/JSONLDWriter.java#L399

These could be optimized a lot when converting back and froth by keeping track of the internal RDF4J object.

[filip26]

Current Titanium API works as a supplier (PULL), and accepts only complete data (PULL).

requirements

• keep the exchange API simple as possible
  • avoid modeling RDF (commons-rdf)
  • use simple data types only, i.e. String, URI?
• PUSH instead of PULL

PUSH: Titanium -> Jena

// Jena calls toRDF
void JsonLd.toRDF(jsonld).options(...).produce(consumer);
   //Titanium writes RDF to the provided consumer inside produce method
  loop [
    // Jena provided code called inside produce method
     consumer.dataset(...);
     consumer.accept(...) 
  ]
// Jena gets back full control and might close the consumer
consumer.end(); // can be dropped, as toRDF's done writing

PUSH: Jena -> Titanium

// Jena calls fromRDF 
<RDFConsumer, Supplier<JsonArray>> consumer = JsonLd.fromRDF().options(...).consumeAndSupply();
// Jena writes RDF to the consumer
loop [
  consumer.dataset(...).accept(...).accept(...).accept(...)  // Flow?
          .dataset(...).accept(...);
]
// Jena closes the consumer
consumer.end();
// Jena gets JSON-LD
JsonArray jsonld = consumer.get(); // could act as end()

API

interface RDFConsumer {
    dataset(name); 
    accept(resource, predicate, value, datatype);
    accept(resource, predicate, value);
    // end can be dropped to keep it minimal 
    end();
}

Just brainstorming, thank you for the comments.

[rvesse]

Yeah I think this looks like a good direction to explore.

For the RDFConsumer interface you suggested using simple types, effectively delaying the choice of when to turn those into a consumers own RDF model to when the consumer receives it? I think that's a reasonable compromise.

Does this mean there's multiple overloads for accept() e.g.

    accept(String resource, URI predicate, URI value);
    accept(URI resource, URI predicate, String value, URI datatype);
    accept(URI resource, URI predicate, URI value);

The main possible confusion I could see there is how you send Blank Nodes. For the subject position it's kinda obvious that String is a blank node and URI a URI, but for the object position you've got the confusion with literals as well.

I think the other possibility that was suggested was something like the following:

interface RDFConsumer<T> {
    // Converters from simple types to the consumers strong type
    T uri(String uri);
    T blankNode(String id);
    T literal(String value, String lang, String datatype);

    // Output
    dataset(T name); 
    accept(T resource, T predicate, T value);
    end();
}

And then your loop becomes something like the following:

loop [
    // Jena provided code called inside produce method
    T graph = ...;
    T subject = consumer.uri(u);
     // etc.
     consumer.dataset(...);
     consumer.accept(...) 
  ]

Makes the logic on Titanium's side slightly more complex but its allowing the consumer to control the target types directly.

For going the other way it gets more complex, so maybe you need T fromUri(String uri) and String toUri(T uri), not sure if I'm overcomplicating at this point?

[hmottestad]

Creating Java URI objects is very expensive, performance wise. I recommend avoiding that at all cost.

Generics is preferable for RDF4J.