Embedded Janus Graph and Spark for Loading

[dasari2828]

Hello All,
I am trying to load data using Spark and Embedded Graph Traversal and I was able to load some 1.5 million records in under 25 seconds. But as my load goes up to 20 million, it considerably becomes slower and I am tweaking lot of configurations within the code to allow me to commit 500 vertices at a time, and then open up a new graph instance for every 10,000 vertices that are loaded. That takes 23 minutes. This is slower than a simple Janus client, which took 8.5 minutes to load 20 million rows. I have some 8 billion vertices to be loaded. FYI.

With embedded Janus Graph and spark, for loading 20 million vertices, i am facing the below issues.
I am running the Janus server on m5n.24xlarge, 3 Scylla DB instances on i3.8xlarge and m5.xlarge, 11 node EMR Cluster.

java.util.concurrent.CompletionException: java.lang.IllegalStateException: Connection to server is no longer active
at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
at org.apache.tinkerpop.gremlin.driver.ResultSet.one(ResultSet.java:119)
at org.apache.tinkerpop.gremlin.driver.ResultSet$1.hasNext(ResultSet.java:171)

Is there anyone who has tried loading this large dataset? I read from the Janus Graph Documentation that it can support upto quintillion vertices etc.

[vtslab]

Hi,
Is this the same experiment as in https://groups.google.com/g/gremlin-users/c/5ohW4pYRwfM ? Is the current discussion #2605 still relevant?
Best wishes, Marc

[dasari2828]

2605 is done. I'm able to load it. That can be closed

…

On Sat, May 15, 2021, 6:18 AM vtslab ***@***.***> wrote: Hi, Is this the same experiment as in https://groups.google.com/g/gremlin-users/c/5ohW4pYRwfM ? Is the current discussion #2605 <#2605> still relevant? Best wishes, Marc — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2605 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGIR4FYCPAXYB2VV4DOCJ3TNZDANANCNFSM44KMIH5A> .