The TruenoDB distributed computation server.

Trueno Compute Server Trueno Compute Server relies on Spark Job Server, and GraphX.

1. Page Rank Algorithm.
2. Connected Components.
3. Strongly Connected Components.
4. Triangle Counting.

We designed an algorithm generator. The architecture includes connectivity to a key value store in which the parameters of new algorithms will be stored. The algorithm generator will create the algorithms-release.jar that will be uploaded into the Spark Job Server in order to receive REST requests. The results will be stored in the Database (e.g. Apache Cassandra, Scylla).

Neo4j uses mazarunner for big data graph processing. Mazerunner is an unmanaged extension [1]. Apache Spark’s GraphX module constitutes the main component of Mazerunner. When an agent job is dispatched, a subgraph is exported from Neo4j and written to Apache Hadoop HDFS. After Neo4j exports a subgraph to HDFS, a separate service for Spark is notified to begin processing that data [1].

One focus of this approach is on data safety, that’s why it uses a persistent queue (RabbitMQ) to communicate data between Neo4j and Spark [2].

Since they both depend on Spark GraphX the running time for the provided algorithms is roughly the same. TruenoDB takes a little bit longer when requesting a job to the spark engine (Spark Job Server).

The Neo4j Spark Connector uses the binary Bolt protocol to transfer data from and to a Neo4j server.

Neo4j Spark Connector offers Spark-2.0 APIs for RDD, DataFrame, GraphX and GraphFrames.

Trueno access ElasticSearch Backend via a REST (http) connection. The REST connection incurs on some overhead and latency.

Read vertices (and all its properties).

Neo4j is 3.6 times faster than Trueno reading.

Create vertices. In case of Trueno, the load was reduced since the server could not handle more than 10000 vertices.

Neo4j is comparable with Trueno writing.

Retrieve a vertex, and set/update a property. In case of Trueno, the load was reduced since the server could not handle more than 5000 concurrent request for reads/writes. Also, Trueno needed two operations (calls) to accomplish this test instead of one (Neo4j)

Neo4j is 12 times faster than Trueno reading/writing.

Ask for all the direct neighbors of a vertex. In case of Trueno, the load was reduced since the server could not handle more than 500 concurrent request for neighbors. Also, Trueno needed two operations (calls) to accomplish this test instead of one (Neo4j)

Neo4j is 44 times faster than Trueno finding the direct neighbors of vertices.

This benchmark compares the single read performance of the following configurations:

• Trueno (REST). Current configuration, where data is retrieved from the ElasticSearch backend using a REST connection.
• Trueno (Navite node.js). Trueno access ElasticSearch Backend via a tunelling bridge (socket), which establish a connection with the backend using a native driver (Java).
• Trueno (Native direct). Similar to the previous case, but the test retrieved the data from the ElasticSearch backend directly from the Bridge Server (Native ElasticSearch API), without using the Trueno websocket (basically, we bypassed Trueno). The connection does not incurr on pre-processing and post-processing done by Trueno.
• Neo4j. Neo4j standalone configuration.

Read vertices (and all its properties).

* The test could not be performed using a larger dataset due a timeout on the connection.

Neo4j is 3 times faster than Trueno reading.

This benchmark compares the single read performance of the following configurations:

• Trueno (REST). Current configuration, where data is retrieved from the ElasticSearch backend using a REST connection.
• Trueno (Transport Client using Elasticsearch Native API). Trueno access ElasticSearch Backend via a tunelling bridge (written in Java), which establish a connection with the backend using a native driver (Java). The comunication between the client and the tuneeling bridge is done via web socket (which include less overhead than the traditional socket.io communication that was before). The transport client connects with the Elasticsearch backend via messages (using the Elasticsearch Native API).
• Trueno (Native direct). Similar to the previous case, but the difference is that we use a Native Client instead of the Transport Client. A Native Client requires to instantiate a Elasticsearch node in the backend. Even thougth that this configuration is faster, is not recommended by Elasticsearch because in the long run will translate in more overhead to the backend.
• Neo4j. Neo4j standalone configuration.

Read vertices (and all its properties).

* The test could not be performed using a larger dataset due a timeout on the connection.

Trueno is 2 times faster than Neo4J reading.

In this experiment, we inserted the movies dataset in batches of 300 components until completion. We ran separate inserts(for vertices and edges).

• [1] https://neo4j.com/blog/using-apache-spark-neo4j-big-data-graph-analytics/
• [2] https://neo4j.com/developer/apache-spark/
• [3] https://neo4j.com/blog/neo4j-3-0-apache-spark-connector/