How reliable is this comparison?
In this blog, the LDBC Social Network Benchmark test result will be assessed between AgensGraph(v1.0) vs. Neo4j(v3.1). The benchmark results will vary depending on software, testing environment, time of testing, and numerous other conditions. This benchmark is simply a snapshot in time. For fair comparison, we tried to fully optimize their application performance. However, they may have a different approach to optimizing their program. Also, Platform Lab was used for their testing. Again, when it comes to performance comparisons, there really isn’t a “winner takes all.” There are just too many factors to take into consideration and it is extremely difficult to compare performance on an exact criteria.
What is the Linked Data Benchmark Council (LDBC)?
The LDBC is a consortium comprised of industry leading companies and organizations in the DBMS industry. Their objective is to “establish an industry-neutral entity for developing graph and RDF benchmarks, auditing benchmark results, and publishing audited results.” (1)
For this report, the LDBC Social Network Benchmark (SNB) - v0.2.4 criteria was used. The social network benchmark was used because it fits “characteristics of graph-shaped data.” (2)
Queries used for testing include the following: 14 Complex Queries, 7 Short Queries, 8 Update Queries.
Neo4j is a leading vendor in the Graph Database Industry.
AgensGraph is a multi-model database management system developed by Bitnine. The user can query both Cypher and SQL simultaneously. AgensGraph is based on PostgreSQL.
Hardware/Software used for testing:
- AgensGraph v1.0; Neo4j v3.1
- Database Size: AgensGraph (66GB, Index: 51GB); Neo4j (45GB [data + indexes])
- Cache Condition: AgensGraph (Loading all objects as cache with the pg_prewarm); Neo4j (performing of warm-up [10,000])
- Architecture: x86_64
- Processors: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, 40 cores
- Memory: 198GB
- Operating system Ubuntu 14.05 LTS
- Total disks capacity: 2.7TB
AgensGraph outperforms in the performance test in all three query categories – complex, short, update. Based on this sample size, on average, AgensGraph is approximately 50 times faster in complex queries, 20 times faster in short queries, and 10 times faster in update queries. Some of the reasons AgensGraph is faster are due to the following:
- Decreased number of join operation in making the edges of one-to-N relationship denormalized on a vertex.
- Use of multi-version concurrency control (MVCC), this creates no conflict between read and write operations.
Complex Query #4 (AgensGraph Best Performance):
Query Description: Given a start Person, find Tags that are attached to Posts that were created by that Person’s friends. Only include Tags that were attached to friends’ Posts created within a given time interval, and that were never attached to friends’ Posts created before this interval. (2)
AgensGraph’s relative best performance is Query 4. AgensGraph is 1,800 times faster in this query. It takes AgensGraph 0.17 seconds vs. 307 seconds. The reason is because AgensGraph can perform the query by calculating edge weight first. Above is the query description and result.
Complex Query #13 (AgensGraph Worst Performance):
Query Description: Given two Persons, find the shortest path between these two Persons in the subgraph induced by the Knows relationships. Return the length of this path. (2)
AgensGraph’s relative worst performance is query # 13. Neo4j is faster than AgensGraph in the result of shortest path search and variable length relationship search. The reason is that AgensGraph uses Common Table Expression (CTE) to perform the two of queries and this buffers the interim result. This weakness will be improved in the next released version. AgensGraph performs at 0.85 seconds while Neo4j performs at 0.05 seconds. Neo4j is 16 times faster.
Overall, based on this sample, AgensGraph is faster. However, it is very difficult to get the exact environment to compare. Having said that, AgensGraph is approximately 50 times faster in complex queries, 20 times faster in short queries, and 10 times faster in update queries. Neo4j is faster in complex query 13 and the AgensGraph Engineering team is working to improve this for the next release. All relevant source codes can be found at the Github links in the reference section below.
1.) LDBC. Retrieved 5/01/2017. http://ldbcouncil.org/industry/origins
2.) LDBC. LDBC Social Network Benchmark (SNB) - v0.2.4.
Retrieved 5/01/2017. http://ldbc.github.io/ldbc_snb_docs/snb_doc.pdf