Friday, February 17, 2017

Graph databases and rapid prototyping (part 3)

In this last part, I am comparing the development experiences when using SQL Server and Neo4j graph database. There is also an overview of Neo4j tools and functionality, and how they accelerate web development and prototyping.

Part 1: How I got hooked on graph databases
Part 2: What is a graph database
Part 3: Rapid prototyping
"Java and Javascript are similar like Car and Carpet are similar" - fun fact
Now when I think about it, a graph database feels like a dynamically-typed language, and a SQL database - like a statically-typed one. As you would consider a dynamically-typed language, such as JavaScript, for the web development and prototyping, a graph database is also a good option for these cases.

"If all you have is a hammer, everything looks like a nail" - the law of the instrument
We have successfully used SQL Server on many projects, and it was our default database choice. For our recent web application, it required a good amount of design and plumbing before we can expose data through Web API. As requirements refined, our initial design underwent many changes. Even with good tools and automated steps, it was painful and time-consuming making changes to the data schema, and, despite 2 abstraction layers, the app was often broken afterwards.

"Go there, don't know where" - Russian folktale
When requirements are unclear, and you do not know what's feasible or even make sense, one approach is to quickly experiment and build prototypes to gather meaningful feedback. I accomplished just that with JavaScript and a graph database.

Neo4j and OrientDB both provide REST API for browsing graphs and running queries. The default representation is JSON, so it's very easy to work with in JavaScript. For the prototype, I was calling this API directly from my web page 1, and the whole development experience was like editing scripts and hitting refresh in the browser. Note that the development environment for Neo4j and OrientDB is browser-based, and it allows running and profiling queries, browsing graphs, and visualizing query results as a graph or a table.

It took me minutes to add new properties, from loading data 2 to displaying it in the user interface. Integrating a new data source into my app took just a few hours, end-to-end. A tremendous help here was D3.js, a fantastic library that can fetch data from API and bind it to SVG or DOM elements, creating dynamic user interfaces.

Even if you have a requirement to use a SQL database, time saved by prototyping with a graph database may justify future migration. As an added benefit, you can start fresh with a clean data schema, and leave all those scars and technical debt behind.

A typical concern regarding graph databases is the aggregation performance, so I was considering a hybrid approach (a graph for traversals and a SQL tables for reports). In my case, the performance of the Neo4j graph database was good, and it's keeping up with the demand so far with little tweaks to the data model.

As a closing remark; a graph database is an easy to use and powerful tool, and having it in your toolbox will broaden your perspective and increase your productivity.

For production use, you will need an additional layer for authentication and authorization.
Neo4j provides powerful features to load text files (e.g. comma-separated), and merge them with the existing data.

No comments:

Post a Comment