Graph Databases

Description

Apache Hop works with graph databases which speak Cypher or Gremlin. You connect to them with a graph database connection, and use the same transforms and actions for all of them.

Database Language Connection type Plugin More information

Neo4j

Cypher

Neo4j (Bolt)

Neo4j

Neo4j

Memgraph

Cypher

Memgraph (Bolt)

Neo4j

Memgraph

Amazon Neptune

openCypher, Gremlin

Amazon Neptune (Bolt) or Gremlin

Neo4j, Gremlin

Gremlin

FalkorDB

Cypher

FalkorDB (Redis protocol)

FalkorDB

FalkorDB

Apache AGE

Cypher

Apache AGE (PostgreSQL)

Apache AGE

Apache AGE

Apache TinkerPop Gremlin Server, JanusGraph

Gremlin

Gremlin

Gremlin

Gremlin

Requests to Amazon Neptune aren’t signed with AWS Signature Version 4, so IAM database authentication isn’t supported: connect to a Neptune cluster with IAM authentication disabled, over TLS. Neptune doesn’t use the username and password of the connection.

The graph plugins (Neo4j, which holds the transforms and actions as well as the Bolt connection types, FalkorDB, Apache AGE and Gremlin) are optional plugins: install them with the marketplace or use the full client.

Relational databases with SQL/PGQ, like Oracle 23ai, can expose their tables as a property graph with the Create property graph action.

Transforms and actions

The graph database connection page lists which of these work with which database.

Execution logging and information

Hop can keep the history of your executions in a graph database which speaks Cypher:

Vectors and embeddings

Embeddings, for example from the Embed text transform, travel through a pipeline as fields of the Vector value type. The graph transforms store them as a list of numbers:

  • In a graph model, give the property the type Vector. Graph output then writes the field as a vector.

  • In Graph query, pick Vector as the graph data type of a parameter to pass a vector to a statement, for example to a vector search. Give a return value the Hop type Vector to read a stored vector back into a Vector field.

  • A text field holding [0.1,0.2,0.3], as Embed text writes it with output format STRING, is converted too.

  • Generate graph CSVs writes a vector as a float[] column, with ; between the elements.

How each database stores the list:

Database Stored as

Neo4j, Memgraph

A list of floats, which their vector indexes use.

FalkorDB

A vector: Graph output wraps the value in vecf32(), which the FalkorDB vector index needs. In a Graph query statement, write vecf32($param) yourself. Cypher output and Cypher builder don’t add vecf32(): use Graph output or Graph query to write vectors to FalkorDB.

Apache AGE

An agtype array. AGE has no vector index; keep the embeddings in a pgvector table next to the graph when you need vector search.

Amazon Neptune, Gremlin servers

A list, if the database accepts list properties. This is not covered by the integration tests.

Create and drop vector indexes on nodes or relationships with the Graph index action, index type VECTOR, on Neo4j, Memgraph and FalkorDB. Search them with the Graph vector search transform: one output row per node, or relationship, found, with its similarity score and the properties you pick. The score is the same on every database: the cosine similarity, from -1 to 1, for cosine indexes and 1 / (1 + squared euclidean distance) for euclidean ones. Vector search over relationships needs Neo4j 5.18 or later or Memgraph 3; the supportingRelationshipVectorSearch capability of a connection tells whether it has it.

The transform calls the vector search procedure of the database, which you can also call yourself in a Graph query with a Vector parameter, for example to filter or expand the nodes found:

  • Neo4j: CALL db.index.vector.queryNodes('index', 5, $q) YIELD node, score, where the score is between 0 and 1: (1 + cosine similarity) / 2, or 1 / (1 + squared distance).

  • Memgraph: CALL vector_search.search('index', 5, $q) YIELD node, distance, where the distance is 1 - cosine similarity, or the squared distance: lower is nearer. Memgraph’s similarity column is the absolute value of 1 - distance, which ranks opposite vectors as near ones.

  • FalkorDB: CALL db.idx.vector.queryNodes('Label', 'property', 5, vecf32($q)) YIELD node, score, where the score is a distance, 1 - cosine similarity or the euclidean distance: lower is nearer.

And for relationships:

  • Neo4j: CALL db.index.vector.queryRelationships('index', 5, $q) YIELD relationship, score

  • Memgraph: CALL vector_search.search_edges('index', 5, $q) YIELD edge, distance

  • FalkorDB: CALL db.idx.vector.queryRelationships('TYPE', 'property', 5, vecf32($q)) YIELD relationship, score, again with a distance.

Upgrading from Hop 2.20 or earlier

The graph functionality of Hop used to be Neo4j specific. From Hop 2.21 on it works with every graph database above. Existing projects keep working without changes:

  • Neo4j connections keep working. You can convert them to graph database connections, see Converting a Neo4j connection.

  • The transforms and actions have vendor neutral names, icons and the category Graph. Their plugin IDs didn’t change, so pipelines and workflows open as before:

    Before Now

    Neo4j Cypher

    Graph query

    Neo4j graph output

    Graph output

    Neo4j output

    Cypher output

    Neo4j Cypher builder

    Cypher builder

    Neo4j split graph

    Split graph

    Neo4j generate CSVs

    Generate graph CSVs

    Get Neo4j logging info

    Get graph logging info

    Neo4j Cypher script

    Graph script

    Neo4j index

    Graph index

    Neo4j constraint

    Graph constraint

    Check Neo4j connections

    Check graph database connections

    Neo4j perspective

    Graph logging perspective

    Neo4j location

    Graph database location

  • HOP_GRAPH_LOGGING_CONNECTION replaces NEO4J_LOGGING_CONNECTION, which keeps working when the new variable isn’t set.

  • The Cypher builder writes parameters as $name. It wrote them as {name}, which Neo4j 4 and later reject, so statements without UNWIND work again.

  • Get graph logging info finds the previous executions of pipelines and workflows: it looked for execution types which the logging doesn’t write. The workflow logging now writes the status of a workflow, which these lookups need.

  • The graph index, graph constraint, graph script and check connections actions report their own success, also after a failure hop.

  • Graph output validates against the graph model on every database: the use of the model always, its indexes and unique constraints when the database can list them.