Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding guide for Import from database #47

Merged
merged 33 commits into from
Dec 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
9f2fff3
New Browser preview guide
Oct 10, 2024
1e35685
Added Import from database guide
Oct 23, 2024
e0f6310
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
6bab53e
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
d7db179
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
b9d4c20
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
f8cf2c9
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
196d2a0
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
a68f1d6
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
658e5b4
Update documentation/import-from-rdbms.adoc
gregoryking Oct 25, 2024
c47cad7
Update documentation/import-from-rdbms.adoc
gregoryking Oct 25, 2024
1e21d58
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
e5e83a7
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
b84b1c9
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
304921e
Update documentation/import-from-rdbms.adoc
gregoryking Oct 25, 2024
00cceda
Update documentation/import-from-rdbms.adoc
gregoryking Oct 25, 2024
5010a20
Update documentation/import-from-rdbms.adoc
gregoryking Oct 25, 2024
701b771
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
5b3b182
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
f886c99
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
7d65613
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
8405b5b
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
849ef58
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
1dbbb98
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
048d8dd
Update documentation/browser-preview.workspace.adoc
gregoryking Oct 25, 2024
eb899fd
Update documentation/import-from-rdbms.adoc
gregoryking Oct 25, 2024
97d0555
Update documentation/import-from-rdbms.adoc
gregoryking Oct 25, 2024
3332b36
Update documentation/import-from-rdbms.adoc
gregoryking Oct 25, 2024
79fd317
Update documentation/import-from-rdbms.adoc
gregoryking Oct 25, 2024
a3967a3
Update documentation/import-from-rdbms.adoc
gregoryking Oct 25, 2024
06eb40d
Update documentation/import-from-rdbms.adoc
gregoryking Oct 25, 2024
2fb94bd
Update documentation/import-from-rdbms.adoc
gregoryking Oct 25, 2024
28a4eb1
Apply suggestions from code review
gregoryking Oct 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 116 additions & 0 deletions documentation/browser-preview.workspace.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
= New Browser Preview
//:images: img/browser-preview
// http://localhost:4000/browser-preview.workspace.json

== Introduction

Welcome to the New Browser Preview!
Browser's querying functionality has been modernized and integrated into Neo4j Aura, and these benefits are now available in the new Browser, bundled with the Neo4j on-prem distribution.

If you use Neo4j Aura you’ll see the same capabilities as this New Browser Preview offered under the **Query** part of the integrated experience.
Query and Browser are similar, but Query provides features you might expect from a cloud offering like user content being backed by cloud storage so it’s available wherever you log in.
For the standalone Browser, content continues to be backed by LocalStorage and therefore tied to the browser you use to access the application.

With that context, read on to learn more about the big new updates in this preview:

- New Cypher editor
- Richer table visualization
- More scalable graph visualization
- New history and quick search
- Improved saved Cypher experience
- Revised parameters

== New Cypher editor

One of the biggest improvements is the new editor which is now built from the same source as the Cypher language.
The Cypher language is ever-evolving and the new editor helps you stay up-to-date with the latest language features.
Here are a few ways the new editor helps you do that:

Syntax highlighting:: Richer syntax highlighting identifies potential errors from things like property values being misinterpreted as Cypher keywords and differentiates more tokens for ease of reading.

image::cypher-editor-syntax-highlighting.png[]


Auto completions:: The underpinnings of the Cypher editor now have an improved understanding of valid constructs, meaning it will, for example, more accurately complete the next valid keyword.

image::cypher-editor-auto-completions.gif[]

Inline help:: When using a procedure or making a function call, helpful descriptions and signatures are presented, allowing you to better understand what’s expected as an input or what might be returned.

image::cypher-editor-inline-help.png[]

Cypher linting:: One of the most useful new features is Cypher linting, which lets you see useful warnings and errors _before_ running a query.
This helps shorten the feedback loop, so you don’t have to run a query to spot syntactic or semantic errors.
This is especially useful when editing larger queries or using new language constructs where you might not know what to expect.

image::cypher-editor-linting.png[]

Cypher reference:: As a complement to the in-editor help, a searchable Cypher reference is now available in the sidebar.
This is useful to check the syntax of a Cypher construct or want a quick reference of which temporal functions exist and how to use them, for example.
The reference also provides links to the Cypher manual for more in-depth explanations and examples.

image::cypher-reference.gif[]

== Richer table visualization

The table visualization now employs reactive result fetching by default.
This means that the number of rows returned by a query is automatically limited, regardless of any `LIMIT` statements you apply.
This is useful for both new and experienced users since an unintentional omission of a `LIMIT` statement can lead to an unexpectedly large amount of results returned, which could previously have overwhelmed Browser.
The table now also employs virtualized rendering meaning that even if you do return a larger number of rows, only those in view are rendered to keep the application more responsive.

There are functional improvements too.
A new render for paths, relationships, and nodes allows you to more readily discern labels/types and key/value pairs.
By default, the render is compact, but optionally it allows graph objects to be pretty-printed for situations where readability is more important than compactness.

Simple result filtering is now also available, allowing you to perform simple text matches on returned results.
It's often simpler and quicker than modifying a Cypher statement to contain a WHERE clause.

image::table-visualization.gif[]


== More scalable graph visualization

The graph visualization has been rebuilt using a new engine while maintaining the familiar layout algorithm from the outgoing graph visualization.
This makes the visualization more responsive and allows for a larger number of nodes and relationships to be visualized.

Another important functional change is that the visualization now only displays items you explicitly return in Cypher statements - it no longer automatically connects returned nodes with relationships that exist between them.
This can still be achieved on-demand via a context menu in the visualization to “show all relationships”.

image::scalable-graph-visualization.gif[]


== New history and quick search

The familiar and convenient up/down arrow history feature remains but has been supplemented with a dedicated sidebar to review and manage history, allowing history to be selectively deleted or exported.

For convenience, the history available in the sidebar is also searchable via a fuzzy search from the main editor, allowing you to quickly find a query in your history.
This can be invoked from the […] more menu on the main editor or via Ctrl+R.

image::history-quick-search.gif[]


== Improved saved Cypher experience

The “Favorites” experience from Browser is now “Saved Cypher” and provides a refreshed way of creating and managing Cypher.
You can now more readily re-order items among nested folders, and Cypher statements saved from the editors recall the last folders used for convenience.

image::saved-cypher.gif[]

== Revised paramaters

The parameters implementation has rationalized the `:param` and `:params` variants that differed in Browser.
While singular and plural versions of the command are respected, they behave the same way and offer support for setting parameters singularly via the arrow syntax `:param a => 1` or multiply via Cypher maps `:param {a:1, b: 2}`.

== And finally...

Hope you enjoy using the new Browser preview.
Please note that the following are not yet supported:

- `:server [connect | disconnect]` commands and frames. For the time being the connection header at the top of the page is used to connect to instances
- `:sysinfo` command to review details of your dbms cluster
- `:play` commands to play builtin and custom guides
- `:server user [add | list]` commands to support user management - this is possible via the cypher surface, see docs for more details.
- GraSS and the `:style` command to set custom styles. Some basic styling support is provided via the UI to change colors, captions and size as well as ordering the priority of styles when multiple labels apply to a node.

This new Browser preview will ultimately replace the existing Browser as the default and later, only experience.
If these items or anything else is important to you, please head over to link:https://feedback.neo4j.com/query[] and drop your feedback there.
Binary file added documentation/cypher-editor-auto-completions.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/cypher-editor-inline-help.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/cypher-editor-linting.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/cypher-reference.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/history-quick-search.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/img/import-add-new-data-source.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/img/import-graph-models.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/img/import-import-jobs.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/img/import-iterate-graph-model.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/img/import-source-config-form.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/img/import-source-table-schema.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/img/import-tables-mapped.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
151 changes: 151 additions & 0 deletions documentation/import-from-rdbms.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
= Import from Relational Database
:imagesdir: https://neo4j-graph-examples.github.io/get-started/documentation/img/

[role=NX_TAB_NAV,tab=import]

== Introduction

The import service now offers a way to import directly from data sources such as relational databases.
If your data is already in reasonable shape for use in a graph database, it is possible to connect directly to your source and configure an import job.

This guide will take you through a three-stage process:

. Configuring a data source
. Reviewing a generated graph model and mapping to the data source
. Running the import job


=== Connectivity requirements

- Your database should be one of the supported types, including Postgres, MySQL, SQL Server and Oracle
- It should be accessible via a public IP address
- You should have username / password authentication setup. We recommend using a read-only user.

=== Limitations

The following are not yet supported:

- Advanced configuration options such as custom certificates for data sources
- Imports from data sources in a Virtual Private Cloud (VPC) into a Neo4j managed VPC


Start by setting up a connection to a relational database and complete your first import.


== Configuring a data source

[role=NX_TAB_NAV,tab=import]

If you haven't already, navigate to the `Data Serices > Import` menu item to access the import service.

Once in the Import service, the first step is to configure a new data source.
Two categories of data sources are available:

- Direct connection -- you provide the import service with access to your data source, including connection URL, database name, and credentials.
- Proivision of CSV files -- you provide CSV files from your filesystem as the data source.
A separate guide, _Import from CSV_, is available to take you through this process.

This guide focuses on connecting directly to a data source.
The "New data source" button on the Data Sources tab, presents you with a number of data source options.
In this guide, you will connect to PostgreSQL.


Complete the form with the required information for your data source.
If there are any problems connecting to your data source, you stay on the connection form and have the opportunity to tweak the details before continuing.

image::import-source-config-form.png[]

[TIP]
====
Currently, error reporting is limited, so you should validate that all your details are correct by successfully connecting from your development machine.
Also, note that your database needs to be publicly accessible to connect to it from Neo4j Aura.
====

Once the connection is successful, you are prompted to review and confirm the table schema for the database you've selected to connect to.
Here you can see a summary of the tables and columns in your database to help validate that you're connected to the right data source.
If everything looks good, hit `Confirm`.
This adds your data source and saves it for future re-use.

image::import-source-table-schema.png[]

[TIP]
====
While your connection details are retained, your password is not.
You need to re-provide the password for your data source when you use it again or trigger the import job.
====

Once you've added a data source, you can either start generating a candidate graph from it, or hit `Later` to come back and do this later.
If you come back later, use the [...] button on a row and the `New graph model from data source` option to start the graph generation flow.
In this guide, let's move immediately onto the next stage of generating a graph model.


== Generating a graph model


Use the `Generate graph model` button once you have added a data source.
Doing this inspects any constraints that exist in your database, looking for primary key constraints to create nodes with IDs and foreign keys to generate relationships between them.
If suitable constraints are found, a corresponding candidate graph is generated, and the tables and columns are mapped to it.
If your dataset doesn't have any such constraints (or only part of the schema does), you have to map your tables to nodes and relationships manually.
If you're not familiar with your data source, it may take some time to understand the different entities and connections present in your tables.

If no constraints are found *at all* in your source database, everything is mapped to a node for your convenience.
This saves you from doing some manual mapping, but note that a graph with just nodes doesn't make for a connected graph!
From here, delete nodes that should be relationships and recreate relationships between the remaining nodes as appropriate for your data.


[TIP]
=====
The generation and mapping doesn't currently support composite primary or foregin keys, so you won't see mappings related to these, nor be able to import them easily via the import UI just yet.
=====

=== Refining a model

Even in this example where a well-connected graph model has been generated, the model can benefit from some tidy-up.
For example, tables mapped to nodes use the corresponding table name as the label, but this can be made more suitable for your graph.
The same applies to relationships that by default use a concatenation of related table names.
Relationship direction may also benefit from being reversed for the chosen semantics.
See the example below where the concepts of Person, Order, and Employee and their corresponding relationships are tidied up.

image::import-iterate-graph-model.gif[]

In the Data source panel on the left-hand side, blue icons denote that the column from a particular table has been mapped to the graph model.
Reviewing the list of tables to any that have not been mapped can be useful.

image::import-tables-mapped.png[]


If the data is in the right shape, it can be mapped manually to the graph model.
You can learn more about how nodes and relationships are mapped in the _Import from CSV_ guide that covers the same concepts required here.
It explains node IDs and relationship mappings in more detail with CSV files instead of tables.

When all parts of the graph model (nodes and relationships) are successfully mapped to tables, they are annotated with a green check mark.
If an import is attempted and anything is missing, it is highlighted in red to be fixed before proceeding with the import.

== Running an import job

Once you're happy with your graph model and how your tables are mapped to it, you can runthe import job.
If you're already connected to an instance, this instance is used.
Otherwise, you are prompted to select an instance.

When submitting the job, you are prompted for two sets of credentials:

- The data source password (as used when setting up the data source)
- The destination neo4j instance password. These were provided when you created the instance and optionally made available to download in a `.txt` file of the format `Neo4j-<instance_id>-Created-<date>.txt`

Once the job has been successfully submitted, you are redirected to the import jobs page, which tracks the current status of the import job.

image::import-import-jobs.png[]

[TIP]
====
The high-level status of the import job is refreshed periodically but does not currently show progress.
If you want to see live updates of the nodes and relationships being imported into the graph, you can switch to the Query app, monitor the Database Information sidebar, and refresh it to see counts increment.

image::import-query-db-info-refresh.gif[]

====

If you want to return to a model, you can find it stored in the Graph models tab.
From here you can open a model and rerun an import or adapt the model as required before rerunning.

image::import-graph-models.png[]
Binary file added documentation/saved-cypher.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/scalable-graph-visualization.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added documentation/table-visualization.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.