jupyter/NodeEmbeddingsJava.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2f0eabc4",
   "metadata": {},
   "source": [
    "# Node Embeddings\n",
    "\n",
    "This notebook demonstrates different methods for node embeddings and how to further reduce their dimensionality to be able to visualize them in a 2D plot. \n",
    "\n",
    "Node embeddings are essentially an array of floating point numbers (length = embedding dimension) that can be used as \"features\" in machine learning. These numbers approximate the relationship and similarity information of each node and can also be seen as a way to encode the topology of the graph.\n",
    "\n",
    "## Considerations\n",
    "\n",
    "Due to dimensionality reduction some information gets lost, especially when visualizing node embeddings in two dimensions. Nevertheless, it helps to get an intuition on what node embeddings are and how much of the similarity and neighborhood information is retained. The latter can be observed by how well nodes of the same color and therefore same community are placed together and how much bigger nodes with a high centrality score influence them. \n",
    "\n",
    "If the visualization doesn't show a somehow clear separation between the communities (colors) here are some ideas for tuning: \n",
    "- Clean the data, e.g. filter out very few nodes with extremely high degree that aren't actually that important\n",
    "- Try directed vs. undirected projections\n",
    "- Tune the embedding algorithm, e.g. use a higher dimensionality\n",
    "- Tune t-SNE that is used to reduce the node embeddings dimension to two dimensions for visualization. \n",
    "\n",
    "It could also be the case that the node embeddings are good enough and well suited the way they are despite their visualization for the down stream task like node classification or link prediction. In that case it makes sense to see how the whole pipeline performs before tuning the node embeddings in detail. \n",
    "\n",
    "## Note about data dependencies\n",
    "\n",
    "PageRank centrality and Leiden community are also fetched from the Graph and need to be calculated first.\n",
    "This makes it easier to see if the embeddings approximate the structural information of the graph in the plot.\n",
    "If these properties are missing you will only see black dots all of the same size.\n",
    "\n",
    "<br>  \n",
    "\n",
    "### References\n",
    "- [jqassistant](https://jqassistant.org)\n",
    "- [Neo4j Python Driver](https://neo4j.com/docs/api/python-driver/current)\n",
    "- [Tutorial: Applied Graph Embeddings](https://neo4j.com/developer/graph-data-science/applied-graph-embeddings)\n",
    "- [Visualizing the embeddings in 2D](https://github.com/openai/openai-cookbook/blob/main/examples/Visualizing_embeddings_in_2D.ipynb)\n",
    "- [scikit-learn TSNE](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html#sklearn.manifold.TSNE)\n",
    "- [AttributeError: 'list' object has no attribute 'shape'](https://bobbyhadz.com/blog/python-attributeerror-list-object-has-no-attribute-shape)\n",
    "- [Fast Random Projection (neo4j)](https://neo4j.com/docs/graph-data-science/current/machine-learning/node-embeddings/fastrp)\n",
    "- [HashGNN (neo4j)](https://neo4j.com/docs/graph-data-science/2.6/machine-learning/node-embeddings/hashgnn)\n",
    "- [node2vec (neo4j)](https://neo4j.com/docs/graph-data-science/current/machine-learning/node-embeddings/node2vec) computes a vector representation of a node based on second order random walks in the graph. \n",
    "- [Complete guide to understanding Node2Vec algorithm](https://towardsdatascience.com/complete-guide-to-understanding-node2vec-algorithm-4e9a35e5d147)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4191f259",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from IPython.display import display\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plot\n",
    "import typing as typ\n",
    "import numpy as np\n",
    "from openTSNE.sklearn import TSNE\n",
    "from neo4j import GraphDatabase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8ef41ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openTSNE import __version__ as openTSNE_version\n",
    "print('The openTSNE version is: {}'.format(openTSNE_version))\n",
    "print('The pandas version is: {}'.format(pd.__version__))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c5dab37",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Please set the environment variable \"NEO4J_INITIAL_PASSWORD\" in your shell \n",
    "# before starting jupyter notebook to provide the password for the user \"neo4j\". \n",
    "# It is not recommended to hardcode the password into jupyter notebook for security reasons.\n",
    "\n",
    "driver = GraphDatabase.driver(uri=\"bolt://localhost:7687\", auth=(\"neo4j\", os.environ.get(\"NEO4J_INITIAL_PASSWORD\")))\n",
    "driver.verify_connectivity()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1db254b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cypher_query_from_file(filename):\n",
    "    with open(filename) as file:\n",
    "        return ' '.join(file.readlines())\n",
    "    \n",
    "\n",
    "def query_cypher_to_data_frame(filename, parameters_: typ.Optional[typ.Dict[str, typ.Any]] = None):\n",
    "    records, summary, keys = driver.execute_query(get_cypher_query_from_file(filename),parameters_=parameters_)\n",
    "    return pd.DataFrame([r.values() for r in records], columns=keys)\n",
    "\n",
    "\n",
    "def query_first_non_empty_cypher_to_data_frame(*filenames : str, parameters: typ.Optional[typ.Dict[str, typ.Any]] = None):\n",
    "    \"\"\"\n",
    "    Executes the Cypher queries of the given files and returns the first result that is not empty.\n",
    "    If all given file names result in empty results, the last (empty) result will be returned.\n",
    "    By additionally specifying \"limit=\" the \"LIMIT\" keyword will appended to query so that only the first results get returned.\n",
    "    \"\"\"\n",
    "    result=pd.DataFrame()\n",
    "    for filename in filenames:\n",
    "        result=query_cypher_to_data_frame(filename, parameters)\n",
    "        if not result.empty:\n",
    "            print(\"The results have been provided by the query filename: \" + filename)\n",
    "            return result\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d2e62d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO option to choose between directed and undirected projection\n",
    "\n",
    "def create_undirected_projection(parameters: dict) -> bool: \n",
    "    \"\"\"\n",
    "    Creates an undirected homogenous in-memory Graph projection for/with Neo4j Graph Data Science Plugin.\n",
    "    It returns True if there is data available for the given parameter and False otherwise.\n",
    "    Parameters\n",
    "    ----------\n",
    "    dependencies_projection : str\n",
    "        The name prefix for the in-memory projection for dependencies. Example: \"java-package-embeddings-notebook\"\n",
    "    dependencies_projection_node : str\n",
    "        The label of the nodes that will be used for the projection. Example: \"Package\"\n",
    "    dependencies_projection_weight_property : str\n",
    "        The name of the node property that contains the dependency weight. Example: \"weight25PercentInterfaces\"\n",
    "    dependencies_projection_embedding_dimension : str\n",
    "        The number of the dimensions and therefore size of the resulting array of floating point numbers\n",
    "    \"\"\"\n",
    "    \n",
    "    is_data_missing=query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_0_Check_Projectable.cypher\", parameters).empty\n",
    "    if is_data_missing: return False\n",
    "\n",
    "    query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_1_Delete_Projection.cypher\", parameters)\n",
    "    query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_2_Delete_Subgraph.cypher\", parameters)\n",
    "    # To include the direction of the relationships use the following line to create the projection:\n",
    "    # query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_3_Create_Projection.cypher\", parameters)\n",
    "    query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_4_Create_Undirected_Projection.cypher\", parameters)\n",
    "    query_cypher_to_data_frame(\"../cypher/Dependencies_Projection/Dependencies_5_Create_Subgraph.cypher\", parameters)\n",
    "    return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1aa86093",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature ideas\n",
    "# TODO option to choose between directed and undirected projection\n",
    "# TODO option to not read already existing node embeddings to experiment with different (hpyer) parameters\n",
    "# TODO run a community detection algorithm co-located in here when \"communityId\" is missing\n",
    "# TODO run a centrality algorithm co-located in here when \"centrality\" score is missing\n",
    "\n",
    "def create_node_embeddings(cypher_file_name: str, parameters: dict) -> pd.DataFrame: \n",
    "    \"\"\"\n",
    "    Creates an in-memory Graph projection by calling \"create_undirected_projection\", \n",
    "    runs the cypher Query given as cypherFileName parameter to calculate and stream the node embeddings\n",
    "    and returns a DataFrame with the results.\n",
    "    \n",
    "    cypher_file_name\n",
    "    ----------\n",
    "    Name of the file containing the Cypher query that executes node embeddings procedure.\n",
    "\n",
    "    parameters\n",
    "    ----------\n",
    "    dependencies_projection : str\n",
    "        The name prefix for the in-memory projection for dependencies. Example: \"java-package-embeddings-notebook\"\n",
    "    dependencies_projection_node : str\n",
    "        The label of the nodes that will be used for the projection. Example: \"Package\"\n",
    "    dependencies_projection_weight_property : str\n",
    "        The name of the node property that contains the dependency weight. Example: \"weight25PercentInterfaces\"\n",
    "    dependencies_projection_embedding_dimension : str\n",
    "        The number of the dimensions and therefore size of the resulting array of floating point numbers\n",
    "    \"\"\"\n",
    "    \n",
    "    is_data_available=create_undirected_projection(parameters)\n",
    "    \n",
    "    if not is_data_available:\n",
    "        print(\"No projected data for node embeddings calculation available\")\n",
    "        empty_result = pd.DataFrame(columns=[\"codeUnitName\", 'projectName', 'communityId', 'centrality', 'embedding'])\n",
    "        return empty_result\n",
    "\n",
    "    existing_embeddings_query_filename=\"../cypher/Node_Embeddings/Node_Embeddings_0a_Query_Calculated.cypher\"\n",
    "    embeddings = query_first_non_empty_cypher_to_data_frame(existing_embeddings_query_filename, cypher_file_name, parameters=parameters)\n",
    "    display(embeddings.head()) # Display the first entries of the table\n",
    "    return embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6ec6a9b",
   "metadata": {},
   "source": [
    "### Dimensionality reduction with t-distributed stochastic neighbor embedding (t-SNE)\n",
    "\n",
    "The following function takes the original node embeddings with a higher dimensionality, e.g. 64 floating point numbers, and reduces them into a two dimensional array for visualization. \n",
    "\n",
    "> It converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data.\n",
    "\n",
    "(see https://opentsne.readthedocs.io)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "720aebd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_node_embeddings_for_2d_visualization(embeddings: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"\n",
    "    Reduces the dimensionality of the node embeddings (e.g. 64 floating point numbers in an array)\n",
    "    to two dimensions for 2D visualization.\n",
    "    see https://opentsne.readthedocs.io\n",
    "    \"\"\"\n",
    "\n",
    "    if embeddings.empty: \n",
    "        print(\"No projected data for node embeddings dimensionality reduction available\")\n",
    "        return embeddings\n",
    "    \n",
    "    # Calling the fit_transform method just with a list doesn't seem to work (anymore?). \n",
    "    # It leads to an error with the following message: 'list' object has no attribute 'shape'\n",
    "    # This can be solved by converting the list to a numpy array using np.array(..).\n",
    "    # See https://bobbyhadz.com/blog/python-attributeerror-list-object-has-no-attribute-shape\n",
    "    embeddings_as_numpy_array = np.array(embeddings.embedding.to_list())\n",
    "\n",
    "    # Use t-distributed stochastic neighbor embedding (t-SNE) to reduce the dimensionality \n",
    "    # of the previously calculated node embeddings to 2 dimensions for visualization\n",
    "    t_distributed_stochastic_neighbor_embedding = TSNE(n_components=2, verbose=1, random_state=47)\n",
    "    two_dimension_node_embeddings = t_distributed_stochastic_neighbor_embedding.fit_transform(embeddings_as_numpy_array)\n",
    "    display(two_dimension_node_embeddings.shape) # Display the shape of the t-SNE result\n",
    "\n",
    "    # Create a new DataFrame with the results of the 2 dimensional node embeddings\n",
    "    # and the code unit and artifact name of the query above as preparation for the plot\n",
    "    node_embeddings_for_visualization = pd.DataFrame(data = {\n",
    "        \"codeUnit\": embeddings.codeUnitName,\n",
    "        \"artifact\": embeddings.projectName,\n",
    "        \"communityId\": embeddings.communityId,\n",
    "        \"centrality\": embeddings.centrality,\n",
    "        \"x\": [value[0] for value in two_dimension_node_embeddings],\n",
    "        \"y\": [value[1] for value in two_dimension_node_embeddings]\n",
    "    })\n",
    "    display(node_embeddings_for_visualization.head()) # Display the first line of the results\n",
    "    return node_embeddings_for_visualization\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d937e26e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_2d_node_embeddings(node_embeddings_for_visualization: pd.DataFrame, title: str):\n",
    "    if embeddings.empty:\n",
    "        print(\"No projected data to plot available\")\n",
    "        return\n",
    "\n",
    "    plot.scatter(\n",
    "        x=node_embeddings_for_visualization.x,\n",
    "        y=node_embeddings_for_visualization.y,\n",
    "        s=node_embeddings_for_visualization.centrality * 300,\n",
    "        c=node_embeddings_for_visualization.communityId,\n",
    "        cmap=main_color_map,\n",
    "    )\n",
    "    plot.title(title)\n",
    "    plot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da9e8edb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#The following cell uses the build-in %html \"magic\" to override the CSS style for tables to a much smaller size.\n",
    "#This is especially needed for PDF export of tables with multiple columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9deaabce",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "<style>\n",
    "/* CSS style for smaller dataframe tables. */\n",
    ".dataframe th {\n",
    "    font-size: 8px;\n",
    "}\n",
    ".dataframe td {\n",
    "    font-size: 8px;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2496caf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Main Colormap\n",
    "main_color_map = 'nipy_spectral'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c68aa20",
   "metadata": {},
   "source": [
    "## 1. Java Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "145dca19",
   "metadata": {},
   "source": [
    "### 1.1 Generate Node Embeddings using Fast Random Projection (Fast RP) for Java Packages\n",
    "\n",
    "[Fast Random Projection](https://neo4j.com/docs/graph-data-science/current/machine-learning/node-embeddings/fastrp) is used to reduce the dimensionality of the node feature space while preserving most of the distance information. Nodes with similar neighborhood result in node embedding with similar vectors.\n",
    "\n",
    "**👉Hint:** To skip existing node embeddings and always calculate them based on the parameters below edit `Node_Embeddings_0a_Query_Calculated` so that it won't return any results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8efca2cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "java_package_embeddings_parameters={\n",
    "    \"dependencies_projection\": \"java-package-embeddings-notebook\",\n",
    "    \"dependencies_projection_node\": \"Package\",\n",
    "    \"dependencies_projection_weight_property\": \"weight25PercentInterfaces\",\n",
    "    \"dependencies_projection_write_property\": \"embeddingsFastRandomProjection\",\n",
    "    \"dependencies_projection_embedding_dimension\":\"32\"\n",
    "}\n",
    "embeddings = create_node_embeddings(\"../cypher/Node_Embeddings/Node_Embeddings_1d_Fast_Random_Projection_Stream.cypher\", java_package_embeddings_parameters)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76d8bca1",
   "metadata": {},
   "source": [
    "### 1.2 Dimensionality reduction with t-distributed stochastic neighbor embedding (t-SNE)\n",
    "\n",
    "This step takes the original node embeddings with a higher dimensionality, e.g. 64 floating point numbers, and reduces them into a two dimensional array for visualization. For more details look up the function declaration for \"prepare_node_embeddings_for_2d_visualization\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "031abacc",
   "metadata": {},
   "outputs": [],
   "source": [
    "node_embeddings_for_visualization = prepare_node_embeddings_for_2d_visualization(embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f908c47f",
   "metadata": {},
   "source": [
    "### 1.3 Visualization of the node embeddings reduced to two dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "459a819c",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_2d_node_embeddings(\n",
    "    node_embeddings_for_visualization, \n",
    "    \"Java Package positioned by their dependency relationships (FastRP node embeddings + t-SNE)\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b690b9a7",
   "metadata": {},
   "source": [
    "### 1.4 Node Embeddings for Java Packages using HashGNN\n",
    "\n",
    "[HashGNN](https://neo4j.com/docs/graph-data-science/2.6/machine-learning/node-embeddings/hashgnn) resembles Graph Neural Networks (GNN) but does not include a model or require training. It combines ideas of GNNs and fast randomized algorithms. For more details see [HashGNN](https://neo4j.com/docs/graph-data-science/2.6/machine-learning/node-embeddings/hashgnn). Here, the latter 3 steps are combined into one for HashGNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d3cfb7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "java_package_embeddings_parameters={\n",
    "    \"dependencies_projection\": \"java-package-embeddings-notebook\",\n",
    "    \"dependencies_projection_node\": \"Package\",\n",
    "    \"dependencies_projection_weight_property\": \"weight25PercentInterfaces\",\n",
    "    \"dependencies_projection_write_property\": \"embeddingsHashGNN\",\n",
    "    \"dependencies_projection_embedding_dimension\":\"64\"\n",
    "}\n",
    "embeddings = create_node_embeddings(\"../cypher/Node_Embeddings/Node_Embeddings_2d_Hash_GNN_Stream.cypher\", java_package_embeddings_parameters)\n",
    "node_embeddings_for_visualization = prepare_node_embeddings_for_2d_visualization(embeddings)\n",
    "plot_2d_node_embeddings(\n",
    "    node_embeddings_for_visualization, \n",
    "    \"Java Package positioned by their dependency relationships (HashGNN node embeddings + t-SNE)\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "248d88b4",
   "metadata": {},
   "source": [
    "### 2.5 Node Embeddings for Java Packages using node2vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62c40c45",
   "metadata": {},
   "outputs": [],
   "source": [
    "java_package_embeddings_parameters={\n",
    "    \"dependencies_projection\": \"java-package-embeddings-notebook\",\n",
    "    \"dependencies_projection_node\": \"Package\",\n",
    "    \"dependencies_projection_weight_property\": \"weight25PercentInterfaces\",\n",
    "    \"dependencies_projection_write_property\": \"embeddingsNode2Vec\",\n",
    "    \"dependencies_projection_embedding_dimension\":\"32\"\n",
    "}\n",
    "embeddings = create_node_embeddings(\"../cypher/Node_Embeddings/Node_Embeddings_3d_Node2Vec_Stream.cypher\", java_package_embeddings_parameters)\n",
    "node_embeddings_for_visualization = prepare_node_embeddings_for_2d_visualization(embeddings)\n",
    "plot_2d_node_embeddings(\n",
    "    node_embeddings_for_visualization, \n",
    "    \"Java Package positioned by their dependency relationships (node2vec node embeddings + t-SNE)\"\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "JohT"
   }
  ],
  "code_graph_analysis_pipeline_data_validation": "ValidateJavaPackageDependencies",
  "kernelspec": {
   "display_name": "codegraph",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  },
  "title": "Object Oriented Design Quality Metrics for Java with Neo4j"
 },
 "nbformat": 4,
 "nbformat_minor": 5
}