Skip to content

Commit

Permalink
Merge - v2.0.0
Browse files Browse the repository at this point in the history
  • Loading branch information
DaJaime committed Apr 22, 2024
2 parents abf32e1 + 05fb663 commit 8820d41
Show file tree
Hide file tree
Showing 67 changed files with 2,080 additions and 892 deletions.
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ target/
!.mvn/wrapper/maven-wrapper.jar
!**/src/main/**/target/
!**/src/test/**/target/
goblinWeaver_data

### STS ###
.apt_generated
Expand Down Expand Up @@ -31,4 +32,4 @@ build/

### VS Code ###
.vscode/
osvData/
osvData/
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
</parent>
<groupId>com.cifre.sap.su</groupId>
<artifactId>goblinWeaver</artifactId>
<version>1.0.0</version>
<version>2.0.0</version>
<name>goblinWeaver</name>
<description>Weaver to query Neo4j ecosystem dependency graph</description>
<properties>
Expand Down
35 changes: 23 additions & 12 deletions readme.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,31 @@
# Gobelin Weaver
# Goblin Weaver
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE.txt)

The Weaver is a REST API that takes a Cypher query and desired additional information as input and returns the result of this query on a Neo4j ecosystem dependency graph, enriching it on the fly according to the users needs.
The Weaver is a REST API for querying the Maven Central's dependency graph and enriching it by adding information to it according to the user's needs.

This weaver comes in addition to the ecosystem dependency graph miner available here: https://github.com/Goblin-Ecosystem/goblinWeaver.
An example of usage of this tool is available here: https://github.com/Goblin-Ecosystem/mavenDatasetExperiences
A Zenodo archive that contains the associated dataset dump and the Weaver jar is available here: https://zenodo.org/records/10291589.
This weaver use the Maven ecosystem dependency graph available here: https://zenodo.org/records/10605656

If you use the dataset dump present in Zenodo, please use a Neo4j version 4.x.
This graph was created by the Goblin Miner, and can be updated by it: https://github.com/Goblin-Ecosystem/goblinMiner

An example of usage of this tool is available here: https://github.com/Goblin-Ecosystem/mavenDatasetExperiences (on Weaver version 1.0.0).

If you use the dataset dump present in Zenodo, please use a Neo4j database version 4.x.

## Added values
- CVE: We use the osv.dev dataset to get CVEs information (https://osv.dev/).
- CVE_AGGREGATED: Aggregate release and dependencies (with transitivity) CVE.
- FRESHNESS: Corresponds, for a specific release, to the number of more recent releases available and to the time elapsed in milliseconds between the specific release and the most recent release.
- FRESHNESS_AGGREGATED: Aggregate release and dependencies (with transitivity) freshness.
- POPULARITY_1_YEAR: Corresponds, for a specific release, the number of version released within a maximum of one year after the current graph date using the specified release.
- POPULARITY_1_YEAR_AGGREGATED: Aggregate release and dependencies (with transitivity) POPULARITY_1_YEAR.
- SPEED: Corresponds to the average number of releases per day of an artifact. More information here: https://benevol2022.github.io/papers/DamienJaime.pdf

## Memoization
To avoid having to calculate the same metrics several times, added values are stored in the database graph once calculated.
These new "AddedValue" type nodes are linked with "Release" or "Artifact" nodes.
Due to their changing nature, routes are available in the API to remove them.
However, the Weaver itself can delete them on change: when updating the CVE database, the "CVE" added values are deleted, when updating the database "FRESHNESS", "POPULARITY" and "SPEED" added values are deleted.

## Requirements
- Java 17
- Maven, with MAVEN_HOME defines
Expand All @@ -31,10 +41,10 @@ The program will first download the osv.dev dataset and create a folder called "
If you already have downloaded this dataset and you don't want to update it, you can add the "noUpdate" argument on the java -jar command.

Example:
> java -Dneo4jUri="bolt://localhost:7687/" -Dneo4jUser="neo4j" -Dneo4jPassword="Password1" -jar .\target\goblinWeaver-1.0.0.jar
> java -Dneo4jUri="bolt://localhost:7687/" -Dneo4jUser="neo4j" -Dneo4jPassword="Password1" -jar .\target\goblinWeaver-2.0.0.jar

> java -Dneo4jUri="bolt://localhost:7687/" -Dneo4jUser="neo4j" -Dneo4jPassword="Password1" -jar .\target\goblinWeaver-1.0.0.jar
> java -Dneo4jUri="bolt://localhost:7687/" -Dneo4jUser="neo4j" -Dneo4jPassword="Password1" -jar .\target\goblinWeaver-2.0.0.jar noUpdate
## Use the API
Pre-designed requests are available, but you can also send your own Cypher requests directly to the API.
Expand All @@ -47,9 +57,10 @@ A swagger documentation of the API is available here:
The Weaver is designed to be extensible, allowing a user to easily add information their research need.
Here's how to add an added value:
1. Go to weaver/addedValue/AddedValueEnum and add the name of your new value.
2. Create a new class that implements weaver/addedValue/AddedValue.
3. Write your internal logic in this new class.
4. Go to weaver/Weaver and modify the "getValuesToAdd" method to define which type of element your new value can be applied to (Release, Artifact, dependency edge or relationship_AR edges).
2. Fill the three methods of this enumeration with your new added value
3. Create a new class that extends weaver/addedValue/AbstractAddedValue.
4. (optional) If you also want to create an aggregated value of your new added value, create a new class that extends your previous new class and implements the "AggregateValue" interface.
5. Write your internal logic in this new class.

## Licensing
Copyright 2023 SAP SE or an SAP affiliate company and Neo4j Ecosystem Weaver. Please see our [LICENSE](LICENSE) for copyright and license information.
Copyright 2024 SAP SE or an SAP affiliate company and Neo4j Ecosystem Weaver. Please see our [LICENSE](LICENSE) for copyright and license information.
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
package com.cifre.sap.su.goblinWeaver;

import com.cifre.sap.su.goblinWeaver.graphDatabase.GraphDatabaseSingleton;
import com.cifre.sap.su.goblinWeaver.utils.GraphUpdatedChecker;
import com.cifre.sap.su.goblinWeaver.utils.OsvProceeding;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Neo4jEcosystemWeaverApplication {

public static void main(String[] args) {
GraphDatabaseSingleton.getInstance(); // Init database connection
GraphUpdatedChecker.deleteAddedValuesIfUpdated(); // Check if database was updated
OsvProceeding.initOsvData(args); // Download CVE dataset
SpringApplication.run(Neo4jEcosystemWeaverApplication.class, args); // Run API
}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
package com.cifre.sap.su.goblinWeaver.api.controllers;

import com.cifre.sap.su.goblinWeaver.api.entities.ArtifactQuery;
import com.cifre.sap.su.goblinWeaver.graphDatabase.GraphDatabaseInterface;
import com.cifre.sap.su.goblinWeaver.graphDatabase.GraphDatabaseSingleton;
import com.cifre.sap.su.goblinWeaver.graphEntities.InternGraph;
import com.cifre.sap.su.goblinWeaver.weaver.Weaver;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.json.simple.JSONObject;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;

@RestController
@Tag(name = "Artifacts")
public class ArtifactController {

@Operation(
description = "Get a specific artifact from groupId:ArtifactId with added values",
summary = "Get a specific artifact from GA"
)
@PostMapping("/artifact")
public JSONObject getSpecificArtifact(@RequestBody ArtifactQuery artifactQuery) {
GraphDatabaseInterface gdb = GraphDatabaseSingleton.getInstance();
InternGraph graph = gdb.executeQuery(gdb.getQueryDictionary().getSpecificArtifactQuery(artifactQuery.toString()));
Weaver.weaveGraph(graph, artifactQuery.getAddedValues());
return graph.getJsonGraph();
}

@Operation(
description = "Get all releases of an artifact from groupId:ArtifactId with added values",
summary = "Get all releases of an artifact from GA"
)
@PostMapping("/artifact/releases")
public JSONObject getArtifactReleases(@RequestBody ArtifactQuery artifactQuery) {
GraphDatabaseInterface gdb = GraphDatabaseSingleton.getInstance();
InternGraph graph = gdb.executeQuery(gdb.getQueryDictionary().getArtifactReleasesQuery(artifactQuery.toString()));
Weaver.weaveGraph(graph, artifactQuery.getAddedValues());
return graph.getJsonGraph();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
package com.cifre.sap.su.goblinWeaver.api.controllers;

import com.cifre.sap.su.goblinWeaver.api.entities.CypherQuery;
import com.cifre.sap.su.goblinWeaver.graphDatabase.GraphDatabaseSingleton;
import com.cifre.sap.su.goblinWeaver.graphEntities.InternGraph;
import com.cifre.sap.su.goblinWeaver.weaver.Weaver;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.json.simple.JSONObject;
import org.springframework.web.bind.annotation.*;

@RestController
@Tag(name = "Cypher query")
public class CypherController {

@Operation(
description = "Execute a cypher query to the dependency graph with added values",
summary = "Execute a cypher query"
)
@PostMapping("/cypher")
public JSONObject executeCypherQuery(@RequestBody CypherQuery queryRequest) {
InternGraph graph = GraphDatabaseSingleton.getInstance().executeQuery(queryRequest.getQuery());
Weaver.weaveGraph(graph, queryRequest.getAddedValues());
return graph.getJsonGraph();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
package com.cifre.sap.su.goblinWeaver.api.controllers;

import com.cifre.sap.su.goblinWeaver.api.entities.ReleaseQueryList;
import com.cifre.sap.su.goblinWeaver.graphDatabase.GraphDatabaseSingleton;
import com.cifre.sap.su.goblinWeaver.graphEntities.InternGraph;
import com.cifre.sap.su.goblinWeaver.graphEntities.edges.DependencyEdge;
import com.cifre.sap.su.goblinWeaver.graphEntities.nodes.NodeObject;
import com.cifre.sap.su.goblinWeaver.graphEntities.nodes.NodeType;
import com.cifre.sap.su.goblinWeaver.graphEntities.nodes.ReleaseNode;
import com.cifre.sap.su.goblinWeaver.weaver.Weaver;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.json.simple.JSONObject;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;

import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;

@RestController
@Tag(name = "Graph")
public class GraphController {

@Operation(
description = "Get the project rooted graph",
summary = "Get the project rooted all graph from releases dependencies list"
)
@PostMapping("/graph/rootedGraph")
public JSONObject getRootedGraph(@RequestBody ReleaseQueryList releaseQueryList) {
InternGraph resultGraph = new InternGraph();
resultGraph.addNode(new ReleaseNode("ROOT", "ROOT", 0, ""));
for (ReleaseQueryList.Release release : releaseQueryList.getReleases()) {
resultGraph.addEdge(new DependencyEdge("ROOT", release.getGa(), release.getVersion(), "compile"));
}
resultGraph.mergeGraph(
GraphDatabaseSingleton.getInstance()
.getRootedGraph(
releaseQueryList.getReleases().stream().map(ReleaseQueryList.Release::getGav).collect(Collectors.toSet()
)
)
);
Weaver.weaveGraph(resultGraph, releaseQueryList.getAddedValues());
return resultGraph.getJsonGraph();
}

@Operation(
description = "Get the project rooted all possibilities graph",
summary = "Get the project rooted all possibilities graph from releases dependencies list"
)
@PostMapping("/graph/allPossibilitiesRooted")
public JSONObject getAllPossibilitiesRootedGraph(@RequestBody ReleaseQueryList releaseQueryList) {
InternGraph resultGraph = new InternGraph();
resultGraph.addNode(new ReleaseNode("ROOT", "ROOT", 0, ""));
for (ReleaseQueryList.Release release : releaseQueryList.getReleases()) {
resultGraph.addEdge(new DependencyEdge("ROOT", release.getGa(), release.getVersion(), "compile"));
}
resultGraph.mergeGraph(
GraphDatabaseSingleton.getInstance()
.getAllPossibilitiesGraph(
releaseQueryList.getReleases().stream().map(ReleaseQueryList.Release::getGa).collect(Collectors.toSet()
)
)
);
Weaver.weaveGraph(resultGraph, releaseQueryList.getAddedValues());
return resultGraph.getJsonGraph();
}

@Operation(
description = "Get the project rooted direct dependencies possibilities graph",
summary = "Get the project rooted direct dependencies possibilities graph from releases dependencies list"
)
@PostMapping("/graph/directPossibilitiesRooted")
public JSONObject getDirectPossibilitiesRootedGraph(@RequestBody ReleaseQueryList releaseQueryList) {
InternGraph resultGraph = new InternGraph();
resultGraph.addNode(new ReleaseNode("ROOT", "ROOT", 0, ""));
for (ReleaseQueryList.Release release : releaseQueryList.getReleases()) {
resultGraph.addEdge(new DependencyEdge("ROOT", release.getGa(), release.getVersion(), "compile"));
}
resultGraph.mergeGraph(
GraphDatabaseSingleton.getInstance()
.getDirectPossibilitiesGraph(
releaseQueryList.getReleases().stream().map(ReleaseQueryList.Release::getGa).collect(Collectors.toSet()
)
)
);
Weaver.weaveGraph(resultGraph, releaseQueryList.getAddedValues());
return resultGraph.getJsonGraph();
}

@Operation(
description = "Get the project rooted direct dependencies possibilities graph with transitive",
summary = "Get the project rooted direct dependencies possibilities graph with transitive version from releases dependencies list"
)
@PostMapping("/graph/directPossibilitiesWithTransitiveRooted")
public JSONObject getDirectPossibilitiesWithTransitiveRootedGraph(@RequestBody ReleaseQueryList releaseQueryList) {
InternGraph resultGraph = new InternGraph();
resultGraph.addNode(new ReleaseNode("ROOT", "ROOT", 0, ""));
for (ReleaseQueryList.Release release : releaseQueryList.getReleases()) {
resultGraph.addEdge(new DependencyEdge("ROOT", release.getGa(), release.getVersion(), "compile"));
}
// Get direct all possibilities
InternGraph directAllPossibilities = GraphDatabaseSingleton.getInstance()
.getDirectPossibilitiesGraph(releaseQueryList.getReleases().stream().map(ReleaseQueryList.Release::getGa).collect(Collectors.toSet()));
resultGraph.mergeGraph(directAllPossibilities);
// Get Releases dependencies
Map<String, Object> parameters = new HashMap<>();
Set<String> visitedReleases = new HashSet<>();
Set<String> releasesToTreat = directAllPossibilities.getGraphNodes().stream().filter(n -> n.getType().equals(NodeType.RELEASE)).map(NodeObject::getId).collect(Collectors.toSet());
while (!releasesToTreat.isEmpty()){
parameters.put("releaseIdList",releasesToTreat);
InternGraph queryResult = GraphDatabaseSingleton.getInstance().executeQueryWithParameters(GraphDatabaseSingleton.getInstance().getQueryDictionary().getDependencyGraphFromReleaseIdListParameter(), parameters);
resultGraph.mergeGraph(queryResult);
visitedReleases.addAll(releasesToTreat);
Set<String> newReleaseToTreat = resultGraph.getGraphNodes().stream().filter(node -> node instanceof ReleaseNode).map(NodeObject::getId).collect(Collectors.toSet());
newReleaseToTreat.removeAll(visitedReleases);
releasesToTreat.clear();
releasesToTreat.addAll(newReleaseToTreat);
}

Weaver.weaveGraph(resultGraph, releaseQueryList.getAddedValues());
return resultGraph.getJsonGraph();
}

@Operation(
description = "Get the project rooted direct dependencies possibilities graph with transitive",
summary = "Get the project rooted direct dependencies possibilities graph with transitive version from releases dependencies list"
)
@PostMapping("/graph/directNewPossibilitiesWithTransitiveRooted")
public JSONObject getDirectNewPossibilitiesWithTransitiveRootedGraph(@RequestBody ReleaseQueryList releaseQueryList) {
InternGraph resultGraph = new InternGraph();
resultGraph.addNode(new ReleaseNode("ROOT", "ROOT", 0, ""));
for (ReleaseQueryList.Release release : releaseQueryList.getReleases()) {
resultGraph.addEdge(new DependencyEdge("ROOT", release.getGa(), release.getVersion(), "compile"));
}
// Get direct all possibilities
InternGraph directAllPossibilities = GraphDatabaseSingleton.getInstance()
.getDirectNewPossibilitiesGraph(releaseQueryList.getReleases());
resultGraph.mergeGraph(directAllPossibilities);
// Get Releases dependencies
Map<String, Object> parameters = new HashMap<>();
Set<String> visitedReleases = new HashSet<>();
Set<String> releasesToTreat = directAllPossibilities.getGraphNodes().stream().filter(n -> n.getType().equals(NodeType.RELEASE)).map(NodeObject::getId).collect(Collectors.toSet());
while (!releasesToTreat.isEmpty()){
parameters.put("releaseIdList",releasesToTreat);
InternGraph queryResult = GraphDatabaseSingleton.getInstance().executeQueryWithParameters(GraphDatabaseSingleton.getInstance().getQueryDictionary().getDependencyGraphFromReleaseIdListParameter(), parameters);
resultGraph.mergeGraph(queryResult);
visitedReleases.addAll(releasesToTreat);
Set<String> newReleaseToTreat = resultGraph.getGraphNodes().stream().filter(node -> node instanceof ReleaseNode).map(NodeObject::getId).collect(Collectors.toSet());
newReleaseToTreat.removeAll(visitedReleases);
releasesToTreat.clear();
releasesToTreat.addAll(newReleaseToTreat);
}

Weaver.weaveGraph(resultGraph, releaseQueryList.getAddedValues());
return resultGraph.getJsonGraph();
}
}
Loading

0 comments on commit 8820d41

Please sign in to comment.