From a15362974f9256eeff1529f9e01253e43ce4064e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Karthika=20Geethanand=20=20=C2=AF=5C=5F=28=E3=83=84=29=5F/?= =?UTF-8?q?=C2=AF?= <40568919+karthika-g@users.noreply.github.com> Date: Fri, 8 Sep 2023 15:48:32 +0530 Subject: [PATCH 1/7] Smoke Test Bug fixes (#86) * initial docs * latest sbom * Property addition in Dataconstant file and appsettings updation * Import multiple SBOM files from customer maven ,nuget,npm * CI by vg_dev and internal property addition in package identifier output_15062023 * SBOM creation * SBOM parsing For nuget and Maven * identifier type added * sbom_import debian * New pattern added for input filetype * Review comment implementation * Build failure fix * Multiple Sbom reading changes for NPM,DEBIAN * Unique Package reading for NPM * update * Check in for bom reading * UT & IT changes * maven cycloneDx parsing logic * ununsed method removal * PR Review changes * Update CA_UsageDocument.md * Update CA-Tool--SBOM-Vision.md * Npm Change * Update CA_UsageDocument.md * Update CA-Tool--SBOM-Vision.md * Update README.md * IT cases refined * IT cases * IT failure * IT Changes * Updated README.md * IT failure resolved * IT Changes * Mavne Files for IT * Update appSettingsSW360IntegrationTest.json * IT Issues * IT test case fix * appsettings.json file update * Nuget Dev Initial changes * UT for Nuget Dev * Issue on UT * Dependency mapping for NPM (#59) * Update compile.yml * Update compile.yml * Update compile.yml * CI by vg_adapting angular16plusversion packagelockjson format in pkg identifier_15062023 * Update compile.yml * dependency mapping check in July7 * IT failure fix for dependencymapping * IT fix * Test case fix for IT * Unit test addtion * Integration test and Review chnages --------- Co-authored-by: Vijayalakshmi027 <58800320+Vijayalakshmi027@users.noreply.github.com> Co-authored-by: MadanReddyK <125262006+MadanReddyK@users.noreply.github.com> Co-authored-by: Viji * changes * IT Changes * PR Review comments added * case issue * nuspec file updated * Package Found issue * code change for devdependency mapping for maven * Nuget Changes * Nuspec update * SBOM/maven dev dependency change 17 july (#63) * Maven development Dependency Change * Unit test failure fix * Integration test case fix * Integration test file changes * Access specifier modification * Review comments * SBOM template reading initial * Dependencymappingfor maven unit and IT test cases (#64) * Nupec Nuget dll updates * error message update * Nuget Assets issue * CA.nuspec update * Unit test file updation * IT fix * Nuspec update * Nuspec update --------- Co-authored-by: Sumanth K B * SBOM UT added * UT cases added * IT added for SBOM Template * Sbom update * IT Updated * Usage Doc updated * Update * Update CA_UsageDocument.md * Dependency mappings * SBOM dependency mapping NuGet (#68) * Dependency mapping Nuget_Aug1 * Dependency Mapping for Nuget * Revert "Dependency Mapping for Nuget" This reverts commit a91fc7250b4f237d7a35ed4c974044f2cb3266bd. * Ut File change * Bug fix for Jfrog Repo identification * Dependency mapping Nuget_commitaug3 * Removed Unwanted methods * CycloneDx filepath read * Removed package lock parsing logic * appsettings update * File Rename * Review comments * Null check added for template * Review changes * Update * Python changes * Remove dev dependency flag addition while creating component (#75) * UT addition for remove devdependency flag * Remove dev dependency while creating component * code changes for maven sbom template adding properties. (#74) * code changes for maven sbom template adding properties. * maven method moved to commenhelper file --------- Co-authored-by: Chalapala RaghavendraReddy * Data Constant added * sonar issue fixed * sonarcube issues fixed * sonarcube issues fixed * Changes * Template update * Bug fixed for maven files parseing input components count * Fix: Integration Test case failure (#82) * Unit Test case addtion * IT failure fix * Method name change * UT failure fix * Sonar issue fixed (#79) Co-authored-by: Chalapala RaghavendraReddy * Updated Readme and UsageDoc/CA_UsageDocument.md (#78) * Updated Readme and UsageDoc/CA_UsageDocument.md * Nuspec update * Review comments update * Review comment updates * Changes for Jfrog Repo Identification * UT added * Code Smell Fixes (#84) * Unit Test case addtion * IT failure fix * Method name change * UT failure fix * Code Smells fix * IT failure fix * Method name change * Log updation * UT Updation * Coverage improvement * Coverage improvement * Code Smell fix * Coverage improvement * Code coverage improvement * Pythom processor test added * UT code change * Coverage improvement * Test case fix * Update PythonParserTests.cs * Code Coverage Added * Python Class Exluded from Coverage * UnChange CC * Python CodeCoverage * CC Change --------- Co-authored-by: Sumanth K B * Update CA_UsageDocument.md * Update pyproject.toml * Updated Readme OSS files * Bug fix for notimplemented exception * Bug Fix for component exclusion * Nuget Change * Docker file update --------- Co-authored-by: Sumanth K B Co-authored-by: Viji Co-authored-by: sumanthkb44 <84563853+sumanthkb44@users.noreply.github.com> Co-authored-by: Vijayalakshmi027 <58800320+Vijayalakshmi027@users.noreply.github.com> Co-authored-by: MadanReddyK <125262006+MadanReddyK@users.noreply.github.com> Co-authored-by: Chalapala RaghavendraReddy Co-authored-by: crvreddy <136427687+crvreddy@users.noreply.github.com> --- src/LCT.PackageIdentifier/MavenProcessor.cs | 4 ++ .../Model/NugetModel/BuildInfoComponent.cs | 2 +- .../NugetDevDependencyParser.cs | 37 +++++++++++++++++-- src/LCT.PackageIdentifier/NugetProcessor.cs | 11 +++++- 4 files changed, 48 insertions(+), 6 deletions(-) diff --git a/src/LCT.PackageIdentifier/MavenProcessor.cs b/src/LCT.PackageIdentifier/MavenProcessor.cs index 78e31545..df303f0a 100644 --- a/src/LCT.PackageIdentifier/MavenProcessor.cs +++ b/src/LCT.PackageIdentifier/MavenProcessor.cs @@ -37,6 +37,7 @@ public Bom ParsePackageFile(CommonAppSettings appSettings) List componentsToBOM = new(); List ListOfComponents = new(); Bom bom = new(); + int noOfExcludedComponents = 0; List dependenciesForBOM = new(); List configFiles; @@ -85,6 +86,9 @@ public Bom ParsePackageFile(CommonAppSettings appSettings) componentsForBOM = ListOfComponents.Distinct(new ComponentEqualityComparer()).ToList(); BomCreator.bomKpiData.DuplicateComponents = totalComponentsIdentified - componentsForBOM.Count; + + componentsForBOM = CommonHelper.RemoveExcludedComponents(componentsForBOM, appSettings.Maven.ExcludedComponents, ref noOfExcludedComponents); + BomCreator.bomKpiData.ComponentsExcluded += noOfExcludedComponents; bom.Components = componentsForBOM; bom.Dependencies = dependenciesForBOM; diff --git a/src/LCT.PackageIdentifier/Model/NugetModel/BuildInfoComponent.cs b/src/LCT.PackageIdentifier/Model/NugetModel/BuildInfoComponent.cs index 6c0cb5c8..19a801db 100644 --- a/src/LCT.PackageIdentifier/Model/NugetModel/BuildInfoComponent.cs +++ b/src/LCT.PackageIdentifier/Model/NugetModel/BuildInfoComponent.cs @@ -158,7 +158,7 @@ public override bool Equals(object obj) public override int GetHashCode() { - throw new NotImplementedException(); + return 0; } } } diff --git a/src/LCT.PackageIdentifier/NugetDevDependencyParser.cs b/src/LCT.PackageIdentifier/NugetDevDependencyParser.cs index a113151a..55eaee8e 100644 --- a/src/LCT.PackageIdentifier/NugetDevDependencyParser.cs +++ b/src/LCT.PackageIdentifier/NugetDevDependencyParser.cs @@ -18,6 +18,7 @@ using System.Security.Cryptography; using LCT.PackageIdentifier.Model.NugetModel; using System.Text.Json; +using System.Runtime.InteropServices; namespace LCT.PackageIdentifier { @@ -122,9 +123,37 @@ private static void ParseJsonFile(string filePath, Container container) LockFile assetFile = assetFileReader.Read(filePath); if (assetFile.PackageSpec != null) { - isTestProject = IsTestProject(assetFile.PackageSpec.RestoreMetadata.ProjectPath); - - container.Name = Path.GetFileName(assetFile.PackageSpec.RestoreMetadata.ProjectPath); + if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) + { + Logger.Debug($"ParseJsonFile():Windows Asset FileName: " + assetFile.PackageSpec.RestoreMetadata.ProjectPath); + isTestProject = IsTestProject(assetFile.PackageSpec.RestoreMetadata.ProjectPath); + container.Name = Path.GetFileName(assetFile.PackageSpec.RestoreMetadata.ProjectPath); + Logger.Debug($"ParseJsonFile():Windows Asset File: IsTestProject: " + isTestProject); + } + else + { + string csprojFilePath = ""; + string dirName = Path.GetDirectoryName(filePath); + if (dirName.Contains("obj")) + { + dirName = dirName.Replace("obj", ""); + string[] filePaths = Directory.GetFiles(dirName, "*.csproj"); + csprojFilePath = filePaths.Length > 0 ? filePaths[0] : ""; + } + if(!string.IsNullOrEmpty(csprojFilePath) && File.Exists(csprojFilePath)) + { + Logger.Debug($"ParseJsonFile():Linux Asset FileName: " + csprojFilePath); + isTestProject = IsTestProject(csprojFilePath); + container.Name = Path.GetFileName(csprojFilePath); + Logger.Debug($"ParseJsonFile():Linux Asset File: IsTestProject: " + isTestProject); + } + else + { + Logger.Debug($"ParseJsonFile():Linux Asset FileName Not Found!! "); + isTestProject = false; + container.Name = Path.GetFileName(filePath); + } + } if (isTestProject) { @@ -138,6 +167,8 @@ private static void ParseJsonFile(string filePath, Container container) ParseLibrary(library, isTestProject, components, assetFile); } } + + Logger.Debug($"ParseJsonFile():Asset file found components: " + components.Count); } } diff --git a/src/LCT.PackageIdentifier/NugetProcessor.cs b/src/LCT.PackageIdentifier/NugetProcessor.cs index 9c70f2e8..d28cd959 100644 --- a/src/LCT.PackageIdentifier/NugetProcessor.cs +++ b/src/LCT.PackageIdentifier/NugetProcessor.cs @@ -392,7 +392,15 @@ private void ParsingInputFileForBOM(CommonAppSettings appSettings, ref List listofComponents = new(); ParseInputFiles(appSettings, filepath, listofComponents); ConvertToCycloneDXModel(listComponentForBOM, listofComponents, dependencies); - bom.Dependencies = dependencies; + if (bom.Dependencies == null) + { + bom.Dependencies = dependencies; + dependencies = new List(); + } + else + { + bom.Dependencies.AddRange(dependencies); + } BomCreator.bomKpiData.ComponentsinPackageLockJsonFile = listComponentForBOM.Count; } } @@ -429,7 +437,6 @@ private static void GetDetailsforManuallyAdded(List componentsForBOM, private static void ConvertToCycloneDXModel(List listComponentForBOM, List listofComponents, List dependencies) { - foreach (var prop in listofComponents) { Component components = new Component From 0f29458a4f39d3923733cedb839e0eda67b45903 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Karthika=20Geethanand=20=20=C2=AF=5C=5F=28=E3=83=84=29=5F/?= =?UTF-8?q?=C2=AF?= <40568919+karthika-g@users.noreply.github.com> Date: Fri, 8 Sep 2023 19:16:40 +0530 Subject: [PATCH 2/7] Update compile.yml --- .github/workflows/compile.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/compile.yml b/.github/workflows/compile.yml index ed87c95c..c46e47ac 100644 --- a/.github/workflows/compile.yml +++ b/.github/workflows/compile.yml @@ -116,7 +116,7 @@ jobs: DOCKERDEVARTIFACTORY: ${{ secrets.DOCKERDEVARTIFACTORY }} - name: Test - #if: ${{ false }} # disable for now + if: ${{ false }} # disable for now run: | $TestProjects = Get-ChildItem -Path *test*.csproj -Recurse -exclude TestUtilities.csproj,UnitTestUtilities.csproj Write-Host "**************************The test projects considered for execution: $TestProjects ******************************" From ebdcfae9fd08b2f078c2eece2c439e297d63eec5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Karthika=20Geethanand=20=20=C2=AF=5C=5F=28=E3=83=84=29=5F/?= =?UTF-8?q?=C2=AF?= <40568919+karthika-g@users.noreply.github.com> Date: Fri, 8 Sep 2023 19:59:26 +0530 Subject: [PATCH 3/7] Update CA.nuspec --- CA.nuspec | 1 + 1 file changed, 1 insertion(+) diff --git a/CA.nuspec b/CA.nuspec index c6c712e3..68126752 100644 --- a/CA.nuspec +++ b/CA.nuspec @@ -82,6 +82,7 @@ + From ecc6bbf0c7bb2569a87dc3e5e7ea3b06ff28c1dc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Karthika=20Geethanand=20=20=C2=AF=5C=5F=28=E3=83=84=29=5F/?= =?UTF-8?q?=C2=AF?= <40568919+karthika-g@users.noreply.github.com> Date: Fri, 8 Sep 2023 20:08:44 +0530 Subject: [PATCH 4/7] Update MavenProcessor.cs --- src/LCT.PackageIdentifier/MavenProcessor.cs | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/LCT.PackageIdentifier/MavenProcessor.cs b/src/LCT.PackageIdentifier/MavenProcessor.cs index 8c57a6f6..2ba4d133 100644 --- a/src/LCT.PackageIdentifier/MavenProcessor.cs +++ b/src/LCT.PackageIdentifier/MavenProcessor.cs @@ -87,8 +87,6 @@ public Bom ParsePackageFile(CommonAppSettings appSettings) BomCreator.bomKpiData.DuplicateComponents = totalComponentsIdentified - componentsForBOM.Count; - componentsForBOM = CommonHelper.RemoveExcludedComponents(componentsForBOM, appSettings.Maven.ExcludedComponents, ref noOfExcludedComponents); - BomCreator.bomKpiData.ComponentsExcluded += noOfExcludedComponents; if (appSettings.Maven.ExcludedComponents != null) { From adb8575043e2b4741f442a42ddf0b8d89dbb05d3 Mon Sep 17 00:00:00 2001 From: MadanReddyK <125262006+MadanReddyK@users.noreply.github.com> Date: Fri, 8 Sep 2023 23:00:03 +0530 Subject: [PATCH 5/7] Update compile.yml --- .github/workflows/compile.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/compile.yml b/.github/workflows/compile.yml index c46e47ac..0ef64c47 100644 --- a/.github/workflows/compile.yml +++ b/.github/workflows/compile.yml @@ -3,8 +3,7 @@ name: Build & Test on: push: pull_request: - schedule: - - cron: "30 0 * * 5" + jobs: From 85eb9779ff92ec9b09889dee73eccec8f5e82649 Mon Sep 17 00:00:00 2001 From: MadanReddyK <125262006+MadanReddyK@users.noreply.github.com> Date: Thu, 14 Sep 2023 11:40:54 +0530 Subject: [PATCH 6/7] Update compile.yml --- .github/workflows/compile.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/compile.yml b/.github/workflows/compile.yml index 0ef64c47..77416f94 100644 --- a/.github/workflows/compile.yml +++ b/.github/workflows/compile.yml @@ -115,7 +115,7 @@ jobs: DOCKERDEVARTIFACTORY: ${{ secrets.DOCKERDEVARTIFACTORY }} - name: Test - if: ${{ false }} # disable for now + #if: ${{ false }} # disable for now run: | $TestProjects = Get-ChildItem -Path *test*.csproj -Recurse -exclude TestUtilities.csproj,UnitTestUtilities.csproj Write-Host "**************************The test projects considered for execution: $TestProjects ******************************" From 71290a342a498faef13ce383f3987fd7e64fa7cd Mon Sep 17 00:00:00 2001 From: karthika Date: Thu, 28 Sep 2023 09:38:34 +0530 Subject: [PATCH 7/7] Usage and Readme Updation for python workflow,Version update --- CA.nuspec | 6 +++--- README.md | 2 +- doc/UsageDoc/CA_UsageDocument.md | 6 +++--- src/LCT.Common/appSettings.json | 2 +- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/CA.nuspec b/CA.nuspec index 68126752..77065fa8 100644 --- a/CA.nuspec +++ b/CA.nuspec @@ -4,7 +4,7 @@ continuous-clearing - 4.0.0 + 4.1.0 Siemens AG continuous-clearing contributors https://github.com/siemens/continuous-clearing @@ -13,12 +13,12 @@ false The License clearing tool helps the Project Manager/Developer, to reduce the manual effort and enable the faster license clearing process, - by automatically identifying the third party oss components used in their project(i.e., npm, nuget, maven and Debian type) and it creates them in the sw360 and fossology + by automatically identifying the third party oss components used in their project(i.e., npm, nuget, maven,python and Debian type) and it creates them in the sw360 and fossology for clearing license Copyright 2023 - ClearingTool SW360 OSS Clearing Software Continuous-Clearing NPM NUGET DEBIAN MAVEN SBOM + ClearingTool SW360 OSS Clearing Software Continuous-Clearing NPM NUGET DEBIAN MAVEN PYTHON SBOM diff --git a/README.md b/README.md index 623227a4..e95ff292 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ # Introduction -The Continuous Clearing Tool scans and collects the 3rd party OSS components used in a NPM/NuGet/Maven/Debian and uploads it to SW360 and Fossology by accepting respective project ID for license clearing. +The Continuous Clearing Tool scans and collects the 3rd party OSS components used in a NPM/NuGet/Maven/Python/Debian and uploads it to SW360 and Fossology by accepting respective project ID for license clearing. The tool helps the developer/project manager to enable the clearing process faster by reducing the manual effort of creating SW360 and FOSSology workflows. diff --git a/doc/UsageDoc/CA_UsageDocument.md b/doc/UsageDoc/CA_UsageDocument.md index 7280c9e4..c57f0346 100644 --- a/doc/UsageDoc/CA_UsageDocument.md +++ b/doc/UsageDoc/CA_UsageDocument.md @@ -53,13 +53,13 @@ Continuous Clearing Tool reduces the effort in creating components in SW360 and # Continuous Clearing Tool workflow diagram - Package Identifier - - [NPM/NUGET/MAVEN](../usagedocimg/packageIdentifiernpmnuget.PNG) + - [NPM/NUGET/MAVEN/PYTHON](../usagedocimg/packageIdentifiernpmnuget.PNG) - [Debian](../usagedocimg/packageIdentifierdebian.PNG) - SW360 Package Creator - - [NPM/NUGET/MAVEN](../usagedocimg/packageCreatirnpmnuget.PNG) + - [NPM/NUGET/MAVEN/PYTHON](../usagedocimg/packageCreatirnpmnuget.PNG) - [Debian](../usagedocimg/packagecreatordebian.PNG) - Artifactory Uploader - - [NPM/NUGET/MAVEN](../usagedocimg/artifactoryuploader.PNG) + - [NPM/NUGET/MAVEN/PYTHON](../usagedocimg/artifactoryuploader.PNG) # Prerequisite diff --git a/src/LCT.Common/appSettings.json b/src/LCT.Common/appSettings.json index e00c4e32..401a2f74 100644 --- a/src/LCT.Common/appSettings.json +++ b/src/LCT.Common/appSettings.json @@ -5,7 +5,7 @@ // -------------------------------------------------------------------------------------------------------------------- { - "CaVersion": "4.0.0", + "CaVersion": "4.1.0", "TimeOut": 200, "ProjectType": "", "SW360ProjectName": "",