Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve: Parse POM from maven repo to extract and save license information in Java DB #21

Open
wants to merge 45 commits into
base: main
Choose a base branch
from
Open

Improve: Parse POM from maven repo to extract and save license information in Java DB #21

wants to merge 45 commits into from

Commits on Aug 7, 2023

  1. improve: [aquasecurity/trivy#4236] Parse POM from maven repo to extra… 

    …ct and save license information in Java DB
    @namandf
    namandf committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    c3c9e2c View commit details
    Browse the repository at this point in the history

  2. Add notes to trim license string beyond a certain limit

    @namandf
    namandf committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    e5e60fa View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2023

  1. Remove dependency on an external module for POM parsing.

    @namandf
    namandf committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    eb56cc3 View commit details
    Browse the repository at this point in the history

  2. Update separator for license strings to | instead of ,

    @namandf
    namandf committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    ecab1d4 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2023

  1. go mod tidy

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    8ff0829 View commit details
    Browse the repository at this point in the history

  2. Deduplicate licenses fetched from POM

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    6ea37a5 View commit details
    Browse the repository at this point in the history

  3. Use license classifier for information obtained from POM

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    239ddc8 View commit details
    Browse the repository at this point in the history

  4. Cache license information and use it on the fly while writing data to db

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    4c23e78 View commit details
    Browse the repository at this point in the history

  5. Cache directory creation

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    302e619 View commit details
    Browse the repository at this point in the history

  6. Cache directory creation

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    153a133 View commit details
    Browse the repository at this point in the history

  7. Cache directory creation

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    4dbd769 View commit details
    Browse the repository at this point in the history

  8. Add test cases for license classifier

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    9bc1dbb View commit details
    Browse the repository at this point in the history

  9. sort license keys before they are written to indexes cache directory

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    28ef2b3 View commit details
    Browse the repository at this point in the history

  10. update comment

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    ff74564 View commit details
    Browse the repository at this point in the history

  11. update comment

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    9a557a4 View commit details
    Browse the repository at this point in the history

  12. precautionary check to limit licnense strings to 30 characters

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    e214d84 View commit details
    Browse the repository at this point in the history

  13. Update timeout for license classification

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    21885b1 View commit details
    Browse the repository at this point in the history

  14. update batch size

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    3a4156e View commit details
    Browse the repository at this point in the history

  15. license meta variable name change

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    afd7dbb View commit details
    Browse the repository at this point in the history

  16. use crawler http client while creating license files

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    c5b40fd View commit details
    Browse the repository at this point in the history

  17. Update license file creation logic

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    c7fca19 View commit details
    Browse the repository at this point in the history

  18. test

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    6608af3 View commit details
    Browse the repository at this point in the history

  19. Add timeout for prepareClassifierData

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    560980f View commit details
    Browse the repository at this point in the history

  20. improvements

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    0608b3e View commit details
    Browse the repository at this point in the history

  21. improvements

    @namandf
    namandf committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    5ec049b View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2023

  1. improve processing of license classifier results

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    2467ee3 View commit details
    Browse the repository at this point in the history

  2. Remove files list from crawler struct

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    30bc298 View commit details
    Browse the repository at this point in the history

  3. Get rid of processFilesMap

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    d483d27 View commit details
    Browse the repository at this point in the history

  4. Add constant for normalized license json file name

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    7be6cf4 View commit details
    Browse the repository at this point in the history

  5. Update test cases

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    2f4c07c View commit details
    Browse the repository at this point in the history

  6. clean up

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    1640f0a View commit details
    Browse the repository at this point in the history

  7. clean up

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    512dba5 View commit details
    Browse the repository at this point in the history

  8. clean up

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    c4108cd View commit details
    Browse the repository at this point in the history

  9. Add a utils file in crawler package

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    7cc48c1 View commit details
    Browse the repository at this point in the history

  10. Update comment

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    33f2473 View commit details
    Browse the repository at this point in the history

  11. Set classification confidence

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    b001b9f View commit details
    Browse the repository at this point in the history

  12. Set classification confidence

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    029dcfe View commit details
    Browse the repository at this point in the history

  13. Add check for license string length

    @namandf
    namandf committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    176caa8 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2023

  1. Speed up license file generation

    @namandf
    namandf committed Aug 11, 2023
    Configuration menu
    Copy the full SHA
    87c7b41 View commit details
    Browse the repository at this point in the history

  2. Add comment

    @namandf
    namandf committed Aug 11, 2023
    Configuration menu
    Copy the full SHA
    a0769b5 View commit details
    Browse the repository at this point in the history

  3. reuse http client

    @namandf
    namandf committed Aug 11, 2023
    Configuration menu
    Copy the full SHA
    d7a9bdf View commit details
    Browse the repository at this point in the history

  4. skip file copy error

    @namandf
    namandf committed Aug 11, 2023
    Configuration menu
    Copy the full SHA
    3f3344b View commit details
    Browse the repository at this point in the history

  5. Handle empty normalized license strings

    @namandf
    namandf committed Aug 11, 2023
    Configuration menu
    Copy the full SHA
    e991f76 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2023

  1. Add temporary log to check license strings being trimmed

    @namandf
    namandf committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    674534e View commit details
    Browse the repository at this point in the history

  2. Increase license string length threshold to 150 characters

    @namandf
    namandf committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    8a924f3 View commit details
    Browse the repository at this point in the history