Feature request: An option to check to ignore development dependencies

[bruceadams]

Some internal projects I work on, the big concern is licenses on dependencies that we use at runtime. We are more flexible about licenses for development dependencies. I would love to be about to use pylic check with an option to ignore development dependencies, much like poetry show --no-dev.

[bruceadams]

Looks like I can work around this; this is a little quirky.

poetry install --no-dev
poetry run pip install pylic
poetry run pylic check

[ubersan]

Hi @bruceadams

Thanks a lot for raising this issue! Indeed, you're right, that tracking a specific subset of dependencies is not directly supported by pylic. Instead pylic always simply considers all installed packages to be relevant.

This has the advantage to be simple regarding the API, but also easy to understand by the user (I hope at least 😄 )

But your issue is still totally valid. This would be especially useful once poetry releases dependency groups, where grouping of different sets of dependencies is straight forward. So perhaps an additional attribute in the [tool.pylic] section could solve this:

[tool.poetry.group.production.dependencies]
httpx = "*"
pendulum = "*"

[tool.poetry.group.dev.dependencies]
pytest = "^6.0.0"
pytest-mock = "*"

[tool.pylic]
safe_licenses = ["..."]
unsafe_packages = ["..."]
check_only_groups = ["production"]  # defaults to all groups if not set

This is just from the top of my head, so this might be not (easily) possible. I'm thinking e.g. about packages that are installed as dependencies from production packages, but are not explicitly mentioned in the corresponding section. Perhaps this information is in the poetry.lock file, but a) I'm not sure and b) what if it's out of date?

I will give this more thought, in the meantime you're stuck with creating an environment where only the packages are installed you want to license check and then install and run pylic on top of it.

Happy for input or solution attemps!

Cheers 🙂

[bruceadams]

My problem is: I need to fail on GPL-like licenses, but only if they appear in my main dependencies. I use pylint, and occasionally other GPL tools, in my dev dependencies.

What might be both understandable and practical to implement is another option in the [tools.pylic] section. Something like ignore_packages which, much like the existing unsafe_packages option says: don't fail on this package, even though its license isn't listed in my safe_licensees section.

[ubersan]

That sounds like a reasonable proposal and as you said would fit into the existing structure along the unsafe_packages option. I'm a bit short on time, so let me get back to you later again.

[ubersan]

The problem I see with e.g. ignore_packages is that the package could change the license without you knowing and transfer to something you wouldn't be allowed to use. But you wouldn't notice since the package would be always ignored when checking. In contrast unsafe_packages are flagged if they suddenly add a license. This can also be a problem already for dev dependencies. Perhaps a flag like pylic check --enable-ignores could dis/enable it, but this feels like the wrong approach. And something like pylic check --no-dev might be too conservative - also there are currently no different sets for allowed_licenses, where pylic could differentiate between allowed dev and prod licenses.

Also in your situation you probably still want to allow e.g. pylint's GPL license in case you're installing the prod and dev dependencies, also to make sure they stay with GPL.

Since pylic doesn't manage any package installations all it (currently) can do is to scan all installed packages and check their licenses. At this stage the packages are not separable by dev or prod or any label coming from poetry.

Also pylic supports package management without poetry, thus there might not always be the [tool.poetry.dependencies] and [tool.poetry.dev-dependencies] entries.

Does this make sense? Still I'm giving this more thought and the discussion is still wide open 🙂

[ubersan]

It will always be somewhat awkward to license check only production dependencies as pylic most probably won't be one of them.

Regarding separate license sets it might make sense to somehow have a similar grouping concept to poetry, where you could influence which license set the installed ones are checked against.

Either something inside [tool.pylic] or something more top-level like [tool.pylic.dev-licenses] and [tool.pylic.licenses] and then the check could be something like pylic check (use licenses) pylic check --dev (uses dev-licenses).

Something along those lines... At least this would allow you to split the licenses into distinctive sets for development and production.

[bruceadams]

Your concern with my proposed ignore_packages makes sense. Doing an ignore means ignoring changes and that's bad.

I can imagine a slightly more complex approach that still fits into pylic fundamental design of looking at installed packages. (This design strikes me as really good!)

How about two additional configuration items:

• restricted_licenses
• restricted_packages

Where the licenses listed in restricted_licenses are only allowed for packages listed in restricted_packages. Possibly with an additional check that there is no overlap between the allowed_licenses and restricted_licenses. This way, all licenses are checked and the user has control to allow them to accept some licenses as specific exceptions, not everywhere.

[ubersan]

Great stuff! This is what I had in mind with "something inside [tool.pylic]" 😄 I have to give it a bit more thought to fully understand the idea and make sure I see all implications, but as of now it feels like we could find a nice API and fit this into pylic 🙂

Also I'm not sure if we want to start simple with a specific solution (e.g restricted) or already start providing a more general feature for different scenarios e.g. (local dev, ci-cd dev, end-to-end tests, production, ...).

But it feels like we're slowly converging towards a solution!

[ubersan]

What about the following API:

Current usage and "base" scenario

[tool.pylic]
safe_licenses = [
    "Apache Software License",
    "Apache License 2.0",
    "...",
]
unsafe_packages = [
    "somPackage",
]

Checking the licenses would still be done via pylic check

Using the new grouping feature

[tool.pylic] # this is equal to [tool.pylic.default]
safe_licenses = [
    "Apache License 2.0",
    "GPL-2.0",
    "...",
]
unsafe_packages = [
    "somePackage",
]

[tool.pylic.prod]
safe_licenses = [
    "Apache Software License",
    "MIT",
    "MIT License",
    "...",
]

Checking the licenses of your production dependencies would then be possible via pylic check --only prod or something similar. pylic check (without any --only flag) would check the installed package against an aggregation of all groups. I would prefer this somewhat over the restricted_licenses|packages option approach as it

• is a bit more general and open to any grouping the user might need
• allows to have separate license sets, as pylic currently does not allow to list safe_licenses which are not used by any package
• doesn't require to list the packages - which could be many!

Would that be something that you could be excited about @bruceadams ? Or would you prefer another API?

[bruceadams]

This sounds good! In your example, I am unsure how pylic will figure out which packages to check for [tool.pylic.prod]. I'd love to hear more.

[ubersan]

This would work the same as it's working now. pylic will simply check against all installed packages. It would be up to the user to do the package management.

As an example:

• in a development build:

poetry install
poetry run pylic check

• in a production environment:

poetry install --no-dev
poetry run pip install pylic
poetry run pylic check --only prod

Regarding the poetry run pip install pylic, one might consider to manage pylic via the extras feature:

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""

[tool.poetry.dependencies]
python = "^3.9"
black = "^22.1.0"
pylic = {version = "^2.2.0", optional = true}

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[tool.poetry.extras]
pylic = ["pylic"]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

This would mean

• in a development build:

poetry install -E pylic
poetry run pylic check

• in a production environment:

poetry install --no-dev -E pylic
poetry run pylic check --only prod

But as one most probably uses pylic in a CI/CD use case which means this extra pip install step is in code and in one place only, it might be a bit over the top to go the extras route.

Did this help? 🙂

[ubersan]

hey @bruceadams we recently added the ignore_packages option, perhaps this might help you with this use case 🙂