use hooks for main kernel/u-boot version defaulting/overriding; remove symlink usage from many families #6100

rpardini · 2023-12-28T00:05:33Z

use hooks for main kernel/u-boot version defaulting/overriding; remove symlink usage from many families

core: semantic change, KERNELSOURCE must be explicitly 'none' to avoid kernel compilation (not unset)
main-config: mark mirror variables readonly; add todo's
lib/family config: remove resquices of KERNELDIR which is completely unused for a long time
- also MAINLINE_KERNEL_DIR not needed then
main-config: arches: don't default KERNELSOURCE to mainline (already done in common), cleanup comments
main-config: arch common: kernel/u-boot config defaulting+overriding via hook instead of pre-setting
- common.conf delegating to new mainline_kernel_decide_version hooks
- special case handling for v6.7-rc7
- some examples
- late-stage "use rolling branch version"
uefi: don't spell out KERNELBRANCH anymore, KERNEL_MAJOR_MINOR is sufficient now
meson64: family common: drop KERNELBRANCH, rely on KERNEL_MAJOR_MINOR only, no more symlinks
rockchip64: family common: drop KERNELBRANCH, rely on KERNEL_MAJOR_MINOR only, no more symlinks
- rockchip64_common is used by BOARDFAMILY's with their own LINUXFAMILY, so we can't be very generic here
meson (32-bit): family common: drop KERNELBRANCH, rely on KERNEL_MAJOR_MINOR only, no more symlinks
[move] rockchip-rk3588 edge 6.7: move patches to archive, do not add symlinks
sunxi/sunxi64: move version locking to shared hook in mainline-kernel.conf.sh
rockchip (32-bit): move version locking to shared hook in mainline-kernel.conf.sh
main-config: arch common: review logging levels, squash some todo, add some todo
lib: implement and sprinkle some very basic config var change-tracking across board/family sourcing and several hook calls
main-config: arch common: check LINUXFAMILY is set before using it
sunxi/sunxi64: move version locking back to family code
rockchip (32-bit): move version locking back to family code
mainline-kernel.conf.sh: explain better why we need to handle 6.7-rc7 in a special way
mainline-kernel.conf.sh: we don't need the 6.7-rc7 hack anymore, comment it out and leave it as an example for future occurences

rpardini · 2023-12-28T00:29:47Z

Yeah I guess everyone is affected here, but mostly @viraniac @paolosabatino @igorpecovnik

families now only set KERNEL_MAJOR_MINOR, and the framework resolves KERNELBRANCH for us in a centralized way, including -rc handling, fixed/tagged versions, and per-release version locking if necessary
The hooks doing the resolution are in config/sources/mainline-kernel.conf.sh, with forced-tags for sunxi* and rockchip.
- user config / etc can also add more hooks to do more tricks if necessary.
We're also moving away from using symlinks in patches/kernel, to directly or automatically pointing KERNELPATCHDIR to the patches/kernel/archive/xxx-6.7 directory directly.
That way bumping kernels is standardized across all families. The few special cases left I will handle soon.
- To bump kernels (edge from 6.6 to 6.7): 1) copy 6.6 to 6.7 in archive; 2) change a single line in family config.
- To rollover kernels (current becomes legacy, edge becomes current): two line change in family config, and .config copy or update.
better/similar handling for u-boot parameters

For reviews: please review the commits individually

also: change-tracking code with white checkmarks tracking changes to relevant variables, so we can know who is setting what where

config/sources/mainline-kernel.conf.sh

viraniac · 2023-12-28T07:54:36Z

Is it possible to split this into 2 seperate PRs - one for cleanup and one for the kernel version related changes? That way we can merge the cleanup code immediately. That part seems to look ok

rpardini · 2023-12-28T09:19:45Z

Is it possible to split this into 2 seperate PRs - one for cleanup and one for the kernel version related changes? That way we can merge the cleanup code immediately. That part seems to look ok

Sure, I will remove the cleanups if possible. Sorry for mixing it up -- it was a long day.

rpardini · 2023-12-28T09:23:07Z

Is it possible to split this into 2 seperate PRs - one for cleanup and one for the kernel version related changes? That way we can merge the cleanup code immediately. That part seems to look ok

There, cleanups removed.

viraniac · 2023-12-28T10:11:16Z

config/sources/common.conf

+# If the family does NOT define them, assume sane defaults.
+function late_family_config__common_defaults_for_mainline_kernel() {
+	# @TODO: if KERNELSOURCE is 'none', don't do anything; config is explicitly asking to skip kernel compilation.
+	if [[ "${KERNELSOURCE}" == "none" ]]; then


This whole function looks like a validation function to check if all the required things are present or not. I think this should be moved out of config and to libs perhaps

Before this PR, having KERNELSOURCE unset meant don't build a kernel.
After this PR, having KERNELSOURCE unset means use a mainline kernel, and to avoid building a kernel, KERNELSOURCE must be set to none. This is the same logic as was used for BOOTCONFIG=none to avoid building u-boot.
Also, yes, those are hooks (functions) and can be moved around from config to libs or anywhere that is sourced before exts initialize.

config/sources/common.conf

viraniac · 2023-12-28T10:18:59Z

config/sources/common.conf

+		# If the branch is unset, auto-determine it based on KERNEL_MAJOR_MINOR; allow hooks to customize
+		if [[ -z ${KERNELBRANCH} ]]; then
+			declare original_kernelsource="${KERNELSOURCE}" # store for later comparing
+			call_extension_method "mainline_kernel_decide_version" <<- 'MAINLINE_KERNEL_DECIDE_VERSION'


I am not sure if this should be after all the checks you are adding. Someone can end up using this to unset all the variables that are being set. For example whats stopping me from putting unset LINUXCONFIG and unset LINUXFAMILY into this extension call.

Also I see you are not adding a hook function for the uboot equivalent of this function. So why do we need it here?

None of this has the pretense of being resistant to such unset's. None of Armbian is. Users can use lib.config to unset the hell out of everything anyway.

Ref u-boot: I have moved the minimal set of pre-settage that was done in common to a hook. U-boot does not entail the same level of "auto-updating" that kernel does (eg receives no point releases, only major.minor) so it's actually much simpler.

viraniac · 2023-12-28T10:25:40Z

lib/functions/main/config-prepare.sh

+
+		# call hooks to do late validation/mutation of sources, branches, patch dirs, etc.
+		call_extension_method "late_family_config" <<- 'LATE_FAMILY_CONFIG'
+			*late defaults/overrides, main hook point for KERNELSOURCE/BRANCH and BOOTSOURCE/BRANCH etc*


Do we really need another hook function? Can't we do the same thing with post_family_confg function. I get the validation part. But why mutate things now?

A separate hook is required, otherwise ordering the hooks would be a huge mess.
It also helps to understand that this logic is shared and global across all families, and not part of a single family's handling (which is what post_family_config is for).

Sorry I failed to see why its required. The validation part can be moved into this file. For the default values parrt as I understand we load the family source file first, then common and then arch specific one. So we do already have common.conf and <arch>.conf files to set defaults. Why do we need another hook?

Could you please share the use case so that I can try to see what exactly you are trying to solve by adding another hook function?

The case: centralized, non-family specific, management of mainline kernel versions.
The late_family_config hook was introduced to keep post_family_config logic and ordering unaltered but do global changes to all families / kernels / even vendor ones.
The mainline_kernel_decide_version is specific to mainline kernels.

viraniac

Hit approved by mistake instead of changes requested. See above comments

…d kernel compilation (not unset)

…y unused for a long time - also `MAINLINE_KERNEL_DIR` not needed then

…done in common), cleanup comments

…via hook instead of pre-setting - common.conf delegating to new `mainline_kernel_decide_version` hooks - special case handling for v6.7-rc7 - some examples - late-stage "use rolling branch version"

…ficient now

… only, no more symlinks

…NOR only, no more symlinks - rockchip64_common is used by BOARDFAMILY's with their own LINUXFAMILY, so we can't be very generic here

…R_MINOR only, no more symlinks

…symlinks

….conf.sh

…rnel.conf.sh

…d some todo

…g across board/family sourcing and several hook calls

… in a special way

…ent it out and leave it as an example for future occurences

rpardini · 2023-12-28T11:48:41Z

I don't have it in me to further defend these changes. Closing.

igorpecovnik · 2023-12-28T12:46:12Z

I don't have it in me to further defend these changes.

Those changes affect all people so its better to add more (perhaps useless) then less / no questions. Just to make sure. ~~Currently compilation does't work as hack for 6.7.rc is required after your previous PR and is part of this PR.~~

igorpecovnik · 2023-12-30T13:34:34Z

After #6087 was merged, current Rpi hack stop working.

github-actions bot added size/large PR with 250 lines or more Hardware Hardware related like kernel, U-Boot, ... Framework Framework components labels Dec 28, 2023

rpardini force-pushed the pr/use-hooks-for-main-kernelu-boot-version-defaultingoverriding-remove-symlink-usage-from-many-families branch from 267a25e to 247bd9d Compare December 28, 2023 00:13

rpardini marked this pull request as ready for review December 28, 2023 00:29

rpardini requested review from Tonymac32, amazingfate, efectn, igorpecovnik, lanefu, Manouchehri, TRSx80, ahoneybun, brentr, catalinii, joekhoobyar, krachlatte, paolosabatino, schwar3kat and a team as code owners December 28, 2023 00:29

viraniac requested changes Dec 28, 2023

View reviewed changes

config/sources/mainline-kernel.conf.sh Show resolved Hide resolved

rpardini force-pushed the pr/use-hooks-for-main-kernelu-boot-version-defaultingoverriding-remove-symlink-usage-from-many-families branch from 247bd9d to c1da30c Compare December 28, 2023 09:22

rpardini requested a review from viraniac December 28, 2023 09:27

viraniac approved these changes Dec 28, 2023

View reviewed changes

viraniac self-requested a review December 28, 2023 10:29

viraniac requested changes Dec 28, 2023

View reviewed changes

rpardini force-pushed the pr/use-hooks-for-main-kernelu-boot-version-defaultingoverriding-remove-symlink-usage-from-many-families branch 2 times, most recently from 0d7f4de to 9c4c680 Compare December 28, 2023 11:16

rpardini requested review from viraniac and igorpecovnik December 28, 2023 11:20

rpardini added 19 commits December 28, 2023 12:27

core: semantic change, KERNELSOURCE must be explicitly 'none' to avoi…

8737fb5

…d kernel compilation (not unset)

main-config: mark mirror variables readonly; add todo's

67cbdcc

lib/family config: remove resquices of KERNELDIR which is completel…

b4cae9c

…y unused for a long time - also `MAINLINE_KERNEL_DIR` not needed then

main-config: arches: don't default KERNELSOURCE to mainline (already …

2e3b698

…done in common), cleanup comments

main-config: arch common: kernel/u-boot config defaulting+overriding …

1ebb4f6

…via hook instead of pre-setting - common.conf delegating to new `mainline_kernel_decide_version` hooks - special case handling for v6.7-rc7 - some examples - late-stage "use rolling branch version"

uefi: don't spell out KERNELBRANCH anymore, KERNEL_MAJOR_MINOR is suf…

2c7f7b0

…ficient now

meson64: family common: drop KERNELBRANCH, rely on KERNEL_MAJOR_MINOR…

7595792

… only, no more symlinks

rockchip64: family common: drop KERNELBRANCH, rely on KERNEL_MAJOR_MI…

6198d42

…NOR only, no more symlinks - rockchip64_common is used by BOARDFAMILY's with their own LINUXFAMILY, so we can't be very generic here

meson (32-bit): family common: drop KERNELBRANCH, rely on KERNEL_MAJO…

78d15b8

…R_MINOR only, no more symlinks

[move] rockchip-rk3588 edge 6.7: move patches to archive, do not add …

e2761a9

…symlinks

sunxi/sunxi64: move version locking to shared hook in mainline-kernel…

dc3c1de

….conf.sh

rockchip (32-bit): move version locking to shared hook in mainline-ke…

4901eeb

…rnel.conf.sh

main-config: arch common: review logging levels, squash some todo, ad…

e99b655

…d some todo

lib: implement and sprinkle some very basic config var change-trackin…

04ca659

…g across board/family sourcing and several hook calls

main-config: arch common: check LINUXFAMILY is set before using it

c505bcc

sunxi/sunxi64: move version locking back to family code

82d9652

rockchip (32-bit): move version locking back to family code

6820dc8

mainline-kernel.conf.sh: explain better why we need to handle 6.7-rc7…

88e7ade

… in a special way

mainline-kernel.conf.sh: we don't need the 6.7-rc7 hack anymore, comm…

da1de25

…ent it out and leave it as an example for future occurences

rpardini force-pushed the pr/use-hooks-for-main-kernelu-boot-version-defaultingoverriding-remove-symlink-usage-from-many-families branch from 9c4c680 to da1de25 Compare December 28, 2023 11:28

rpardini closed this Dec 28, 2023

rpardini deleted the pr/use-hooks-for-main-kernelu-boot-version-defaultingoverriding-remove-symlink-usage-from-many-families branch December 28, 2023 11:59

igorpecovnik mentioned this pull request Jan 7, 2024

Introducing mainline kernel defaults, cleaning symlinks #6135

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use hooks for main kernel/u-boot version defaulting/overriding; remove symlink usage from many families #6100

use hooks for main kernel/u-boot version defaulting/overriding; remove symlink usage from many families #6100

rpardini commented Dec 28, 2023 •

edited

Loading

rpardini commented Dec 28, 2023

viraniac commented Dec 28, 2023

rpardini commented Dec 28, 2023

rpardini commented Dec 28, 2023

viraniac Dec 28, 2023 •

edited

Loading

rpardini Dec 28, 2023

viraniac Dec 28, 2023

viraniac Dec 28, 2023

rpardini Dec 28, 2023

viraniac Dec 28, 2023

rpardini Dec 28, 2023

viraniac Dec 28, 2023 •

edited

Loading

rpardini Dec 28, 2023

viraniac left a comment •

edited

Loading

rpardini commented Dec 28, 2023

igorpecovnik commented Dec 28, 2023 •

edited

Loading

igorpecovnik commented Dec 30, 2023

use hooks for main kernel/u-boot version defaulting/overriding; remove symlink usage from many families #6100

use hooks for main kernel/u-boot version defaulting/overriding; remove symlink usage from many families #6100

Conversation

rpardini commented Dec 28, 2023 • edited Loading

use hooks for main kernel/u-boot version defaulting/overriding; remove symlink usage from many families

rpardini commented Dec 28, 2023

viraniac commented Dec 28, 2023

rpardini commented Dec 28, 2023

rpardini commented Dec 28, 2023

viraniac Dec 28, 2023 • edited Loading

Choose a reason for hiding this comment

rpardini Dec 28, 2023

Choose a reason for hiding this comment

viraniac Dec 28, 2023

Choose a reason for hiding this comment

viraniac Dec 28, 2023

Choose a reason for hiding this comment

rpardini Dec 28, 2023

Choose a reason for hiding this comment

viraniac Dec 28, 2023

Choose a reason for hiding this comment

rpardini Dec 28, 2023

Choose a reason for hiding this comment

viraniac Dec 28, 2023 • edited Loading

Choose a reason for hiding this comment

rpardini Dec 28, 2023

Choose a reason for hiding this comment

viraniac left a comment • edited Loading

Choose a reason for hiding this comment

rpardini commented Dec 28, 2023

igorpecovnik commented Dec 28, 2023 • edited Loading

igorpecovnik commented Dec 30, 2023

rpardini commented Dec 28, 2023 •

edited

Loading

viraniac Dec 28, 2023 •

edited

Loading

viraniac Dec 28, 2023 •

edited

Loading

viraniac left a comment •

edited

Loading

igorpecovnik commented Dec 28, 2023 •

edited

Loading