[yocto] Kernel development using externalsrc

[CHMIELARZ Radoslaw]

Hi all,

The company I work for uses yocto to build an image with a custom linux kernel. The documentation (https://www.yoctoproject.org/docs/2.5.1/kernel-dev/kernel-dev.html) suggests a setup where the kernel is under git and the kernel recipe specifies this git repository hence the workflow is the following:

1. Make changes in local kernel directory
2. Submit changes to git (and presumably push to shared space)
3. Run yocto build which fetches the changes

However since I don't want to push the changes to git server before I test locally I have added the kernel sources using externalsrc. Unfortunately this has the drawback that linux-libc-headers runs do_configure and do_install phase which invalidates glibc-initial and requires a recompilation of a couple of hundreds of packages. I would like to avoid it.

Thus my question is: how can I retain this setup where I only make local changes to the kernel without yocto rebuilding glibc and dependent packages?

To add more background we are using morty release and the changes in kernel will be in configuration and devices (probably as modules but some may be included with the kernel binary). Glibc is also an externalsrc project at the moment since I'm trying to upgrade to newer yocto release but this will be taken from git in the long run.

Cheers,
Radek

[Khem Raj]

On Fri, Nov 2, 2018 at 8:47 AM CHMIELARZ Radoslaw

Post by CHMIELARZ Radoslaw
Hi all,
Make changes in local kernel directory
Submit changes to git (and presumably push to shared space)
Run yocto build which fetches the changes
However since I don’t want to push the changes to git server before I test locally I have added the kernel sources using externalsrc. Unfortunately this has the drawback that linux-libc-headers runs do_configure and do_install phase which invalidates glibc-initial and requires a recompilation of a couple of hundreds of packages. I would like to avoid it.
Thus my question is: how can I retain this setup where I only make local changes to the kernel without yocto rebuilding glibc and dependent packages?
To add more background we are using morty release and the changes in kernel will be in configuration and devices (probably as modules but some may be included with the kernel binary). Glibc is also an externalsrc project at the moment since I’m trying to upgrade to newer yocto release but this will be taken from git in the long run.

may be you want to take a look at devtool workflow
https://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#using-devtool-in-your-workflow

Post by CHMIELARZ Radoslaw
Cheers,
Radek
--
_______________________________________________
yocto mailing list
https://lists.yoctoproject.org/listinfo/yocto

--

[Uwe Geuder]

Post by CHMIELARZ Radoslaw
The company I work for uses yocto to build an image with a custom
linux kernel. The documentation
(https://www.yoctoproject.org/docs/2.5.1/kernel-dev/kernel-dev.html)
suggests a setup where the kernel is under git and the kernel recipe
Make changes in local kernel directory
Submit changes to git (and presumably push to shared space)
Run yocto build which fetches the changes
However since I don’t want to push the changes to git server before I
test locally I have added the kernel sources using
externalsrc.

Devtool is of course what they recommend, but can always change the git
repo to a local one in a .bbappend, so you save the push step and Yocto
build saves fetching over the network. (Making a git commit for each
build attempt is very little overhead and I would recommend it anyway as
a good working practice. You can clean up using "git rebase -i" once you
are happy with your code and before you publish it.)

In one of my development branches I find

SRC_URI = "\
git:///home/myname/projects/yoctobuild/pkgsrc/somerepo/;protocol=file;branch=debug/#675-v2.16.0 \
file://some.patch \
file://other.patch \
"

SRCREV = "${AUTOREV}"

I have not done it for the kernel, but I see no reason why it would not
work there.

(Obviously the absolute path is nothing you would be able to share
unmodified with others. Maybe it can be made a bit more portable using
${THISDIR}, but I have not had any need to try it.)

Post by CHMIELARZ Radoslaw
Unfortunately this has the drawback that
linux-libc-headers runs do_configure and do_install phase which
invalidates glibc-initial and requires a recompilation of a couple of
hundreds of packages. I would like to avoid it.

How editing the kernel source would cause rebuilds trailing
linux-libc-haeaders I cannot see in my build here

$ bitbake-diffsigs -t linux-libc-headers configure
NOTE: Starting bitbake server...
ERROR: Only one matching sigdata file found for the specified task (linux-libc-headers do_configure)

Well, maybe I have never changed the kernel in this build area, so let's
check what it depends on:

$ bitbake-dumpsig -t linux-libc-headers configure
NOTE: Starting bitbake server...
ERROR: Metadata does not support finding signature data files

Hmm, no idea what that means. Until someone tells us what's wrong let's
use stamp files instead

$ bitbake-dumpsig tmp/stamps/corei7-64-poky-linux/linux-libc-headers/4.15.7-r0.do_configure.sigdata...
This task depends on the checksums of files: []
Hash for dependent task
/home/geuder/projects/yoctobuild/meta-nel/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.15.7.bb.do_prepare_recipe_sysroot is 5ac6adcdd9edfe2705fd461a42942613

$ bitbake-dumpsig tmp/stamps/corei7-64-poky-linux/linux-libc-headers/4.15.7-r0.do_prepare_recipe_sysroot.sigdata...
This task depends on the checksums of files: []
Hash for dependent task /home/geuder/projects/yoctobuild/meta-nel/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.15.7.bb.do_fetch is 362b694aea2202ce72ad259579d001ad

$ bitbake-dumpsig tmp/stamps/corei7-64-poky-linux/linux-libc-headers/4.15.7-r0.do_fetch.sigdata...
Variable SRC_URI value is ${KERNELORG_MIRROR}/linux/kernel/v${HEADER_FETCH_VER}/linux-${PV}.tar.${KORG_ARCHIVE_COMPRESSION}
Tasks this task depends on: []
This task depends on the checksums of files: []

So my build gets kernel headers from kernel.org (mirror), it does not
use the kernel source I use to build my kernel. Maybe that's different
for you?

Of course the output of the bitbake-dumpsig commands is massively
shortened, I just picked those lines that seemed likely candidates to
cause rebuilds. To get the whole truth in your build call...

$ bitbake-diffsigs -t linux-libc-headers configure

... after you made a small kernel edit and rebuilt.

Regards,

Uwe

Uwe Geuder
Neuro Event Labs Oy
Tampere, Finland
uwe.gexder at neuroeventlabs.com (Bot check: fix one obvious typo)
--

[Dimitris Tassopoulos]

Another thing you could do is to work with git branches and have a bbappend
file to set the branch and commit hash. You will still need to push the
changes to the remote git repo, but you don't have to merge to your master
branch.
Then if you don't like just drop the branch or cherry-pick only the commits
that are valuable to the master.

Regards,
Dimitris

On Fri, Nov 2, 2018 at 5:47 PM CHMIELARZ Radoslaw

Post by CHMIELARZ Radoslaw
The company I work for uses yocto to build an image with a custom
linux kernel. The documentation
(https://www.yoctoproject.org/docs/2.5.1/kernel-dev/kernel-dev.html)
suggests a setup where the kernel is under git and the kernel recipe
Make changes in local kernel directory
Submit changes to git (and presumably push to shared space)
Run yocto build which fetches the changes
However since I don’t want to push the changes to git server before I
test locally I have added the kernel sources using
externalsrc.

Devtool is of course what they recommend, but can always change the git
repo to a local one in a .bbappend, so you save the push step and Yocto
build saves fetching over the network. (Making a git commit for each
build attempt is very little overhead and I would recommend it anyway as
a good working practice. You can clean up using "git rebase -i" once you
are happy with your code and before you publish it.)
In one of my development branches I find
SRC_URI = "\
git:///home/myname/projects/yoctobuild/pkgsrc/somerepo/;protocol=file;branch=debug/#675-v2.16.0
\
file://some.patch \
file://other.patch \
"
SRCREV = "${AUTOREV}"
I have not done it for the kernel, but I see no reason why it would not
work there.
(Obviously the absolute path is nothing you would be able to share
unmodified with others. Maybe it can be made a bit more portable using
${THISDIR}, but I have not had any need to try it.)

Post by CHMIELARZ Radoslaw
Unfortunately this has the drawback that
linux-libc-headers runs do_configure and do_install phase which
invalidates glibc-initial and requires a recompilation of a couple of
hundreds of packages. I would like to avoid it.

How editing the kernel source would cause rebuilds trailing
linux-libc-haeaders I cannot see in my build here
$ bitbake-diffsigs -t linux-libc-headers configure
NOTE: Starting bitbake server...
ERROR: Only one matching sigdata file found for the specified task
(linux-libc-headers do_configure)
Well, maybe I have never changed the kernel in this build area, so let's
$ bitbake-dumpsig -t linux-libc-headers configure
NOTE: Starting bitbake server...
ERROR: Metadata does not support finding signature data files
Hmm, no idea what that means. Until someone tells us what's wrong let's
use stamp files instead
$ bitbake-dumpsig
tmp/stamps/corei7-64-poky-linux/linux-libc-headers/4.15.7-r0.do_configure.sigdata...
This task depends on the checksums of files: []
Hash for dependent task
/home/geuder/projects/yoctobuild/meta-nel/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.15.7.bb.do_prepare_recipe_sysroot
is 5ac6adcdd9edfe2705fd461a42942613
$ bitbake-dumpsig
tmp/stamps/corei7-64-poky-linux/linux-libc-headers/4.15.7-r0.do_prepare_recipe_sysroot.sigdata...
This task depends on the checksums of files: []
Hash for dependent task
/home/geuder/projects/yoctobuild/meta-nel/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.15.7.bb.do_fetch
is 362b694aea2202ce72ad259579d001ad
$ bitbake-dumpsig
tmp/stamps/corei7-64-poky-linux/linux-libc-headers/4.15.7-r0.do_fetch.sigdata...
Variable SRC_URI value is
${KERNELORG_MIRROR}/linux/kernel/v${HEADER_FETCH_VER}/linux-${PV}.tar.${KORG_ARCHIVE_COMPRESSION}
Tasks this task depends on: []
This task depends on the checksums of files: []
So my build gets kernel headers from kernel.org (mirror), it does not
use the kernel source I use to build my kernel. Maybe that's different
for you?
Of course the output of the bitbake-dumpsig commands is massively
shortened, I just picked those lines that seemed likely candidates to
cause rebuilds. To get the whole truth in your build call...
$ bitbake-diffsigs -t linux-libc-headers configure
... after you made a small kernel edit and rebuilt.
Regards,
Uwe
Uwe Geuder
Neuro Event Labs Oy
Tampere, Finland
uwe.gexder at neuroeventlabs.com (Bot check: fix one obvious typo)
--
_______________________________________________
yocto mailing list
https://lists.yoctoproject.org/listinfo/yocto