[yocto] Working on Dependent Recipes in eSDK

Discussion:

A***@selinc.com

2018-10-18 17:37:27 UTC

My team recently started using the eSDK for day-to-day development.
Previously, they were using the bitbake command to build, which works, but
has overhead. Yesterday three different people came up to me a question
about workflow with the eSDK, and they all had to do with dependent
recipes in the eSDK.

For example, a developer would sit down to make something new that relies
on a library provided by a recipe in the eSDK. So they "devtool
sdk-install needed-library", and get to work. While working, they
discover, I need to add something to this library I depend on so that it
does what I want in the way I want. So they "devtool modify
needed-library" and change the source of that library to do what they
want, and build that library using its build system (CMake or Autotools).
Then they return to working on the new something and find that none of
their changes to the library they depend on are available.

At this point, they are lost. They just start trying things to make it
work, like "devtool build needed-library", which successfully completes,
but doesn't help. Some people also try to use the build system of the
library they modified to install it into the machine sysroot of the eSDK
with a command like, "make install
DESTDIR=<ESDK_PATH>/tmp/sysroots/<MACHINE>", and that actually does help.
They can now use the changes they made to the library in the new something
they are working on. However, when they use the "devtool sdk-install"
command again, their changes to the needed library that they installed
into the machine sysroot are reverted.

So--what is the workflow they are supposed to use?

Pushing the changes to the library upstream, updating the recipe, and
waiting for a new eSDK is not practical given that the developers may have
to iterate on the changes to the library quite a bit before they get it
just right for their new something.

Is there a way to update the eSDK sysroots with recipes that are in the
workspace? Could "devtool build" also update the eSDK sysroots? Or is
there a different way to get this done?
--

Alexander Kanavin

2018-10-18 18:30:17 UTC

Permalink

I don’t think esdk is designed for this kind of ‘full stack’ development. You can go back to working directly with yocto layers, and skip the esdk altogether. Devtool will work exactly same (minus the sdk specific commands).

Alex

Post by A***@selinc.com
My team recently started using the eSDK for day-to-day development.
Previously, they were using the bitbake command to build, which works, but
has overhead. Yesterday three different people came up to me a question
about workflow with the eSDK, and they all had to do with dependent
recipes in the eSDK.
For example, a developer would sit down to make something new that relies
on a library provided by a recipe in the eSDK. So they "devtool
sdk-install needed-library", and get to work. While working, they
discover, I need to add something to this library I depend on so that it
does what I want in the way I want. So they "devtool modify
needed-library" and change the source of that library to do what they
want, and build that library using its build system (CMake or Autotools).
Then they return to working on the new something and find that none of
their changes to the library they depend on are available.
At this point, they are lost. They just start trying things to make it
work, like "devtool build needed-library", which successfully completes,
but doesn't help. Some people also try to use the build system of the
library they modified to install it into the machine sysroot of the eSDK
with a command like, "make install
DESTDIR=<ESDK_PATH>/tmp/sysroots/<MACHINE>", and that actually does help.
They can now use the changes they made to the library in the new something
they are working on. However, when they use the "devtool sdk-install"
command again, their changes to the needed library that they installed
into the machine sysroot are reverted.
So--what is the workflow they are supposed to use?
Pushing the changes to the library upstream, updating the recipe, and
waiting for a new eSDK is not practical given that the developers may have
to iterate on the changes to the library quite a bit before they get it
just right for their new something.
Is there a way to update the eSDK sysroots with recipes that are in the
workspace? Could "devtool build" also update the eSDK sysroots? Or is
there a different way to get this done?
--
_______________________________________________
yocto mailing list
https://lists.yoctoproject.org/listinfo/yocto

Paul Eggleton

2018-10-18 19:30:07 UTC

Permalink

I think we would *like* this to be possible, from what I can tell I think the
issue is that the copies of the library in the sysroots of other dependent
recipes aren't being updated. The eSDK (and devtool) were originally written
when there weren't recipe-specific sysroots, and this stuff was a bit simpler.

Moving back to the full build system won't automatically fix this, though it
does make workarounds possible - you'd need to rebuild the dependent recipes
to get their sysroots updated.

Cheers,
Paul

Post by Alexander Kanavin
I don’t think esdk is designed for this kind of ‘full stack’ development.
You can go back to working directly with yocto layers, and skip the esdk
altogether. Devtool will work exactly same (minus the sdk specific
commands).
Alex

--
Paul Eggleton
Intel Open Source Technology Centre

--

A***@selinc.com

2018-10-18 20:59:11 UTC

Permalink

Post by Paul Eggleton
I think we would *like* this to be possible, from what I can tell I think the
issue is that the copies of the library in the sysroots of other dependent
recipes aren't being updated. The eSDK (and devtool) were originally written
when there weren't recipe-specific sysroots, and this stuff was a bit simpler.
Moving back to the full build system won't automatically fix this, though it
does make workarounds possible - you'd need to rebuild the dependent recipes
to get their sysroots updated.
Cheers,
Paul
... snip ...
from devtool import exec_build_env_command, DevtoolError
"""Entry point for the devtool sdk-sysroot command"""
exec_build_env_command(config.init_path, basepath, 'bitbake

build-sysroots', watch=True)

Post by Paul Eggleton
raise DevtoolError('Failed to bitbake build-sysroots:\n%s' %

(str(e)))

Post by Paul Eggleton
"""Register devtool subcommands from the sdk plugin"""
parser_sdk_sysroot = subparsers.add_parser(
'sdk-sysroot',
help='Update the SDK sysroot',
description='Updates the SDK sysroot so that changes made to

recipes in the workspace will be available,',

Post by Paul Eggleton
group='sdk')
parser_sdk_sysroot.set_defaults(func=sdk_sysroot)

This allows me to "devtool modify <DEPENDENT_RECIPE>", edit the code,
"devtool build <DEPENDENT_RECIPE>", "devtool sdk-sysroot", and use the new
code from the dependent recipe immediately.

This makes me wonder if "bitbake build-sysroots" should not be a part of
"devtool build", like it is for "devtool sdk-install" (where I stole it
from).
--

ChenQi

2018-10-23 02:04:32 UTC

Permalink

Hi Aaron,

The main pain for now is that `devtool sdk-install <DEPEND_RECIPE>' does
not work as expected, right?
I just checked the codes of sdk-install, and found that it does not fit
the current RSS design.
I'd like to check with you if this is the major concern from your team?
Do you have some other concerns?
I'll open enhancements/bugs on bugzilla and hopefully will work on them
in the 2.7 development cycle.

Best Regards,
Chen Qi

Post by Paul Eggleton

Post by Paul Eggleton
I think we would *like* this to be possible, from what I can tell I

think the

Post by Paul Eggleton
issue is that the copies of the library in the sysroots of other

dependent

Post by Paul Eggleton
recipes aren't being updated. The eSDK (and devtool) were originally

written

Post by Paul Eggleton
when there weren't recipe-specific sysroots, and this stuff was a bit

simpler.

Post by Paul Eggleton
Moving back to the full build system won't automatically fix this,

though it

Post by Paul Eggleton
does make workarounds possible - you'd need to rebuild the dependent

recipes

Post by Paul Eggleton
to get their sysroots updated.
Cheers,
Paul
... snip ...
from devtool import exec_build_env_command, DevtoolError
"""Entry point for the devtool sdk-sysroot command"""
exec_build_env_command(config.init_path, basepath, 'bitbake

build-sysroots', watch=True)

Post by Paul Eggleton
raise DevtoolError('Failed to bitbake build-sysroots:\n%s' %

(str(e)))

recipes in the workspace will be available,',

Post by Paul Eggleton
group='sdk')
parser_sdk_sysroot.set_defaults(func=sdk_sysroot)

This allows me to "devtool modify <DEPENDENT_RECIPE>", edit the code,
"devtool build <DEPENDENT_RECIPE>", "devtool sdk-sysroot", and use the new
code from the dependent recipe immediately.
This makes me wonder if "bitbake build-sysroots" should not be a part of
"devtool build", like it is for "devtool sdk-install" (where I stole it
from).

A***@selinc.com

2018-10-25 15:28:40 UTC

Permalink

Post by ChenQi
Hi Aaron,
The main pain for now is that `devtool sdk-install <DEPEND_RECIPE>' does
not work as expected, right?
I just checked the codes of sdk-install, and found that it does not fit
the current RSS design.
I'd like to check with you if this is the major concern from your team?
Do you have some other concerns?
I'll open enhancements/bugs on bugzilla and hopefully will work on them
in the 2.7 development cycle.
Best Regards,
Chen Qi
... snip ...

Well, I'm not quite sure how "devtool sdk-install <DEPENDENT_RECIPE>"
relates after "devtool modify <DEPENDENT_RECIPE>" has been called.

My developers use devtool to install dependent recipes into the eSDK when
they start working on something. Often they find that they need to make a
change in the library code that a dependent recipe provides. In effect,
they are developing a patch for the dependent recipe. So they use devtool
to say they're going to modify that dependent recipe, and they go and
make the changes they want. Then what? How do they get those changes,
that patch, available to the eSDK?

Any changes made to the library code provided by the dependent recipe are
not available to the eSDK, except through devtool commands. In effect
there are two different worlds at play, the eSDK sysroots versus the
recipe sysroots. The devtool commands seem to make use of the recipe
sysroots, but in my mind that defeats the purpose of an eSDK. So I made
a plug-in for devtool were I can force the eSDK sysroots to be updated.
I'm not sure how that works, but it seems to propagate the workspace
(recipe sysroot land) to the eSDK, so changes that developers make are
now available for use.

I want my developers to use the toolchain (cmake, make, gcc, etc)
provided by the eSDK and by things installed into it
(devtool sdk-install), rather than having to use devtool as their
iterative build command.

Ultimately, I would like to be able to skip my devtool plug-in, as it
requires the use of "devtool build <DEPENDENT_RECIPE>" and
"devtool sdk-sysroot" each time a dependent recipe is modified. But I am
not sure how that would work, other than the devtool build command
should be updating the eSDK sysroots itself.
--

Khem Raj

2018-10-29 16:29:02 UTC

Permalink

Post by A***@selinc.com

Well, I'm not quite sure how "devtool sdk-install <DEPENDENT_RECIPE>"
relates after "devtool modify <DEPENDENT_RECIPE>" has been called.
My developers use devtool to install dependent recipes into the eSDK when
they start working on something. Often they find that they need to make a
change in the library code that a dependent recipe provides. In effect,
they are developing a patch for the dependent recipe. So they use devtool
to say they're going to modify that dependent recipe, and they go and
make the changes they want. Then what? How do they get those changes,
that patch, available to the eSDK?
Any changes made to the library code provided by the dependent recipe are
not available to the eSDK, except through devtool commands. In effect
there are two different worlds at play, the eSDK sysroots versus the
recipe sysroots. The devtool commands seem to make use of the recipe
sysroots, but in my mind that defeats the purpose of an eSDK. So I made
a plug-in for devtool were I can force the eSDK sysroots to be updated.
I'm not sure how that works, but it seems to propagate the workspace
(recipe sysroot land) to the eSDK, so changes that developers make are
now available for use.
I want my developers to use the toolchain (cmake, make, gcc, etc)
provided by the eSDK and by things installed into it
(devtool sdk-install), rather than having to use devtool as their
iterative build command.
Ultimately, I would like to be able to skip my devtool plug-in, as it
requires the use of "devtool build <DEPENDENT_RECIPE>" and
"devtool sdk-sysroot" each time a dependent recipe is modified. But I am
not sure how that would work, other than the devtool build command
should be updating the eSDK sysroots itself.

I think you have two options here

1. Feedback the tweaks to packages back into source and re-generate
SDK and if you setup SDK feeds then users/devs can simply do
devtool sdk-update
and this time the package in question will get updated in their SDKs
with the change that was initiated.

second option could be to generate a downstream derivative SDK using
devtool build-sdk and use this SDK

Post by A***@selinc.com
--
_______________________________________________
yocto mailing list
https://lists.yoctoproject.org/listinfo/yocto

A***@selinc.com

2018-10-18 19:19:22 UTC

Permalink

Post by Alexander Kanavin
I don’t think esdk is designed for this kind of ‘full stack’
development. You can go back to working directly with yocto layers,
and skip the esdk altogether. Devtool will work exactly same (minus
the sdk specific commands).
Alex

I'm trying to get the developers away from using bitbake and devtool to
build their code, as those tools have overhead and don't easily allow the
developers to iterate quickly or allow them to run other make targets for
unit testing, etc. I really want the developers at the "make" level for
the majority of their time. When it comes time for recipes and image
creation, then they'll use devtool.

Post by Alexander Kanavin

u=https-3A__lists.yoctoproject.org_listinfo_yocto&d=DwIFaQ&c=-
_uRSsrpJskZgEkGwdW-
sXvhn_FXVaEGsm0EI46qilk&r=GBJubmJUwwrr1B4TJFRqESMrRsDipvY66vPwm4-
gtNQ&m=-

C-70rjnGksaogzIkgE5MdoY3IfA3x62MZVOwbZmw4c&s=T5PxZ3bNB429M76sWwBqgrtMsMhB-

Post by Alexander Kanavin
kz1BXkdY8hOIRY&e=