The role of "distro" and "image"

[Darren Hart]

I'm trying to wrap up my work on meta-tiny and integrate it into poky
proper. I'm having some difficulty drawing a line between the
responsibility of the distro definition versus that of the image definition.

For example, if I define a distro which uses tmpdevfs (no udev or mdev)
and specifies KMACHINE=yocto/standard/tiny to build the tiny variant of
the linux-yocto kernel, and customizes the busybox build to leave out
various things - would it make sense for someone to then try to build
core-image-sato with it? It doesn't to me, but nothing prevents the user
from doing that (other than a likely build failure).

So what does the distro define and what does the image define?

Digging around in conf/distro for meta and meta-yocto, I see things like
naming convention, additional dependencies, choice of toolchain and
libc, source mirrors, default virtual providers, etc.

The image seems to basically define the package list for the image as
well as the format of the rootfs and boot media.

If I understand this correctly, a new "tiny" distro definition would
change the preferred linux-yocto provider to a linux-yocto-tiny recipe
(which would in turn specify the default KMACHINE/KTYPE as well, the
TCLIBC, DISTRO_FEATURES_LIBC, and some naming rules.

I currently define a new task-core-tiny.bb to reset some
MACHINE_ESSENTIAL bits (qemu pulls in more than is necessary for tiny)
and redefine the REDEPENDS for the image. I believe I could do this
instead with assignments in the new distro definition and reuse
task-core-boot.bb.

As for images, I might be able to reuse core-image-minimal - but it
oddly contains "${POKY_EXTRA_INSTALL}" in the IMAGE_INSTALL. Since
core-image-minimal.bb is defined in oe-core and POKY is a distro notion
of meta-yocto, this contributes to my confusion over distro vs. image.

I'd appreciate some discussion to help me clarify where these lines
should be.

Thanks,

[Joshua Lock]

Post by Darren Hart
I'm trying to wrap up my work on meta-tiny and integrate it into poky
proper. I'm having some difficulty drawing a line between the
responsibility of the distro definition versus that of the image definition.
For example, if I define a distro which uses tmpdevfs (no udev or mdev)
and specifies KMACHINE=yocto/standard/tiny to build the tiny variant of
the linux-yocto kernel, and customizes the busybox build to leave out
various things - would it make sense for someone to then try to build
core-image-sato with it? It doesn't to me, but nothing prevents the user
from doing that (other than a likely build failure).

These all sounds like policy (and therefore distro) decisions to me. As
with much of our system there's no hard and fast rule but this certainly
sounds like a distro to me.

I think it's perfectly reasonable to assume that you can't build every
image you find in the wild for every distro configuration you create.

Post by Darren Hart
So what does the distro define and what does the image define?

Distro defines the build configuration policy, image defines image contents.

You shouldn't set anything in the image definition which affects how
things are built as the packages can then be used in another image.

Post by Darren Hart
Digging around in conf/distro for meta and meta-yocto, I see things like
naming convention, additional dependencies, choice of toolchain and
libc, source mirrors, default virtual providers, etc.

*nod*

Post by Darren Hart
The image seems to basically define the package list for the image as
well as the format of the rootfs and boot media.

*nod* though arguably the latter 3 are likely poked elsewhere also.

Post by Darren Hart
If I understand this correctly, a new "tiny" distro definition would
change the preferred linux-yocto provider to a linux-yocto-tiny recipe
(which would in turn specify the default KMACHINE/KTYPE as well, the
TCLIBC, DISTRO_FEATURES_LIBC, and some naming rules.

Yes!

Post by Darren Hart
I currently define a new task-core-tiny.bb to reset some
MACHINE_ESSENTIAL bits (qemu pulls in more than is necessary for tiny)
and redefine the REDEPENDS for the image. I believe I could do this
instead with assignments in the new distro definition and reuse
task-core-boot.bb.

Seems like a good approach.

Post by Darren Hart
As for images, I might be able to reuse core-image-minimal - but it
oddly contains "${POKY_EXTRA_INSTALL}" in the IMAGE_INSTALL. Since
core-image-minimal.bb is defined in oe-core and POKY is a distro notion
of meta-yocto, this contributes to my confusion over distro vs. image.
I'd appreciate some discussion to help me clarify where these lines
should be.

That's just a stray variable that has yet to be properly renamed by the
looks of it.

It sounds to me like your understanding is correct and this variable has
just thrown you for a loop.

Cheers,
Joshua

[Khem Raj]

Post by Joshua Lock

Post by Darren Hart
As for images, I might be able to reuse core-image-minimal - but it
oddly contains "${POKY_EXTRA_INSTALL}" in the IMAGE_INSTALL. Since
core-image-minimal.bb is defined in oe-core and POKY is a distro notion
of meta-yocto, this contributes to my confusion over distro vs. image.
I'd appreciate some discussion to help me clarify where these lines
should be.

That's just a stray variable that has yet to be properly renamed by the
looks of it.

yes there was a patch to rename that to IMAGE_EXTRA_INSTALL or
COREIMAGE_EXTRA_INSTALL

[Richard Purdie]

Post by Darren Hart
I'm trying to wrap up my work on meta-tiny and integrate it into poky
proper. I'm having some difficulty drawing a line between the
responsibility of the distro definition versus that of the image definition.
For example, if I define a distro which uses tmpdevfs (no udev or mdev)
and specifies KMACHINE=yocto/standard/tiny to build the tiny variant of
the linux-yocto kernel, and customizes the busybox build to leave out
various things - would it make sense for someone to then try to build
core-image-sato with it? It doesn't to me, but nothing prevents the user
from doing that (other than a likely build failure).
So what does the distro define and what does the image define?

I think Joshua's response is in line with what I'm thinking. Distro is
where all the policy is specified. The image is just a list of pieces of
software to include, its not meant to be much more than that.

Post by Darren Hart
Digging around in conf/distro for meta and meta-yocto, I see things like
naming convention, additional dependencies, choice of toolchain and
libc, source mirrors, default virtual providers, etc.

Right, all of this is "policy" at some level or another.

Post by Darren Hart
The image seems to basically define the package list for the image as
well as the format of the rootfs and boot media.

The latter is often machine specific. There are some images which
specify those things but they're a minority.

Post by Darren Hart
If I understand this correctly, a new "tiny" distro definition would
change the preferred linux-yocto provider to a linux-yocto-tiny recipe
(which would in turn specify the default KMACHINE/KTYPE as well, the
TCLIBC, DISTRO_FEATURES_LIBC, and some naming rules.

Yes, you'd inherit something for the base config and then customise it.

Post by Darren Hart
I currently define a new task-core-tiny.bb to reset some
MACHINE_ESSENTIAL bits (qemu pulls in more than is necessary for tiny)
and redefine the REDEPENDS for the image. I believe I could do this
instead with assignments in the new distro definition and reuse
task-core-boot.bb.

The latter reuse of task-core-boot would be ideal. If that can't work
I'd be interested to see why and if we couldn't enhance it so it could
work.

Post by Darren Hart
As for images, I might be able to reuse core-image-minimal - but it
oddly contains "${POKY_EXTRA_INSTALL}" in the IMAGE_INSTALL. Since
core-image-minimal.bb is defined in oe-core and POKY is a distro notion
of meta-yocto, this contributes to my confusion over distro vs. image.

This is a bug and there are some patches out there to turn it into
COREIMAGE_EXTRA_INSTALL. Its a feature of the core-image.bbclass
(formerly poky.bbclass, hence the name confusion).

Not all images can be built with a given distro, our base config is
rather ride ranging though for obvious reasons. You could "enforce" this
by adding some anonymous python to the distro which does something like:

if inherits("image") and PN != core-image-minimal:
raise SkipPackage("Image PN not compatible with DISTRO=XXX")

Cheers,

Richard

[Chris Larson]

On Tue, Dec 13, 2011 at 2:36 PM, Richard Purdie

Post by Richard Purdie
Not all images can be built with a given distro, our base config is
rather ride ranging though for obvious reasons. You could "enforce" this
   raise SkipPackage("Image PN not compatible with DISTRO=XXX")

This is a case where one is best 'controlling' this via policy, not
mechanism, in my opinion. This sort of arbitrary technical limitation
tends to be foolish and often bites someone somewhere down the line.
I'm sure you just wanted to note how it could be done, not recommend
that it should be done, but I thought it should be made clear. I
wouldn't recommend that anyone do this unless there is an extremely
good reason for it.

[Chris Larson]

Post by Chris Larson
On Tue, Dec 13, 2011 at 2:36 PM, Richard Purdie

Post by Richard Purdie
Not all images can be built with a given distro, our base config is
rather ride ranging though for obvious reasons. You could "enforce" this
   raise SkipPackage("Image PN not compatible with DISTRO=XXX")

This is a case where one is best 'controlling' this via policy, not
mechanism, in my opinion. This sort of arbitrary technical limitation
tends to be foolish and often bites someone somewhere down the line.
I'm sure you just wanted to note how it could be done, not recommend
that it should be done, but I thought it should be made clear. I
wouldn't recommend that anyone do this unless there is an extremely
good reason for it.

We had a similar problem at work, people were sprinkling COMPATIBLE_MACHINE left and right just to mark "I tested recipe X on machine Y". You can guess what happened when new machines needed to get added.

Haha, ouch. It's also worth taking a moment to emphasize that the way
distro/machine/image is structured is by design, not accidental.
Having these pieces be orthogonal buys us a great deal of flexibility
and capability, which is why we did it this way. Now, whether a given
distro/machine/image combination is *useful* is a different question,
and one I think is best addressed via policy / documentation.

[Richard Purdie]

Post by Chris Larson
On Tue, Dec 13, 2011 at 2:36 PM, Richard Purdie

Post by Richard Purdie
Not all images can be built with a given distro, our base config is
rather ride ranging though for obvious reasons. You could "enforce" this
raise SkipPackage("Image PN not compatible with DISTRO=XXX")

This is a case where one is best 'controlling' this via policy, not
mechanism, in my opinion. This sort of arbitrary technical limitation
tends to be foolish and often bites someone somewhere down the line.
I'm sure you just wanted to note how it could be done, not recommend
that it should be done, but I thought it should be made clear. I
wouldn't recommend that anyone do this unless there is an extremely
good reason for it.

We had a similar problem at work, people were sprinkling

COMPATIBLE_MACHINE left and right just to mark "I tested recipe X on
machine Y". You can guess what happened when new machines needed to
get added.
Haha, ouch. It's also worth taking a moment to emphasize that the way
distro/machine/image is structured is by design, not accidental.
Having these pieces be orthogonal buys us a great deal of flexibility
and capability, which is why we did it this way. Now, whether a given
distro/machine/image combination is *useful* is a different question,
and one I think is best addressed via policy / documentation.

I understand the flexibility but I also think we need to consider
usability a little. We have people using the system to whom it might not
be immediately obvious why building a sato image with the tiny distro is
a bad idea.

If you know enough to be able to make that work, the warning is easily
bypassed and isn't going to bother you.

So I think there is a case for warning the user if the combination
they're selecting is not one that is likely to work and we can easily
predict it. There are many combinations we can't predict which is fine.

Like any mechanism, it can be abused of course and using
COMPATIBLE_MACHINE as a sign of recipe testing clearly isn't a good idea
or what it was designed for.

Cheers,

Richard

[Chris Larson]

On Thu, Dec 15, 2011 at 2:18 PM, Richard Purdie

Post by Richard Purdie

Post by Chris Larson
On Tue, Dec 13, 2011 at 2:36 PM, Richard Purdie

Post by Richard Purdie
Not all images can be built with a given distro, our base config is
rather ride ranging though for obvious reasons. You could "enforce" this
   raise SkipPackage("Image PN not compatible with DISTRO=XXX")

This is a case where one is best 'controlling' this via policy, not
mechanism, in my opinion. This sort of arbitrary technical limitation
tends to be foolish and often bites someone somewhere down the line.
I'm sure you just wanted to note how it could be done, not recommend
that it should be done, but I thought it should be made clear. I
wouldn't recommend that anyone do this unless there is an extremely
good reason for it.

We had a similar problem at work, people were sprinkling

COMPATIBLE_MACHINE left and right just to mark "I tested recipe X on
machine Y". You can guess what happened when new machines needed to
get added.
Haha, ouch. It's also worth taking a moment to emphasize that the way
distro/machine/image is structured is by design, not accidental.
Having these pieces be orthogonal buys us a great deal of flexibility
and capability, which is why we did it this way. Now, whether a given
distro/machine/image combination is *useful* is a different question,
and one I think is best addressed via policy / documentation.

I understand the flexibility but I also think we need to consider
usability a little. We have people using the system to whom it might not
be immediately obvious why building a sato image with the tiny distro is
a bad idea.
If you know enough to be able to make that work, the warning is easily
bypassed and isn't going to bother you.
So I think there is a case for warning the user if the combination
they're selecting is not one that is likely to work and we can easily
predict it. There are many combinations we can't predict which is fine.

There's a rather large difference between *warning* the user and
raising SkipPackage, making it not possible to build without modifying
the recipe.

[Richard Purdie]

Post by Chris Larson
On Thu, Dec 15, 2011 at 2:18 PM, Richard Purdie

Post by Richard Purdie

Post by Chris Larson
On Tue, Dec 13, 2011 at 2:36 PM, Richard Purdie

Post by Richard Purdie
Not all images can be built with a given distro, our base config is
rather ride ranging though for obvious reasons. You could "enforce" this
raise SkipPackage("Image PN not compatible with DISTRO=XXX")

This is a case where one is best 'controlling' this via policy, not
mechanism, in my opinion. This sort of arbitrary technical limitation
tends to be foolish and often bites someone somewhere down the line.
I'm sure you just wanted to note how it could be done, not recommend
that it should be done, but I thought it should be made clear. I
wouldn't recommend that anyone do this unless there is an extremely
good reason for it.

We had a similar problem at work, people were sprinkling

COMPATIBLE_MACHINE left and right just to mark "I tested recipe X on
machine Y". You can guess what happened when new machines needed to
get added.
Haha, ouch. It's also worth taking a moment to emphasize that the way
distro/machine/image is structured is by design, not accidental.
Having these pieces be orthogonal buys us a great deal of flexibility
and capability, which is why we did it this way. Now, whether a given
distro/machine/image combination is *useful* is a different question,
and one I think is best addressed via policy / documentation.

I understand the flexibility but I also think we need to consider
usability a little. We have people using the system to whom it might not
be immediately obvious why building a sato image with the tiny distro is
a bad idea.
If you know enough to be able to make that work, the warning is easily
bypassed and isn't going to bother you.
So I think there is a case for warning the user if the combination
they're selecting is not one that is likely to work and we can easily
predict it. There are many combinations we can't predict which is fine.

There's a rather large difference between *warning* the user and
raising SkipPackage, making it not possible to build without modifying
the recipe.

What's needed is something which results in a "stop the build and force
the user to change something to continue" so I'm not sure I see the
problem with that.

I know exactly how much attention people pay to the WARNING messages :(
(those are getting cleaned up so hopefully those WARNINGS that are still
present become more meaningful/serious)

Cheers,

Richard

[Chris Larson]

On Thu, Dec 15, 2011 at 4:06 PM, Richard Purdie

Post by Richard Purdie

Post by Chris Larson
On Thu, Dec 15, 2011 at 2:18 PM, Richard Purdie

Post by Richard Purdie

Post by Chris Larson
On Tue, Dec 13, 2011 at 2:36 PM, Richard Purdie

Post by Richard Purdie
Not all images can be built with a given distro, our base config is
rather ride ranging though for obvious reasons. You could "enforce" this
   raise SkipPackage("Image PN not compatible with DISTRO=XXX")

This is a case where one is best 'controlling' this via policy, not
mechanism, in my opinion. This sort of arbitrary technical limitation
tends to be foolish and often bites someone somewhere down the line.
I'm sure you just wanted to note how it could be done, not recommend
that it should be done, but I thought it should be made clear. I
wouldn't recommend that anyone do this unless there is an extremely
good reason for it.

We had a similar problem at work, people were sprinkling

COMPATIBLE_MACHINE left and right just to mark "I tested recipe X on
machine Y". You can guess what happened when new machines needed to
get added.
Haha, ouch. It's also worth taking a moment to emphasize that the way
distro/machine/image is structured is by design, not accidental.
Having these pieces be orthogonal buys us a great deal of flexibility
and capability, which is why we did it this way. Now, whether a given
distro/machine/image combination is *useful* is a different question,
and one I think is best addressed via policy / documentation.

I understand the flexibility but I also think we need to consider
usability a little. We have people using the system to whom it might not
be immediately obvious why building a sato image with the tiny distro is
a bad idea.
If you know enough to be able to make that work, the warning is easily
bypassed and isn't going to bother you.
So I think there is a case for warning the user if the combination
they're selecting is not one that is likely to work and we can easily
predict it. There are many combinations we can't predict which is fine.

There's a rather large difference between *warning* the user and
raising SkipPackage, making it not possible to build without modifying
the recipe.

What's needed is something which results in a "stop the build and force
the user to change something to continue" so I'm not sure I see the
problem with that.

I strongly disagree. That's exactly what we shouldn't have, at least
unless there's an easy way to disable the behavior. Even our sanity
checks can be opted out, and those are far more critical than building
a rootfs that might not behave in a useful fashion on a particular
piece of hardware.

[Koen Kooi]

Post by Chris Larson
On Tue, Dec 13, 2011 at 2:36 PM, Richard Purdie

Post by Richard Purdie
Not all images can be built with a given distro, our base config is
rather ride ranging though for obvious reasons. You could "enforce" this
raise SkipPackage("Image PN not compatible with DISTRO=XXX")

This is a case where one is best 'controlling' this via policy, not
mechanism, in my opinion. This sort of arbitrary technical limitation
tends to be foolish and often bites someone somewhere down the line.
I'm sure you just wanted to note how it could be done, not recommend
that it should be done, but I thought it should be made clear. I
wouldn't recommend that anyone do this unless there is an extremely
good reason for it.

We had a similar problem at work, people were sprinkling COMPATIBLE_MACHINE left and right just to mark "I tested recipe X on machine Y". You can guess what happened when new machines needed to get added.

regards,

Koen