busybox and mdev, how does it really work?

[Hans Beckérus]

Hi. We have a system based on the poky-tiny distro and we use mdev as
our device manager, or at least we do not set it to something at all
in the distro .conf:

# Use tmpdevfs and the busybox runtime services
VIRTUAL-RUNTIME_dev_manager = ""
#VIRTUAL-RUNTIME_login_manager = "tinylogin"
#VIRTUAL-RUNTIME_init_manager = ""
VIRTUAL-RUNTIME_keymaps = ""

and busybox is configured for have CONFIG_MDEV support.
Other from this we have not added or installed any specific mdev packages.

Things looked ok for a while, /dev was populated properly after boot
and all necessary file systems / mount points were created.
We were using an ext2 fs in RAM for mounting /.
Now, we made a change to instead use a CPIO image compressed using xz
and enabled the support in kernel to handle this.
This is when our problems started :( Suddenly our system booted with
just a very minimalistic /dev folder, containing basically only a few
of the devices probed at boot time.
When checking /etc/init.d we discovered that the mdev script actually
is not part of our image at all? In fact, it never was, even when we
used the ext2 boot!
The only way I could see how this script may be added to the image was
to install busybox-mdev, but that resulted in even more problems.
Now our mtd partitions could no longer be accessed!? They were visible
in /dev but not found by tools like flashcp. If I reduced the mdev
script to only do 'mdev -s' it all seems to work, and /dev looks like
it did before the change of ram boot mode.
My guess here is that the re-mounting of /dev that is performed as
part of the mdev init script destroys something? Also, there seems to
be a mess among the scripts executed at boot time with respect to what
is mounted and not.
We have the mountall.sh script, devpts.sh, mdev and a few others that
all seems to think they need to mount core file systems, but nothing
besides mdev seems to mount /dev? How was this done before adding
mdev?
Obviously it worked well without it, but when moving to CPIO.xz it
seems to be mandatory, but requires changes to work as expected?

So, the questions now are:

- how was /dev populated before when there was no /etc/init.d/mdev?
- why does adding /etc/init.d/mdev corrupt our /dev filesystem and
render some of our devices unusable?
- should you really need to explicitly add busybox-mdev? And if so, is
it expected that we need to change the init script?

Anyone that could describe how this is supposed to work, and what we
probably did wrong?
Seems a little bit too "magical" right now, and it would really feel a
lot better if this could be explained.

Thanks.
Hans

Build Configuration:
BB_VERSION = "1.20.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "SUSE-LINUX-11"
TARGET_SYS = "arm-poky-linux-gnueabi"
DISTRO_VERSION = "1.5"
TUNE_FEATURES = "armv7a vfp cortexa9"
TARGET_FPU = "vfp"
meta
meta-yocto
meta-yocto-bsp = "dora:75bed4086eb83f1d24c31392f3dd54aa5c3679b1"
meta-oe = "dora:513e7ca20ddd0a5c3b649bf292a67c3e0473d3a8"

[Anders Darander]

Post by Hans Beckérus
Things looked ok for a while, /dev was populated properly after boot
and all necessary file systems / mount points were created.
We were using an ext2 fs in RAM for mounting /.
Now, we made a change to instead use a CPIO image compressed using xz
and enabled the support in kernel to handle this.
This is when our problems started :( Suddenly our system booted with
just a very minimalistic /dev folder, containing basically only a few
of the devices probed at boot time.
- how was /dev populated before when there was no /etc/init.d/mdev?

I haven't checked poky-tiny, or more specifically the kernel
configurations for it. But based on the description that I cited above,
I'd guess that poky-tiny has
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
enabled.

That would mean the the kernel itself were managing /dev for you when
you were using ext2. Unfortunately, CONFIG_DEVTMPFS_MOUNT do only
automount devmptfs on /dev if you're not in an initramfs (or initrd).

Thus, you should add a simple initscript which mounts devtmpfs on /dev.
Thereafter, you should be fine.

Regarding your other questions on mdev, it's been a little while since I
used mdev. Let us know if the ideas above is enough, or if you want to
dig more into mdev.

Cheers,
Anders

[Hans Beckérus]

Post by Anders Darander

Post by Hans Beckérus
Things looked ok for a while, /dev was populated properly after boot
and all necessary file systems / mount points were created.
We were using an ext2 fs in RAM for mounting /.
Now, we made a change to instead use a CPIO image compressed using xz
and enabled the support in kernel to handle this.
This is when our problems started :( Suddenly our system booted with
just a very minimalistic /dev folder, containing basically only a few
of the devices probed at boot time.
- how was /dev populated before when there was no /etc/init.d/mdev?

I haven't checked poky-tiny, or more specifically the kernel
configurations for it. But based on the description that I cited above,
I'd guess that poky-tiny has
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
enabled.

Indeed (from our .config):

CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y

Post by Anders Darander
That would mean the the kernel itself were managing /dev for you when
you were using ext2. Unfortunately, CONFIG_DEVTMPFS_MOUNT do only
automount devmptfs on /dev if you're not in an initramfs (or initrd).

Sounds reasonable. But, when moving from ext2.gz to cpio.xz, we still
did not add any init script for mounting /dev, yet it is still created
and populated? But only with a very limited set of devices.
I guess there is a difference when the kernel mounts the cpio.xz image
compared to expanding the ext2.gz in RAM with respect to how /dev is
handled. But I am only guessing here.

Post by Anders Darander
Thus, you should add a simple initscript which mounts devtmpfs on /dev.
Thereafter, you should be fine.

Well, using the one that came from busybox-mdev rendered our /dev
completely useless :(
As I said, if I reduce the script to perform only 'mdev -s', then it
works a lot better. So mounting /dev again seems like a bad idea.

Post by Anders Darander
Regarding your other questions on mdev, it's been a little while since I
used mdev. Let us know if the ideas above is enough, or if you want to
dig more into mdev.

mdev as a device manager I think we have under control, the confusion
is more in our current system behavior at boot.

Post by Anders Darander
Cheers,
Anders
--
Anders Darander
ChargeStorm AB / eStorm AB

[Hans Beckérus]

Post by Anders Darander

Post by Anders Darander

Post by Hans Beckérus
Things looked ok for a while, /dev was populated properly after boot
and all necessary file systems / mount points were created.
We were using an ext2 fs in RAM for mounting /.
Now, we made a change to instead use a CPIO image compressed using xz
and enabled the support in kernel to handle this.
This is when our problems started :( Suddenly our system booted with
just a very minimalistic /dev folder, containing basically only a few
of the devices probed at boot time.
- how was /dev populated before when there was no /etc/init.d/mdev?

I haven't checked poky-tiny, or more specifically the kernel
configurations for it. But based on the description that I cited above,
I'd guess that poky-tiny has
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
enabled.

CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y

Post by Anders Darander
That would mean the the kernel itself were managing /dev for you when
you were using ext2. Unfortunately, CONFIG_DEVTMPFS_MOUNT do only
automount devmptfs on /dev if you're not in an initramfs (or initrd).

Sounds reasonable. But, when moving from ext2.gz to cpio.xz, we still
did not add any init script for mounting /dev, yet it is still created
and populated? But only with a very limited set of devices.
I guess there is a difference when the kernel mounts the cpio.xz image
compared to expanding the ext2.gz in RAM with respect to how /dev is
handled. But I am only guessing here.

Another thing, You say CONFIG_DEVTMPFS_MOUNT does *not* automount /dev
if using the initrd scheme?
But I think it did when using the compressed ext2 file system? How
come /dev was mounted and populated automatically in the ext2 case but
when we changed to cpio.xz things seem to break?

Post by Anders Darander

Post by Anders Darander
Thus, you should add a simple initscript which mounts devtmpfs on /dev.
Thereafter, you should be fine.

Well, using the one that came from busybox-mdev rendered our /dev
completely useless :(
As I said, if I reduce the script to perform only 'mdev -s', then it
works a lot better. So mounting /dev again seems like a bad idea.

Post by Anders Darander
Regarding your other questions on mdev, it's been a little while since I
used mdev. Let us know if the ideas above is enough, or if you want to
dig more into mdev.

mdev as a device manager I think we have under control, the confusion
is more in our current system behavior at boot.

Post by Anders Darander
Cheers,
Anders
--
Anders Darander
ChargeStorm AB / eStorm AB

[Anders Darander]

Post by Hans Beckérus

Post by Anders Darander

Post by Anders Darander

Post by Hans Beckérus
Things looked ok for a while, /dev was populated properly after boot
and all necessary file systems / mount points were created.
We were using an ext2 fs in RAM for mounting /.
Now, we made a change to instead use a CPIO image compressed using xz
and enabled the support in kernel to handle this.
This is when our problems started :( Suddenly our system booted with
just a very minimalistic /dev folder, containing basically only a few
of the devices probed at boot time.
- how was /dev populated before when there was no /etc/init.d/mdev?

I haven't checked poky-tiny, or more specifically the kernel
configurations for it. But based on the description that I cited above,
I'd guess that poky-tiny has
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
enabled.

CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y

Post by Anders Darander
That would mean the the kernel itself were managing /dev for you when
you were using ext2. Unfortunately, CONFIG_DEVTMPFS_MOUNT do only
automount devmptfs on /dev if you're not in an initramfs (or initrd).

Sounds reasonable. But, when moving from ext2.gz to cpio.xz, we still
did not add any init script for mounting /dev, yet it is still created
and populated? But only with a very limited set of devices.
I guess there is a difference when the kernel mounts the cpio.xz image
compared to expanding the ext2.gz in RAM with respect to how /dev is
handled. But I am only guessing here.

I'm running a slow build of poky-tiny on an overloaded machine... Maybe
I can get back to those question in a while.

Post by Hans Beckérus
Another thing, You say CONFIG_DEVTMPFS_MOUNT does *not* automount /dev
if using the initrd scheme?
But I think it did when using the compressed ext2 file system? How
come /dev was mounted and populated automatically in the ext2 case but
when we changed to cpio.xz things seem to break?

This one can I at least answer.

It's because that's how devmtpfs is implemented in the kernel.

"This option does not affect initramfs based booting, here the devtmpfs
filesystem always needs to be mounted manually after the roots is
mounted." quote taken from the help text of CONFIG_DEVMTPFS_MOUNT, see
your kernel tree or e.g.
http://cateee.net/lkddb/web-lkddb/DEVTMPFS_MOUNT.html

Cheers,
Anders

[Hans Beckérus]

Post by Anders Darander

Post by Hans Beckérus

Post by Anders Darander

Post by Anders Darander

Post by Hans Beckérus
Things looked ok for a while, /dev was populated properly after boot
and all necessary file systems / mount points were created.
We were using an ext2 fs in RAM for mounting /.
Now, we made a change to instead use a CPIO image compressed using xz
and enabled the support in kernel to handle this.
This is when our problems started :( Suddenly our system booted with
just a very minimalistic /dev folder, containing basically only a few
of the devices probed at boot time.
- how was /dev populated before when there was no /etc/init.d/mdev?

I haven't checked poky-tiny, or more specifically the kernel
configurations for it. But based on the description that I cited above,
I'd guess that poky-tiny has
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
enabled.

CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y

Post by Anders Darander
That would mean the the kernel itself were managing /dev for you when
you were using ext2. Unfortunately, CONFIG_DEVTMPFS_MOUNT do only
automount devmptfs on /dev if you're not in an initramfs (or initrd).

Sounds reasonable. But, when moving from ext2.gz to cpio.xz, we still
did not add any init script for mounting /dev, yet it is still created
and populated? But only with a very limited set of devices.
I guess there is a difference when the kernel mounts the cpio.xz image
compared to expanding the ext2.gz in RAM with respect to how /dev is
handled. But I am only guessing here.

I'm running a slow build of poky-tiny on an overloaded machine... Maybe
I can get back to those question in a while.

No problem. Take your time ;)

Post by Anders Darander

Post by Hans Beckérus
Another thing, You say CONFIG_DEVTMPFS_MOUNT does *not* automount /dev
if using the initrd scheme?
But I think it did when using the compressed ext2 file system? How
come /dev was mounted and populated automatically in the ext2 case but
when we changed to cpio.xz things seem to break?

This one can I at least answer.
It's because that's how devmtpfs is implemented in the kernel.
"This option does not affect initramfs based booting, here the devtmpfs
filesystem always needs to be mounted manually after the roots is
mounted." quote taken from the help text of CONFIG_DEVMTPFS_MOUNT, see
your kernel tree or e.g.
http://cateee.net/lkddb/web-lkddb/DEVTMPFS_MOUNT.html

Well, I do not think this is absolutely true, or I have misunderstood something.
But we do not *use* the initramfs scheme, we use initrd in both ext2
and cpio case.
So, what from what I can tell we use the initrd scheme and /sbin/init.
And if CONFIG_DEVMTPFS_MOUNT is only affected the initramfs scheme
then it should work similar for initrd for both the ext2 and the cpio
case, which obviously is not the case.

Post by Anders Darander
Cheers,
Anders
--
Anders Darander
ChargeStorm AB / eStorm AB