Question about rebuilding RPM package index for updated RPMs when bitbake completes

Discussion:

Alex J Lennon

2014-04-20 12:15:53 UTC

Hi,

I'm trying to put in place a development workflow using the PR server,
RPM package feeds and smart update/install on a target.

I see that when I modify and rebuild my recipe, foo, the PR server
increments its count within the RPM filename, but the RPM feed data
doesn't seem to update.

e.g. in my deploy/rpm/cortexa9_vfp_neon folder I see a change (say) from

foo-1.7.11-r0.1.cortexa9_vfp_neon.rpm

to

foo-1.7.11-r0.2.cortexa9_vfp_neon.rpm

However if I then run smart on the target (configured to use that rpm
package feed) I don't see new packages

smart update

Similarly if I remove the package and reinstall it I get an error

smart remove foo
smart install foo

Fetching packages...
-> http://pkgrepo.my.local/.../foo-1.7.11-r0.1.cortexa9_vfp_neon.rpm
foo-1.7.11-r0.1... [ 0%]

error: Failed to download packages:
error:
http://pkgrepo.my.local/cortexa9_vfp_neon/foo-1.7.11-r0.1.cortexa9_vfp_neon.rpm:
File not found

I believe this is because the old package has been removed, and the new
package added to the folder,
but the repodata has not been rebuilt, leaving the packages out of sync
with the feed representation.

If then try running

bitbake package-index

When I now run an update on the target it picks up the new package and I
can fetch and install this

smart update

Updating cache... ########################################
[100%]

Channels have 3 new packages:
foo-1.7.11-***@cortexa9_vfp_neon
...

So if I am understanding the above correctly, when I make a change to a
recipe and build it, PR automatically
updates, the old RPM is removed and the new RPM added to the feed
directory. However the package index
for the feed is not updated.

So if at that point I try to make use of the feed on a target I am
likely to find something is broken.

If that is true would it make sense to leave the old RPM in the feed
directory until package-index
is re-ran, or to run package-index automatically at the end of a build
when RPMs have changed?

Thanks,

Alex

--

Mark Hatle

2014-04-22 01:36:28 UTC

Permalink

Post by Alex J Lennon
Hi,
I'm trying to put in place a development workflow using the PR server,
RPM package feeds and smart update/install on a target.
I see that when I modify and rebuild my recipe, foo, the PR server
increments its count within the RPM filename, but the RPM feed data
doesn't seem to update.

...

Post by Alex J Lennon
So if I am understanding the above correctly, when I make a change to a
recipe and build it, PR automatically
updates, the old RPM is removed and the new RPM added to the feed
directory. However the package index
for the feed is not updated.
So if at that point I try to make use of the feed on a target I am
likely to find something is broken.

The feed is normally indexed (createrepo) either when you manually run the
package-index operation, or when you construct a filesystem. Until you do that,
the feed directories are transient.

Post by Alex J Lennon
If that is true would it make sense to leave the old RPM in the feed
directory until package-index
is re-ran, or to run package-index automatically at the end of a build
when RPMs have changed?

I -never- export the feed directories from the project directory. Instead, I
copy the packages from the feed directory to where I share them, and then run
the indexer against the external repository. This preserves the older versions
and also makes the new ones available.

To run the indexer I have to configure and run it manually...

PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH
/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo
--update -q <path to feed>

So for qemux86_64, I end up running the above three times. all, x86_64 and
qemux86_64.

Then on the target I just do:

smart update
smart upgrade -y

--Mark

Post by Alex J Lennon
Thanks,
Alex

Alex J Lennon

2014-04-22 06:50:20 UTC

Permalink

Post by Mark Hatle

...

The feed is normally indexed (createrepo) either when you manually run
the package-index operation, or when you construct a filesystem.
Until you do that, the feed directories are transient.

Is there a reason for this? Surely having a package index that is out of
sync with the packages until a manual operation is performed isn't ideal?

Post by Mark Hatle

I -never- export the feed directories from the project directory.
Instead, I copy the packages from the feed directory to where I share
them, and then run the indexer against the external repository. This
preserves the older versions and also makes the new ones available.
To run the indexer I have to configure and run it manually...
PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH
/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo
--update -q <path to feed>
So for qemux86_64, I end up running the above three times. all,
x86_64 and qemux86_64.
smart update
smart upgrade -y

Thanks, that's useful

However I still believe the core question stands, which is why the
package index is out of sync, and why it's a good thing to have to run
bitbake package-index manually to bring the package index back into sync
with the packages?

I too export the feed directories to a server for 3rd party consumption
via compressed rsync over SSH. I don't want to expose more on the server
than a simple SSH endpoint, and rsync seems a sensible way to minimise
copying times, so either I need the package index to be correct prior to
the rsync, I need to export, say, an NFS filesystem which I don't want
to do, or I need some createrepo specifics on what I would prefer to be
a dumb web-server.

Best,

Alex

Post by Mark Hatle
--Mark

Post by Alex J Lennon
Thanks,
Alex

--
Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>

Alex J Lennon / Director
1 Queensway, Liverpool L22 4RA

mobile: +44 (0)7956 668178

Linkedin <http://www.linkedin.com/in/alexjlennon> Skype
<skype:alexjlennon?add>

This e-mail message may contain confidential or legally privileged
information and is intended only for the use of the intended
recipient(s). Any unauthorized disclosure, dissemination, distribution,
copying or the taking of any action in reliance on the information
herein is prohibited. E-mails are not secure and cannot be guaranteed to
be error free as they can be intercepted, amended, or contain viruses.
Anyone who communicates with us by e-mail is deemed to have accepted
these risks. Company Name is not responsible for errors or omissions in
this message and denies any responsibility for any damage arising from
the use of e-mail. Any opinion and other statement contained in this
message and any attachment are solely those of the author and do not
necessarily represent those of the company.

Mark Hatle

2014-04-22 18:14:34 UTC

Permalink

Post by Mark Hatle

...

The feed is normally indexed (createrepo) either when you manually run the
package-index operation, or when you construct a filesystem. Until you do
that, the feed directories are transient.

Is there a reason for this? Surely having a package index that is out of sync
with the packages until a manual operation is performed isn't ideal?

Yes, if you re-indexed the set of packages after each package, you will add
multiple minutes to the construction of each package, and introduce locking into
the system. (You won't be able to generate multiple recipes in parallel due to
this.) (I don't consider it a package feed until all of the packages are built
and the index is constructed.)

Post by Mark Hatle

I -never- export the feed directories from the project directory. Instead, I
copy the packages from the feed directory to where I share them, and then run
the indexer against the external repository. This preserves the older
versions and also makes the new ones available.
To run the indexer I have to configure and run it manually...
PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH
/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo
--update -q <path to feed>
So for qemux86_64, I end up running the above three times. all, x86_64 and
qemux86_64.
smart update
smart upgrade -y

Thanks, that's useful
However I still believe the core question stands, which is why the package index
is out of sync, and why it's a good thing to have to run bitbake package-index

The other option is to remove the repo indexes when packages are written.
Again, I don't consider these out of sync though, as the files have different
purposes and constraints. They are simply constructed to enable the rootfs
generation. Upgrades are external of the build system.

manually to bring the package index back into sync with the packages?
I too export the feed directories to a server for 3rd party consumption via
compressed rsync over SSH. I don't want to expose more on the server than a
simple SSH endpoint, and rsync seems a sensible way to minimise copying times,
so either I need the package index to be correct prior to the rsync, I need to
export, say, an NFS filesystem which I don't want to do, or I need some
createrepo specifics on what I would prefer to be a dumb web-server.

For an rsync, I would either index on the server using a cron-job.. so when new
packages appear they get added to the index, or have a local mirror of the
directory including the updated index... you just want to make sure the index
files are synced last somehow.

--Mark

Best,
Alex

Post by Mark Hatle
--Mark

Post by Alex J Lennon
Thanks,
Alex

--
Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>
Alex J Lennon / Director
1 Queensway, Liverpool L22 4RA
mobile: +44 (0)7956 668178
Linkedin <http://www.linkedin.com/in/alexjlennon> Skype <skype:alexjlennon?add>
This e-mail message may contain confidential or legally privileged information
and is intended only for the use of the intended recipient(s). Any unauthorized
disclosure, dissemination, distribution, copying or the taking of any action in
reliance on the information herein is prohibited. E-mails are not secure and
cannot be guaranteed to be error free as they can be intercepted, amended, or
contain viruses. Anyone who communicates with us by e-mail is deemed to have
accepted these risks. Company Name is not responsible for errors or omissions in
this message and denies any responsibility for any damage arising from the use
of e-mail. Any opinion and other statement contained in this message and any
attachment are solely those of the author and do not necessarily represent those
of the company.

Alex J Lennon

2014-04-22 20:03:07 UTC

Permalink

Post by Mark Hatle

...

The feed is normally indexed (createrepo) either when you manually run the
package-index operation, or when you construct a filesystem. Until you do
that, the feed directories are transient.

Is there a reason for this? Surely having a package index that is out of sync
with the packages until a manual operation is performed isn't ideal?

Yes, if you re-indexed the set of packages after each package, you
will add multiple minutes to the construction of each package, and
introduce locking into the system. (You won't be able to generate
multiple recipes in parallel due to this.) (I don't consider it a
package feed until all of the packages are built and the index is
constructed.)

I wasn't suggesting that the index is built mid-run . Not knowing the
internals of the bitbake dependency chain, presumably the package index
would/could be rebuilt at the end of the build process and so would not
block parallelisation (which I agree would be a time-sink)

Post by Mark Hatle

I -never- export the feed directories from the project directory. Instead, I
copy the packages from the feed directory to where I share them, and then run
the indexer against the external repository. This preserves the older
versions and also makes the new ones available.
To run the indexer I have to configure and run it manually...
PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH
/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo
--update -q <path to feed>
So for qemux86_64, I end up running the above three times. all, x86_64 and
qemux86_64.
smart update
smart upgrade -y

Thanks, that's useful
However I still believe the core question stands, which is why the package index
is out of sync, and why it's a good thing to have to run bitbake package-index

The other option is to remove the repo indexes when packages are
written. Again, I don't consider these out of sync though, as the
files have different purposes and constraints. They are simply
constructed to enable the rootfs generation. Upgrades are external of
the build system.

Fair comment. I take the view that these feeds are available, could
easily be relied upon not to be transient, and I'd like to use them
locally to speed up my development workflow. I think other users would too.

Much of the value of the build environment is the automation. If bitbake
can do a task that means I don't have to remember to do it, and I don't
have to explain to somebody else that they need to do it, then that
seems like a win to me (even if that is an option I can turn on).

Post by Mark Hatle

manually to bring the package index back into sync with the packages?
I too export the feed directories to a server for 3rd party
consumption via
compressed rsync over SSH. I don't want to expose more on the server than a
simple SSH endpoint, and rsync seems a sensible way to minimise copying times,
so either I need the package index to be correct prior to the rsync, I need to
export, say, an NFS filesystem which I don't want to do, or I need some
createrepo specifics on what I would prefer to be a dumb web-server.

For an rsync, I would either index on the server using a cron-job.. so
when new packages appear they get added to the index, or have a local
mirror of the directory including the updated index... you just want
to make sure the index files are synced last somehow.

To me the server side customisation brings in in extra work that
shouldn't be needed in all cases, (although will undoubtely be needed in
some cases).

I would much prefer not to have to worry about it.

If the package feed could be relied upon to be in sync with the contents
of the tmp/deploy/rpm/* tree at the end of a bitbake build then the
problem simply goes away, potentially at the cost of a little lost time
during the build.

Understanding that other use-cases will be different from mine I have
been wondering today,

- Can I add something into local.conf that would add a dependency to
cause the package-index task to run at the end of the bitbake build
(preferably only if the contents of the package directories have changed?

- Does the build environment have the concept of post-build triggers?
i.e. Understanding that I could script something myself, is there a
recommended
method for defining a post-build trigger script for bitbake to run on
certain conditions e.g to do the the package reindex and feed upload in
this case?

Thanks,

Alex

Post by Mark Hatle
--Mark

Best,
Alex

Post by Mark Hatle
--Mark

Post by Alex J Lennon
Thanks,
Alex

--
Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>
Alex J Lennon / Director
1 Queensway, Liverpool L22 4RA
mobile: +44 (0)7956 668178
Linkedin <http://www.linkedin.com/in/alexjlennon> Skype
<skype:alexjlennon?add>
This e-mail message may contain confidential or legally privileged information
and is intended only for the use of the intended recipient(s). Any unauthorized
disclosure, dissemination, distribution, copying or the taking of any action in
reliance on the information herein is prohibited. E-mails are not secure and
cannot be guaranteed to be error free as they can be intercepted, amended, or
contain viruses. Anyone who communicates with us by e-mail is deemed to have
accepted these risks. Company Name is not responsible for errors or omissions in
this message and denies any responsibility for any damage arising from the use
of e-mail. Any opinion and other statement contained in this message and any
attachment are solely those of the author and do not necessarily represent those
of the company.