[yocto] [opkg-utils PATCH] opkg-make-index: use ctime instead of mtime

[Stefan Agner]

From: Stefan Agner <***@toradex.com>

When using sstate, two parallel builds can produce two packages
with the same mtime but different checksums. When later one of
those two builds fetches the others ipk, the package index does
not get udpated properly (since mtime matches). This ends up with
messages such as:
Downloading file:/../tmp/work/../image/...ipk.
Removing corrupt package file /../sysroot/../var/cache/opkg/volatile/...ipk

However, in that case, ctime is different. Use ctime instead of
mtime to prevent failures like this.

Suggested-by: Khem Raj <***@gmail.com>
Signed-off-by: Stefan Agner <***@toradex.com>
---
This addresses the issue discussed here:
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156348.html

opkg-make-index | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/opkg-make-index b/opkg-make-index
index 3227fc0..db7bf64 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -115,12 +115,12 @@ for abspath in files:
pkg = None
fnameStat = os.stat(abspath)
if filename in old_pkg_hash:
- if filename in pkgsStamps and int(fnameStat.st_mtime) == pkgsStamps[filename]:
+ if filename in pkgsStamps and int(fnameStat.st_ctime) == pkgsStamps[filename]:
if (verbose):
sys.stderr.write("Found %s in Packages\n" % (filename,))
pkg = old_pkg_hash[filename]
else:
- sys.stderr.write("Found %s in Packages, but mtime differs - re-reading\n" % (filename,))
+ sys.stderr.write("Found %s in Packages, but ctime differs - re-reading\n" % (filename,))

if not pkg:
if (verbose):
@@ -137,7 +137,7 @@ for abspath in files:
else:
old_filename = ""
s = packages.add_package(pkg, opt_a)
- pkgsStamps[filename] = fnameStat.st_mtime
+ pkgsStamps[filename] = fnameStat.st_ctime
if s == 0:
if old_filename:
# old package was displaced by newer

[r***@linuxfoundation.org]

Post by Stefan Agner
When using sstate, two parallel builds can produce two packages
with the same mtime but different checksums. When later one of
those two builds fetches the others ipk, the package index does
not get udpated properly (since mtime matches). This ends up with
Downloading file:/../tmp/work/../image/...ipk.
Removing corrupt package file
/../sysroot/../var/cache/opkg/volatile/...ipk
However, in that case, ctime is different. Use ctime instead of
mtime to prevent failures like this.

Acked-by: Richard Purdie <***@linuxfoundation.org>

(I've been following this problem and this makes sense to me)

Post by Stefan Agner
---

http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156348.html

Post by Stefan Agner
opkg-make-index | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/opkg-make-index b/opkg-make-index
index 3227fc0..db7bf64 100755
--- a/opkg-make-index
+++ b/opkg-make-index
pkg = None
fnameStat = os.stat(abspath)
sys.stderr.write("Found %s in Packages\n" %
(filename,))
pkg = old_pkg_hash[filename]
- sys.stderr.write("Found %s in Packages, but mtime
differs - re-reading\n" % (filename,))
+ sys.stderr.write("Found %s in Packages, but ctime
differs - re-reading\n" % (filename,))
old_filename = ""
s = packages.add_package(pkg, opt_a)
- pkgsStamps[filename] = fnameStat.st_mtime
+ pkgsStamps[filename] = fnameStat.st_ctime
# old package was displaced by newer

--

[Khem Raj]

Post by r***@linuxfoundation.org

Post by Stefan Agner
When using sstate, two parallel builds can produce two packages
with the same mtime but different checksums. When later one of
those two builds fetches the others ipk, the package index does
not get udpated properly (since mtime matches). This ends up with
Downloading file:/../tmp/work/../image/...ipk.
Removing corrupt package file
/../sysroot/../var/cache/opkg/volatile/...ipk
However, in that case, ctime is different. Use ctime instead of
mtime to prevent failures like this.

Yep, thats for following up with implementing it.

Post by r***@linuxfoundation.org
(I've been following this problem and this makes sense to me)

Post by Stefan Agner
---

http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156348.html

Post by Stefan Agner
opkg-make-index | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/opkg-make-index b/opkg-make-index
index 3227fc0..db7bf64 100755
--- a/opkg-make-index
+++ b/opkg-make-index
pkg = None
fnameStat = os.stat(abspath)
- if filename in pkgsStamps and int(fnameStat.st_mtime) ==
+ if filename in pkgsStamps and int(fnameStat.st_ctime) ==
sys.stderr.write("Found %s in Packages\n" %
(filename,))
pkg = old_pkg_hash[filename]
- sys.stderr.write("Found %s in Packages, but mtime
differs - re-reading\n" % (filename,))
+ sys.stderr.write("Found %s in Packages, but ctime
differs - re-reading\n" % (filename,))
old_filename = ""
s = packages.add_package(pkg, opt_a)
- pkgsStamps[filename] = fnameStat.st_mtime
+ pkgsStamps[filename] = fnameStat.st_ctime
# old package was displaced by newer

--

[Alejandro Del Castillo]

This post might be inappropriate. Click to display it.

[Stefan Agner]

Hi Alejandro,

Post by Alejandro Del Castillo
makes sense, sounds like this is going to fix a bunch of nasty
intermittent failures, thanks!
merged

Thanks for merging!

With the merge in opkg-utils this is not yet actively used in OE of
course. So my question: Is there a new release of opkg-utils planned
anytime soon? Or should that be added as a patch in the OE recipe for
now?

--
Stefan

Post by Alejandro Del Castillo

Post by Stefan Agner
When using sstate, two parallel builds can produce two packages
with the same mtime but different checksums. When later one of
those two builds fetches the others ipk, the package index does
not get udpated properly (since mtime matches). This ends up with
Downloading file:/../tmp/work/../image/...ipk.
Removing corrupt package file /../sysroot/../var/cache/opkg/volatile/...ipk
However, in that case, ctime is different. Use ctime instead of
mtime to prevent failures like this.
---
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openembedded.org_pipermail_openembedded-2Dcore_2018-2DOctober_156348.html&d=DwIBaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=wNcrL2akRn6jfxhHaKavUrJB_C9JAMXtynjLd8ZzgXQ&m=Innit37H69hUyZPGuuhwO6R5CbUNNTfXQwxbqsEA2NE&s=oFvqASrFTgasDqZ901HeIBFSsf6Cn4FcBieOOBU4MdI&e=
opkg-make-index | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/opkg-make-index b/opkg-make-index
index 3227fc0..db7bf64 100755
--- a/opkg-make-index
+++ b/opkg-make-index
pkg = None
fnameStat = os.stat(abspath)
sys.stderr.write("Found %s in Packages\n" % (filename,))
pkg = old_pkg_hash[filename]
- sys.stderr.write("Found %s in Packages, but mtime differs - re-reading\n" % (filename,))
+ sys.stderr.write("Found %s in Packages, but ctime differs - re-reading\n" % (filename,))
old_filename = ""
s = packages.add_package(pkg, opt_a)
- pkgsStamps[filename] = fnameStat.st_mtime
+ pkgsStamps[filename] = fnameStat.st_ctime
# old package was displaced by newer

--
Cheers,
Alejandro

--

[Alejandro Del Castillo]

Post by Stefan Agner
Hi Alejandro,

Post by Alejandro Del Castillo
makes sense, sounds like this is going to fix a bunch of nasty
intermittent failures, thanks!
merged

Thanks for merging!

np

Post by Stefan Agner
With the merge in opkg-utils this is not yet actively used in OE of
course. So my question: Is there a new release of opkg-utils planned
anytime soon? Or should that be added as a patch in the OE recipe for
now?

I plan to release opkg & opkg-utils, version 0.4.0 on mid-December.
Maybe its worth adding the patch now?