[yocto] QA cycle report for 2.6 M4 RC1

Post by Jain, Sangeeta

Hello All,

Thank you to the Intel and Wind QA teams for performing these tasks.

Will the Build appliance failure keep it from functioning ?

- armin

Post by Jain, Sangeeta
This is the full report for 2.6 M4 RC1:
https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1

*Summary*

All planned tests were executed.

Total Test Executed 3339
Passed Test 3322
Failed Test 7
Blocked Test - 4

There were zero high priority defect. Team had found 4 new defects.
ptest for 1modulefailed in current release but passed in previous 2.6
M3 rc1. For Openssl no test was executed in current release as well
as previous release. Present status of bugs for respective modules is

ModuleName BugId Present Status

gstreamer 12990 - New

Note: For busybox, pass rate is lower than previous release, but no
test cases which passes in previous release failed in current release.
Lower pass rate is due to new test cases added in this release. No bug
filed.

*Performance test*

rootfs performance on ubuntu1604 upgraded by 13.04% in 2.6 M3 RC1
w.r.t. 2.6 M2 RC1

*QA-Hints *
* *
For performance test, in this release, QA team has performed analysis
on results from yocto-perf mailing list, rather than on results from
Linux foundation machines.
Weobserved two major difference between machines used to run
Performance test by yocto-perf mailing list and Linux Foundation

1) Mailing List data points was not constant, sometime more, sometime less
2) Mailing List used machine with different spec
* *
*New Bugs*
* *
[1] Bug 12974 - [2.6 M4 RC1] [OE-Core] Crosstap doesn't work on 2.6
M4 RC1
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12974

[2] Bug 129
<https://bugzilla.yoctoproject.org/show_bug.cgi?id=12952>79 - [2.6 M4
rc1] test_recipetool_create_cmake failed on Fedora 27
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12979
* *
[3] Bug 129
<https://bugzilla.yoctoproject.org/show_bug.cgi?id=12946>91 - [2.6 M4
RC1][Build-Appliance] Bitbake build-appliance-image getting failed
during building image due to webkitgtk package
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12991

[4] Bug 129
<https://bugzilla.yoctoproject.org/show_bug.cgi?id=12954>92 - [2.6 M4
rc1] test_devtool_add_fetch_git failed on Fedora 27
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12992

Thanks & Regards,
Sangeeta Jain

r***@linuxfoundation.org

2018-11-10 16:25:50 UTC

Post by Jain, Sangeeta
https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1

Thanks Sangeeta and team!

Now we have the QA report for YP 2.6 M4 rc1 (Final 2.6) we need to make
a release go or nogo decision. To do this we have the following:

QA Report: https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
Release Criteria: https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.6_Status#Milestone_4.2FFinal_-_Target_Oct._26.2C_2018

We'd be happy to take representations from members and the community to
help reach that decision.

My personal view is that whilst there are a number of issues present in
rc1, we should release it, collect up fixes on the thud branch (aleady
happening) and plan on a 2.6.1 as soon as it looks like we have enough
critical mass behind those as opposed to an rc2 and further delays to
the release.

Cheers,

Richard

--

Randy MacLeod

2018-11-10 16:31:27 UTC

Post by Jain, Sangeeta
https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1

Thanks Sangeeta and team!
Now we have the QA report for YP 2.6 M4 rc1 (Final 2.6) we need to make
QA Report: https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
Release Criteria: https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.6_Status#Milestone_4.2FFinal_-_Target_Oct._26.2C_2018
We'd be happy to take representations from members and the community to
help reach that decision.
My personal view is that whilst there are a number of issues present in
rc1, we should release it, collect up fixes on the thud branch (aleady
happening) and plan on a 2.6.1 as soon as it looks like we have enough
critical mass behind those as opposed to an rc2 and further delays to
the release.

Despite the fact that there are a few release criteria that are not
in the 'Done' state yet, I approve of releasing YP-2.6M4 on the
condition that the Docs and Web site criteria are taken care of
before release.

I have reviewed the open bugs. Several are resolved or will be soon and
the ones that remain appear to be either limited in impact such
as 12974/systemtap or are likely due to builder problems such
as 12991/webkitgtk on the build appliance.

Build time tests have crept up somewhat for the rootfs and eSDK
tests but not so dramatically that I would suggest blocking GA.
We haven't had anyone investigate the root cause yet AFAIK but
that can happen post-release and noted in the release notes.

The package update status is unknown due to the tracker being down
but in M3 we were at 81% done so we're still in a good albeit
unquantified state.

What are the plans for the Documentation checks and Wiki/Web
site update? That needs to be 'Done' but I expect it will
be taken care of in the coming days.

../Randy

Post by r***@linuxfoundation.org
Cheers,
Richard

--
# Randy MacLeod
# Wind River Linux
--

akuster808

2018-11-10 18:02:50 UTC

Post by Jain, Sangeeta
https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1

Regarding.

*BugÂ 12991* <https://bugzilla.yoctoproject.org/show_bug.cgi?id=12991>
-[2.6 M4 RC1][Build-Appliance] Bitbake build-appliance-image getting
failed during building image due to webkitgtk package

Does it mean the Build-Appliance is non functioning ?Â It was broken at
the Sumo release time as well. Should it be dropped as the release criteria?

- Armin

Post by r***@linuxfoundation.org
My personal view is that whilst there are a number of issues present in
rc1, we should release it, collect up fixes on the thud branch (aleady
happening) and plan on a 2.6.1 as soon as it looks like we have enough
critical mass behind those as opposed to an rc2 and further delays to
the release.
Cheers,
Richard

r***@linuxfoundation.org

2018-11-11 10:21:52 UTC

Post by Jain, Sangeeta
https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1

Regarding.
Bug 12991 - [2.6 M4 RC1][Build-Appliance] Bitbake build-appliance-
image getting failed during building image due to webkitgtk package
Does it mean the Build-Appliance is non functioning ? It was broken
at the Sumo release time as well. Should it be dropped as the release
criteria?

Build-appliance is a tricky test as it tests multiple things, roughly:

* vmdk images under vmware
* the web browser
* toaster
* whether we can self host (build poky within poky)

The fact the webkit recipe failed to build may be due to several
reasons:

* random race type condition
* lack of memory in the VM
* phase of the moon
* other things

I'm not convinced its a release blocker, or that it invalidates the b-a
test, or that it would even reproduce. If it does reproduce that would
be more interesting and easier to debug.

I do think we're going to have to split up the b-a test in 2.7 so that
even if its not manual QA'd, we can automatically test some pieces of
what it covers.

Cheers,

Richard

--

akuster808

2018-11-11 21:18:42 UTC

Post by Jain, Sangeeta
https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1

* vmdk images under vmware
* the web browser
* toaster
* whether we can self host (build poky within poky)
The fact the webkit recipe failed to build may be due to several
* random race type condition
* lack of memory in the VM
* phase of the moon
* other things
I'm not convinced its a release blocker, or that it invalidates the b-a
test, or that it would even reproduce. If it does reproduce that would
be more interesting and easier to debug.

ok, but is the b-a functional in 2.6?

- armin

Post by r***@linuxfoundation.org
I do think we're going to have to split up the b-a test in 2.7 so that
even if its not manual QA'd, we can automatically test some pieces of
what it covers.
Cheers,
Richard

r***@linuxfoundation.org

2018-11-11 21:24:40 UTC

https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1

Post by r***@linuxfoundation.org
Thanks Sangeeta and team!
Now we have the QA report for YP 2.6 M4 rc1 (Final 2.6) we need to make
a release go or nogo decision. To do this we have the
https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.6_Status#Milestone_4.2FFinal_-_Target_Oct._26.2C_2018
We'd be happy to take representations from members and the community to
help reach that decision.

Regarding.
Bug 12991 - [2.6 M4 RC1][Build-Appliance] Bitbake build-
appliance-
image getting failed during building image due to webkitgtk package
Does it mean the Build-Appliance is non functioning ? It was broken
at the Sumo release time as well. Should it be dropped as the release
criteria?

ok, but is the b-a functional in 2.6?

It booted under VMWare, I believe the web browser was functional and it
managed a build to a point somewhere in webkitgtk which was the only
failure in the build.

I suspect but have little evidence that the failure was a race or
system resource issue rather than a functionality problem with b-a.
Does that make b-a functional? In my view, yes, your view may vary.

Cheers,

Richard

--

akuster808

2018-11-11 21:38:28 UTC

Post by Jain, Sangeeta
https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1

Regarding.
Bug 12991 - [2.6 M4 RC1][Build-Appliance] Bitbake build-
appliance-
image getting failed during building image due to webkitgtk
package
Does it mean the Build-Appliance is non functioning ? It was broken
at the Sumo release time as well. Should it be dropped as the release
criteria?

ok, but is the b-a functional in 2.6?

It booted under VMWare, I believe the web browser was functional and it
managed a build to a point somewhere in webkitgtk which was the only
failure in the build.
I suspect but have little evidence that the failure was a race or
system resource issue rather than a functionality problem with b-a.
Does that make b-a functional? In my view, yes, your view may vary.

Nope. I am good. If it is functional I am good with it. We can address
the other issues later.

thanks,

Armin

Post by Jain, Sangeeta
Cheers,
Richard

Burton, Ross

2018-11-12 16:12:45 UTC

I'd suggest we schedule 2.6.1 for about month after release, there's
quite a queue of security fixes already.

With that caveat, I'm fine with this plan.

Ross
--

akuster808

2018-11-12 16:20:04 UTC

Post by Burton, Ross

I'd suggest we schedule 2.6.1 for about month after release, there's
quite a queue of security fixes already.

Maybe shoot for before the Dec Holidays instead of a fixed period after
2.6 releases, a nice way to end the year : )

- armin

Post by Burton, Ross
With that caveat, I'm fine with this plan.
Ross

Burton, Ross

2018-11-12 17:08:24 UTC