[yocto] QA cycle report for 2.4.4 RC1

[Jain, Sangeeta]

Hello All,



This is the full report for 2.4.4 RC1:

https://wiki.yoctoproject.org/wiki/WW46_-_2018-11-14_-_Full_Test_Cycle_2.4.4_RC1





Summary



All planned tests were executed.



Total Test Executed - 3330

Passed Test - 3318

Failed Test - 8

Blocked Test - 4



There were zero high priority defect. Team had found 2 new defects.



New Bugs



[1] Bug 13033 - [2.4.4RC1] [Bitbake] [Case 142] PR number is not getting increased with remote server/local client mode

https://bugzilla.yoctoproject.org/show_bug.cgi?id=13033



[2]Bug 13038 - [2.4.4RC1][Package Management][Getting Error Message "Failed to synchronize cache for repo 'repository'"]

https://bugzilla.yoctoproject.org/show_bug.cgi?id=13038



Thanks & Regards,
Sangeeta Jain

[r***@linuxfoundation.org]

Post by Jain, Sangeeta
Hello All,

https://wiki.yoctoproject.org/wiki/WW46_-_2018-11-14_-_Full_Test_Cycle_2.4.4_RC1

Post by Jain, Sangeeta
Summary
All planned tests were executed.
Total Test Executed – 3330
Passed Test – 3318
Failed Test – 8
Blocked Test - 4
There were zero high priority defect. Team had found 2 new defects.
New Bugs
[1] Bug 13033 - [2.4.4RC1] [Bitbake] [Case 142] PR number is not
getting increased with remote server/local client mode
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13033
[2]Bug 13038 - [2.4.4RC1][Package Management][Getting Error Message
"Failed to synchronize cache for repo 'repository'"]
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13038

Thanks for running the QA for this. I do have some
questions/observations.

Firstly, this report is a little misleading as there are only two bugs
mentioned but 8 failures. The full report shows other bugs which were
for example reopened or already open.

Taking the above bugs first, I believe they're both manual versions of
tests which are already automated. I believe the automated tests have
passed and there appears to be some kind of problem with the manual
execution such as the wrong process being documented. I'm not quite
sure why these are being run manually? Was the list of manual tests we
received from Intel incorrect?

Looking at other bugs in the QA report:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12670

It was reopened as it had been fixed in sumo onwards but not rocko.
I've backported the fix to the rocko branch. I don't believe its a
release blocker.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12991

This is webkitgtk failing to build in the build-appliance and looks
entirely related to resources in the VM. How this could have worked in
the past yet fail now makes me wonder about whether it was in fact
tested previously, whether the VM configuration changed or quite what
happened. I don't believe its a blocking bug as it does seem to be a VM
resource problem, not a real failure or problem with build-appliance.
Its hard to tell for sure with the limited data we have though.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12240

A reopened lsb testing bug but its not really showing problems with the
actual image, just testing difficulties.


Based on this, I think 2.4.4 is good to be released, I don't see any
blocking issues. I would like to understand what tests are actually
being run and why though.

Cheers,

Richard


--

[Jain, Sangeeta]

The test cases run for 2.4.4 RC1 are same as run for 2.4 release.

-----Original Message-----
Sent: Saturday, 24 November, 2018 12:07 AM
Subject: Re: QA cycle report for 2.4.4 RC1

Post by Jain, Sangeeta
Hello All,

https://wiki.yoctoproject.org/wiki/WW46_-_2018-11-14_-
_Full_Test_Cycle_2.4.4_RC1

Post by Jain, Sangeeta
Summary
All planned tests were executed.
Total Test Executed – 3330
Passed Test – 3318
Failed Test – 8
Blocked Test - 4
There were zero high priority defect. Team had found 2 new defects.
New Bugs
[1] Bug 13033 - [2.4.4RC1] [Bitbake] [Case 142] PR number is not
getting increased with remote server/local client mode
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13033
[2]Bug 13038 - [2.4.4RC1][Package Management][Getting Error Message
"Failed to synchronize cache for repo 'repository'"]
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13038

Thanks for running the QA for this. I do have some questions/observations.
Firstly, this report is a little misleading as there are only two bugs mentioned but 8
failures. The full report shows other bugs which were for example reopened or
already open.

In report, only new bugs are mentioned. All the bugs which are reopened/already existing are not listed in report mail but can be seen in full report.
However, if its creating any confusion/inconvenience, going forward I can mention all the failing bugs in report.

Taking the above bugs first, I believe they're both manual versions of tests which
are already automated. I believe the automated tests have passed and there
appears to be some kind of problem with the manual execution such as the
wrong process being documented. I'm not quite sure why these are being run
manually? Was the list of manual tests we received from Intel incorrect?

The list manual test we run for 2.4.4 is same as we run for 2.4 release. This test case is automated later than 2.4 release and is still manual as per 2.4 test plan in Testopia. It's an irony that in Yocto Project we don't have any tracking of a test case converting into automated in which release, and no system that we can update test cases in Dot releases. I am trying to streamline the tracking of test case evolution, so as to avoid such scenarios in future.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12670
It was reopened as it had been fixed in sumo onwards but not rocko.
I've backported the fix to the rocko branch. I don't believe its a release blocker.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12991
This is webkitgtk failing to build in the build-appliance and looks entirely related
to resources in the VM. How this could have worked in the past yet fail now
makes me wonder about whether it was in fact tested previously, whether the
VM configuration changed or quite what happened. I don't believe its a blocking
bug as it does seem to be a VM resource problem, not a real failure or problem
with build-appliance.
Its hard to tell for sure with the limited data we have though.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12240
A reopened lsb testing bug but its not really showing problems with the actual
image, just testing difficulties.
Based on this, I think 2.4.4 is good to be released, I don't see any blocking issues.
I would like to understand what tests are actually being run and why though.
Cheers,
Richard

Thanks & Regards,
Sangeeta Jain

--

[r***@linuxfoundation.org]

Post by Jain, Sangeeta
The test cases run for 2.4.4 RC1 are same as run for 2.4 release.

Ok, but I'm not sure they're correct. Are the test cases we're talking
about also run for the 2.5 release? I'll explain more below.

Post by Jain, Sangeeta

Post by r***@linuxfoundation.org
Thanks for running the QA for this. I do have some
questions/observations.
Firstly, this report is a little misleading as there are only two bugs mentioned but 8
failures. The full report shows other bugs which were for example reopened or
already open.

In report, only new bugs are mentioned. All the bugs which are
reopened/already existing are not listed in report mail but can be
seen in full report.
However, if its creating any confusion/inconvenience, going forward I
can mention all the failing bugs in report.

I think it would be useful to have a complete summary of the bugs
found, in particular the reopened ones as otherwise it gives a
misleading view of the release status.

Post by Jain, Sangeeta

Post by r***@linuxfoundation.org
Taking the above bugs first, I believe they're both manual versions of tests which
are already automated. I believe the automated tests have passed and there
appears to be some kind of problem with the manual execution such as the
wrong process being documented. I'm not quite sure why these are being run
manually? Was the list of manual tests we received from Intel incorrect?

The list manual test we run for 2.4.4 is same as we run for 2.4
release. This test case is automated later than 2.4 release and is
still manual as per 2.4 test plan in Testopia. It's an irony that in
Yocto Project we don't have any tracking of a test case converting
into automated in which release, and no system that we can update
test cases in Dot releases. I am trying to streamline the tracking of
test case evolution, so as to avoid such scenarios in future.

Taking:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=13033

as an example, I believe the automated version is:

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/cases/prservice.py?h=rocko#n52

which was present in the first 2.4 release.

Its also present in pyro:

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/prservice.py?h=pyro#n54

and morty:
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/prservice.py?h=morty#n51
krogoth:
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/prservice.py?h=krogoth#n52
jethro:
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/prservice.py?h=jethro#n51

so has been there since at least 2.0.

So either I'm wrong about this being an automated version of the test,
or we really shouldn't be running this test manually but it isn't a
problem of changing test criteria between point releases.

I'd also note that nobody from QA has replied to my question in the
bugzilla.

I'm now worrying about what test cases get run for 2.5 and how these
differ from what we run against 2.6 and master and what QA summarised
in the recent documentation of test cases exercise. Perhaps we need to
document the manual test cases in the 2.5 release too to ensure we have
the right set?

Cheers,

Richard

--

[Jain, Sangeeta]

-----Original Message-----
Sent: Tuesday, 4 December, 2018 11:57 PM
Subject: Re: QA cycle report for 2.4.4 RC1

Post by Jain, Sangeeta
The test cases run for 2.4.4 RC1 are same as run for 2.4 release.

Ok, but I'm not sure they're correct. Are the test cases we're talking about also
run for the 2.5 release? I'll explain more below.

For 2.5 dot release, we'll run the test cases which we run for 2.5 master release.

Post by Jain, Sangeeta

Post by r***@linuxfoundation.org
Thanks for running the QA for this. I do have some
questions/observations.
Firstly, this report is a little misleading as there are only two
bugs mentioned but 8 failures. The full report shows other bugs
which were for example reopened or already open.

In report, only new bugs are mentioned. All the bugs which are
reopened/already existing are not listed in report mail but can be
seen in full report.
However, if its creating any confusion/inconvenience, going forward I
can mention all the failing bugs in report.

I think it would be useful to have a complete summary of the bugs found, in
particular the reopened ones as otherwise it gives a misleading view of the
release status.

Sure.

Post by Jain, Sangeeta

Post by r***@linuxfoundation.org
Taking the above bugs first, I believe they're both manual versions
of tests which are already automated. I believe the automated tests
have passed and there appears to be some kind of problem with the
manual execution such as the wrong process being documented. I'm not
quite sure why these are being run manually? Was the list of manual
tests we received from Intel incorrect?

The list manual test we run for 2.4.4 is same as we run for 2.4
release. This test case is automated later than 2.4 release and is
still manual as per 2.4 test plan in Testopia. It's an irony that in
Yocto Project we don't have any tracking of a test case converting
into automated in which release, and no system that we can update test
cases in Dot releases. I am trying to streamline the tracking of test
case evolution, so as to avoid such scenarios in future.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=13033
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/cases/prse
rvice.py?h=rocko#n52
which was present in the first 2.4 release.
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/prservice.
py?h=pyro#n54
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/prservice.
py?h=morty#n51
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/prservice.
py?h=krogoth#n52
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/prservice.
py?h=jethro#n51
so has been there since at least 2.0.
So either I'm wrong about this being an automated version of the test, or we
really shouldn't be running this test manually but it isn't a problem of changing
test criteria between point releases.
I'd also note that nobody from QA has replied to my question in the bugzilla.

For
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13033
test case has been run manually for 2.4 release as per Testopia data.
To answer your question in Bugzilla, we are working on it and will be updated asap.

I'm now worrying about what test cases get run for 2.5 and how these differ from
what we run against 2.6 and master and what QA summarised in the recent
documentation of test cases exercise. Perhaps we need to document the manual
test cases in the 2.5 release too to ensure we have the right set?

I am working on identifying the test cases which were automated in 2.6 and master. And the new test case document summarised by QA team should be aligned to latest state of test case, automated or manual.
I am also working on documentation of manual test cases for 2.5 and finding which ones are already automated in 2.6, so that we can execute the latest state in next release.

Cheers,
Richard

Thanks & Regards,
Sangeeta Jain

--