Discussion:
Autobuilder changes
(too old to reply)
Richard Purdie
2018-11-23 17:02:50 UTC
Permalink
I've take this quiet period as an opportunity to make some significant
autobuilder changes:

The targets have all been renamed and split up. "nightly-arm" became
"qemuarm" and "beaglebone". The "nightly-" prefixes were dropped.

The nightly target was dropped and replaced by "a-quick" and "a-full".
The difference are a number of targets which don't often find problems.
We're also ramping down on the amount of testing qemumips-lsb/qemuppc-
lsb get for example as they're more minor architectures as things
stand. The "a-" prefix is for sorting in the UI and to separate them
out from the mass of other builds.

There are now selftest targets per "distro" which can replace the
testing that Intel QA have been doing. This is only done for "a-full"
and the selftest on a distro at random is done by "a-quick".

Buildhistory is only being generated for the qemu* targets.

The "template" used for testing qemu or "real-hardware" targets was
standardised as there were weird differences for now good reason.

The structure of "non-release" builds on
https://autobuilder.yocto.io/pub/ was cleaned up and now appears under
a single date+number under the non-release directory which was how it
was meant to work. All the other spurious output that was there has
been cleaned up.

XML test results are now being published for all builds, e.g.:
https://autobuilder.yocto.io/pub/non-release/20181122-11/qemuarm64/

Ptest was added to "a-full" builds for arm and x86 although the arm
builds currently time out. The plan will be to enable arm when we have
an arm build server with KVM support in our rack (next week?). ptest
execution for x86 takes around 3 hours and some sample result output
is:

https://autobuilder.yocto.io/pub/non-release/20181122-11/qemux86-64-ptest/

"meta-oe" and "meta-virt" test targets were added for experimentation,
testing meta-openembedded and meta-virtualization respectively. These
are work in progress currently just doing a "world -k" build for
qemux86-64 which fails (they take around 2.5 hours).

We now run "a-quick" on master at 1am every day which will make
buildhistory more up to date and useful for the master branch and give
us better comparison data for -next and mut.

wine is installed onto the ubuntu1804 workers and the mingw testing was
split into a meta-mingw target, now with 32 and 64 bit build tests and
64 bit runtime tests. With master-next of meta-mingw this appears to
run the test. Huge kudos to Joshua Watt for putting that testing
together. If we install more onto the workers we could probably test 32
bit as well, I'm torn on that right now as its a lot more dependencies
on the workers.

Building older releases is now broken until I sort out the
corresponding helper changes for those branches but one thing at a
time!

If anyone would like to help with xml results file analysis tooling,
that is where we need work next to be able to summarise the ptest
results and do results comparison. I know Ee Peng has some work in this
area which I will be looking at next.

Cheers,

Richard



--
Richard Purdie
2018-11-23 17:19:58 UTC
Permalink
Post by Richard Purdie
I've take this quiet period as an opportunity to make some
I knew I was missing something:

Successful builds are now deleted when they complete using the
background deletion (clobberdir). This should reduce the clutter lying
around on the worker filesystems. It also means builds shouldn't be
contending with background deletion as much.

Failed builds have their build directory renamed to "build-renamed".
This means that the paths all become invalid and can't be accidentally
referenced from other builds.

I did a sweep through the workers deleting all the old build
directories and cleared a lot of the diskspace on them.

Cheers,

Richard

--
akuster
2018-11-24 14:40:18 UTC
Permalink
Post by Richard Purdie
I've take this quiet period as an opportunity to make some significant
The targets have all been renamed and split up. "nightly-arm" became
"qemuarm" and "beaglebone". The "nightly-" prefixes were dropped.
The nightly target was dropped and replaced by "a-quick" and "a-full".
The difference are a number of targets which don't often find problems.
We're also ramping down on the amount of testing qemumips-lsb/qemuppc-
lsb get for example as they're more minor architectures as things
stand. The "a-" prefix is for sorting in the UI and to separate them
out from the mass of other builds.
There are now selftest targets per "distro" which can replace the
testing that Intel QA have been doing. This is only done for "a-full"
and the selftest on a distro at random is done by "a-quick".
Buildhistory is only being generated for the qemu* targets.
The "template" used for testing qemu or "real-hardware" targets was
standardised as there were weird differences for now good reason.
The structure of "non-release" builds on
https://autobuilder.yocto.io/pub/ was cleaned up and now appears under
a single date+number under the non-release directory which was how it
was meant to work. All the other spurious output that was there has
been cleaned up.
https://autobuilder.yocto.io/pub/non-release/20181122-11/qemuarm64/
Ptest was added to "a-full" builds for arm and x86 although the arm
builds currently time out. The plan will be to enable arm when we have
an arm build server with KVM support in our rack (next week?). ptest
execution for x86 takes around 3 hours and some sample result output
https://autobuilder.yocto.io/pub/non-release/20181122-11/qemux86-64-ptest/
"meta-oe" and "meta-virt" test targets were added for experimentation,
testing meta-openembedded and meta-virtualization respectively. These
are work in progress currently just doing a "world -k" build for
qemux86-64 which fails (they take around 2.5 hours).
We now run "a-quick" on master at 1am every day which will make
What Time zone for 1am?
Post by Richard Purdie
buildhistory more up to date and useful for the master branch and give
us better comparison data for -next and mut.
wine is installed onto the ubuntu1804 workers and the mingw testing was
split into a meta-mingw target, now with 32 and 64 bit build tests and
64 bit runtime tests. With master-next of meta-mingw this appears to
run the test. Huge kudos to Joshua Watt for putting that testing
together. If we install more onto the workers we could probably test 32
bit as well, I'm torn on that right now as its a lot more dependencies
on the workers.
Building older releases is now broken until I sort out the
corresponding helper changes for those branches but one thing at a
time!
Does that include Thud ? I expect sumo and below to be problems.

Do the changes need to roll from newest to oldest  or are they
independent? I was hoping to get Sumo tested using the original QA
process to buy us time to get the automation working on it. Does this
changes that ?

- armin
Post by Richard Purdie
If anyone would like to help with xml results file analysis tooling,
that is where we need work next to be able to summarise the ptest
results and do results comparison. I know Ee Peng has some work in this
area which I will be looking at next.
Cheers,
Richard
--
r***@linuxfoundation.org
2018-11-24 21:39:32 UTC
Permalink
Post by akuster
Post by Richard Purdie
We now run "a-quick" on master at 1am every day which will make
What Time zone for 1am?
I'm not 100% sure but I think its UTC.
Post by akuster
Post by Richard Purdie
Building older releases is now broken until I sort out the
corresponding helper changes for those branches but one thing at a
time!
Does that include Thud ? I expect sumo and below to be problems.
Do the changes need to roll from newest to oldest or are they
independent? I was hoping to get Sumo tested using the original QA
process to buy us time to get the automation working on it. Does this
changes that ?
Good news is I've fixed the older releases now and run test builds to
check things. There were a couple of fixes needed but I've sorted those
and done a test artefact deployment too. This just updates everything
to use the new builder names.

The QA process is unchanged. If a given build generates a test report,
whichever release it is, it will be collected up but if not that is
also fine. This means we can opt in to the new code if/as/when we
backport anything. So we're good for sumo.

We do need to sort these gpg issues across various stable releases
though and probably tweak master too.

There were some test results ordering changes I'd be tempted to
backport just as they make reading the build logs so much easier...

Cheers,

Richard



--

Loading...