[yocto] [RFC] Yocto Autobuilder and LAVA Integration

[Anibal Limon]

Hi,

We know the need to execute OE testimage over real HW not only QEMU,

I'm aware that currently there is an implementation on the Yocto
Autobuilder Helper
, this initial implementation looks pretty well separating parts for
template generation [1] and the script to send jobs to LAVA [2].

There are some limitations.

- Requires that the boards are accessible trough SSH (same network?) by the
Autobuilder, so no distributed LAB testing.
- LAVA doesn't know about test results because the execution is injected
via SSH.

In order to do a distributed Boards testing the Yocto Autobuilder needs to
publish in some place accessible the artifacts (image, kernel, etc) to
flash/boot the board and the test suite expected to execute.

Currently there is a functionality called testexport (not too
used/maintained) that allows you to export the test suite.

I created a simple LAVA test definition that allows run testimage (oe-test
runtime) in my own LAVA LAB, is very simplistic only has a regex to parse
results and uses lava-target-ip and lava-echo-ipv4 to get target and server
IP addresses.

In this way the LAVA server handles all the testing and finally the Yocto
Autobuilder can get/poll an event to know what was the actual result of the
job and the job could be send to different LAVA LAB's.

Some of the tasks, I identified, (if is accepted)

- Yocto-aubuilder-helper: Implement/adapt to cover this new behavior , move
the EXTRA_PLAIN_CMDS to a class.
- Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
templates across boards.
- Poky/OE: Review/fix test-export or provide other mechanism to export the
test suite.

Cheers,
Anibal

[1]
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava-templates/generate-jobconfig.jinja2
[2]
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava
[3]
https://github.com/alimon/test-definitions/commit/4691b67daca26658b669ac0e79e4f27cbf6ed88d
[4]
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/config-intelqa-x86_64-lava.json#n127

[r***@linuxfoundation.org]

Hi Anibal,

Post by Anibal Limon
We know the need to execute OE testimage over real HW not only QEMU,
I'm aware that currently there is an implementation on the Yocto
Autobuilder Helper
, this initial implementation looks pretty well separating parts for
template generation [1] and the script to send jobs to LAVA [2].
There are some limitations.
- Requires that the boards are accessible trough SSH (same network?)
by the Autobuilder, so no distributed LAB testing.
- LAVA doesn't know about test results because the execution is
injected via SSH.
In order to do a distributed Boards testing the Yocto Autobuilder
needs to publish in some place accessible the artifacts (image,
kernel, etc) to flash/boot the board and the test suite expected to
execute.
Currently there is a functionality called testexport (not too
used/maintained) that allows you to export the test suite.

I continue to have mixed feelings about testexport. It adds complexity
but I'm not sure its actually worth it.

An alternative would be to specify a set of commit hashes for the
configuration under test (poky or oe-core+bitbake and any other
layers), then have LAVA obtain those pieces and run the tests directly.

Its worth considering that we already now have two difference pieces of
code trying to package up the build system/layers, eSDK and testexport.
Ideally if we had some kind of standardised layer setup/configuration
approach we'd then just have a config file to share, then the tools
could recreate the environment and allow the tests to be run there
without testexport. Layer-setup is itself a harder subject but for
example the layer setup code in autobuilder-helper could easily be
reused as things stand today...

In fact the more I think about it, the more I think we may want to do
it that way...

Post by Anibal Limon
I created a simple LAVA test definition that allows run testimage
(oe-test runtime) in my own LAVA LAB, is very simplistic only has a
regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
get target and server IP addresses.
In this way the LAVA server handles all the testing and finally the
Yocto Autobuilder can get/poll an event to know what was the actual
result of the job and the job could be send to different LAVA LAB's.

That does sound useful and is likely a way we could end up doing this.
Its probably worth highlighting that we now have a way of summarising
the result of the test in the form of the json file the tests all
generate. Sharing that back to the Yocto autobuilder would give us the
test results we need.

Post by Anibal Limon
Some of the tasks, I identified, (if is accepted)
- Yocto-aubuilder-helper: Implement/adapt to cover this new behavior
, move the EXTRA_PLAIN_CMDS to a class.
- Yocto-aubuilder-helper: Create a better approach to re-use LAVA
job templates across boards.
- Poky/OE: Review/fix test-export or provide other mechanism to
export the test suite.

I think some of these are also independent of each other and good
things to work on regardless...

I would like to hear feedback from those at Intel using LAVA who
submitted the existing code.

Cheers,

Richard

--

[Chan, Aaron Chun Yew]

Hi Anibal/RP,

Post by Anibal Limon
In order to do a distributed Boards testing the Yocto Autobuilder
needs to publish in some place accessible the artifacts (image,
kernel, etc) to flash/boot the board and the test suite expected to
execute.

[Reply] That is correct, since Linaro have this in place to use https://archive.validation.linaro.org/directories/ and I have look into this as well, we can leverage on this
but I am up for any suggestion you might have. So the idea here is that we have a placeholder to store the publish artifacts remotely and deploy using
native curl command with token access. Then based on your LAVA job definitions we can instruct LAVA to source the images via https.
Having said that, the deploy stage in LAVA must have some capabilities to read a token file in the LAVA job defintion and pick up the binaries from public repo (git LFS).

In order for Board Distributed Tests to happen, there are 2 items in my wish lists

1. Public hosting of binary repository - with access control
2. Ease Handshaking between two(2) different systems CI (e.g. Jenkins/Autobuilder) with LAVA
a. Exchange build property (metadata) - includes hardware info, system info
b. Test reporting results

Post by Anibal Limon
I created a simple LAVA test definition that allows run testimage
(oe-test runtime) in my own LAVA LAB, is very simplistic only has a
regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
get target and server IP addresses.

[Reply] Although the lava test shell have these capabilities to use lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the way we retrieve the Ipv4
address is reading the logs from LAVA thru XML-RPC and grep the pattern matching string which contains IP even before the HW get initialize entirely then parse
IP back to the Yocto Autobuilder.

http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs

Post by Anibal Limon
Some of the tasks, I identified, (if is accepted)
- Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
move the EXTRA_PLAIN_CMDS to a class.
- Poky/OE: Review/fix test-export or provide other mechanism to export
the test suite. > - Poky/OE: Review/fix test-export or provide other mechanism to export
the test suite.

[Reply] I would like to understand further what is the implementation here and how it addresses the problems that we have today. I believe in the past, Tim has tried
to enable testexport and transfer the testexport into the DUT but it was not very successful and we found breakage.

Post by Anibal Limon
- Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
templates across boards.

[Reply] I couldn’t be more supportive on this having a common LAVA job template across boards but I would like to stress this, we don’t exactly know how
community will define their own LAVA job definition, therefore what I had in mind as per today is to create a placeholde where LAVA job templates
can be define and other boards/community can reuse the same template if it fits their use cases. In general the templates we have today are
created to fit into Yocto Project use cases.

Lastly there are some works I've done on provisiong QEMU on LAVA sourceing from Yocto Project public releases, I am looking at where we can upstream this
https://github.com/lab-github/yoctoproject-lava-test-shell

Thanks!

Cheers,
Aaron Chan
Open Source Technology Center Intel

-----Original Message-----
From: ***@linuxfoundation.org [mailto:***@linuxfoundation.org]
Sent: Thursday, November 8, 2018 6:45 AM
To: Anibal Limon <***@linaro.org>; ***@yoctoproject.org
Cc: Nicolas Dechesne <***@linaro.org>; Chan, Aaron Chun Yew <***@intel.com>
Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration

Hi Anibal,

Post by Anibal Limon
We know the need to execute OE testimage over real HW not only QEMU,
I'm aware that currently there is an implementation on the Yocto
Autobuilder Helper , this initial implementation looks pretty well
separating parts for template generation [1] and the script to send
jobs to LAVA [2].
There are some limitations.
- Requires that the boards are accessible trough SSH (same network?)
by the Autobuilder, so no distributed LAB testing.
- LAVA doesn't know about test results because the execution is
injected via SSH.
In order to do a distributed Boards testing the Yocto Autobuilder
needs to publish in some place accessible the artifacts (image,
kernel, etc) to flash/boot the board and the test suite expected to
execute.
Currently there is a functionality called testexport (not too
used/maintained) that allows you to export the test suite.

I continue to have mixed feelings about testexport. It adds complexity but I'm not sure its actually worth it.

An alternative would be to specify a set of commit hashes for the configuration under test (poky or oe-core+bitbake and any other layers), then have LAVA obtain those pieces and run the tests directly.

Its worth considering that we already now have two difference pieces of code trying to package up the build system/layers, eSDK and testexport.
Ideally if we had some kind of standardised layer setup/configuration approach we'd then just have a config file to share, then the tools could recreate the environment and allow the tests to be run there without testexport. Layer-setup is itself a harder subject but for example the layer setup code in autobuilder-helper could easily be reused as things stand today...

In fact the more I think about it, the more I think we may want to do it that way...

Post by Anibal Limon
I created a simple LAVA test definition that allows run testimage
(oe-test runtime) in my own LAVA LAB, is very simplistic only has a
regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
get target and server IP addresses.
In this way the LAVA server handles all the testing and finally the
Yocto Autobuilder can get/poll an event to know what was the actual
result of the job and the job could be send to different LAVA LAB's.

That does sound useful and is likely a way we could end up doing this.
Its probably worth highlighting that we now have a way of summarising the result of the test in the form of the json file the tests all generate. Sharing that back to the Yocto autobuilder would give us the test results we need.

Post by Anibal Limon
Some of the tasks, I identified, (if is accepted)
- Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
move the EXTRA_PLAIN_CMDS to a class.
- Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
templates across boards.
- Poky/OE: Review/fix test-export or provide other mechanism to export
the test suite.

I think some of these are also independent of each other and good things to work on regardless...

I would like to hear feedback from those at Intel using LAVA who submitted the existing code.

Cheers,

Richard

--

[Anibal Limon]

On Thu, 8 Nov 2018 at 20:49, Chan, Aaron Chun Yew <

Post by Chan, Aaron Chun Yew
Hi Anibal/RP,

Post by Anibal Limon
In order to do a distributed Boards testing the Yocto Autobuilder
needs to publish in some place accessible the artifacts (image,
kernel, etc) to flash/boot the board and the test suite expected to
execute.

[Reply] That is correct, since Linaro have this in place to use
https://archive.validation.linaro.org/directories/ and I have look into
this as well, we can leverage on this
but I am up for any suggestion you might have. So the idea
here is that we have a placeholder to store the publish artifacts remotely
and deploy using
native curl command with token access. Then based on your
LAVA job definitions we can instruct LAVA to source the images via https.
Having said that, the deploy stage in LAVA must have some
capabilities to read a token file in the LAVA job defintion and pick up the
binaries from public repo (git LFS).
In order for Board Distributed Tests to happen, there are 2
items in my wish lists
1. Public hosting of binary repository - with access control

For publish the artifacts (Rootfs, Kernel image, Test suite), if there is a
public build a token isn't needed like targeting some boards already
commercialized and can be published anywhere like in
http://downloads.yoctoproject.org.

Post by Chan, Aaron Chun Yew
2. Ease Handshaking between two(2) different systems CI
(e.g. Jenkins/Autobuilder) with LAVA
a. Exchange build property (metadata) - includes
hardware info, system info

You can add meta-data to a LAVA test definition.

b. Test reporting results
For notify job results LAVA test definition support the notify block in
test jobs or you can poll the API using for both needs a LAVA token.

Post by Chan, Aaron Chun Yew

Post by Anibal Limon
I created a simple LAVA test definition that allows run testimage
(oe-test runtime) in my own LAVA LAB, is very simplistic only has a
regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
get target and server IP addresses.

[Reply] Although the lava test shell have these capabilities to use
lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the
way we retrieve the Ipv4
address is reading the logs from LAVA thru XML-RPC and grep
the pattern matching string which contains IP even before the HW get
initialize entirely then parse
IP back to the Yocto Autobuilder.
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs

Yes, that's my idea the Yocto Autobuilder dosen't need to know about
particular network configuration in tha LAVA server for execute the job, in
this way the Yocto Autobuilder can communicate with LAVA server to retrieve
the testing job results, and in a case that needs to debug the board LAVA
support hacking sessions to allow connect to the board outside the LAB.

https://validation.linaro.org/static/docs/v2/hacking-session.html

Post by Chan, Aaron Chun Yew

Post by Anibal Limon
Some of the tasks, I identified, (if is accepted)
- Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
move the EXTRA_PLAIN_CMDS to a class.
- Poky/OE: Review/fix test-export or provide other mechanism to export
the test suite. > - Poky/OE: Review/fix test-export or provide other

mechanism to export

Post by Anibal Limon
the test suite.

[Reply] I would like to understand further what is the implementation here
and how it addresses the problems that we have today. I believe in the
past, Tim has tried
to enable testexport and transfer the testexport into the
DUT but it was not very successful and we found breakage.

Agree, The testexport functionality is on not usage so there are some bugs
on it.

Post by Chan, Aaron Chun Yew

Post by Anibal Limon
- Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
templates across boards.

[Reply] I couldn’t be more supportive on this having a common LAVA job
template across boards but I would like to stress this, we don’t exactly
know how
community will define their own LAVA job definition,
therefore what I had in mind as per today is to create a placeholde where
LAVA job templates
can be define and other boards/community can reuse the same
template if it fits their use cases. In general the templates we have today
are
created to fit into Yocto Project use cases.

Agree, I not mean a single template but a manner to add easily new LAVA
templates for boards in Yocto Autobuilder, this involves some base LAVA job
templates and a set of scripts to
generate the final template, like you are doing. For example there are
different ways to deploy a board but the login process is the same for
core-image's (login as root wo passwd).

Cheers,
Anibal

Post by Chan, Aaron Chun Yew
Lastly there are some works I've done on provisiong QEMU on LAVA
sourceing from Yocto Project public releases, I am looking at where we can
upstream this
https://github.com/lab-github/yoctoproject-lava-test-shell
Thanks!
Cheers,
Aaron Chan
Open Source Technology Center Intel
-----Original Message-----
Sent: Thursday, November 8, 2018 6:45 AM
Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration
Hi Anibal,

Post by Anibal Limon
We know the need to execute OE testimage over real HW not only QEMU,
I'm aware that currently there is an implementation on the Yocto
Autobuilder Helper , this initial implementation looks pretty well
separating parts for template generation [1] and the script to send
jobs to LAVA [2].
There are some limitations.
- Requires that the boards are accessible trough SSH (same network?)
by the Autobuilder, so no distributed LAB testing.
- LAVA doesn't know about test results because the execution is
injected via SSH.
In order to do a distributed Boards testing the Yocto Autobuilder
needs to publish in some place accessible the artifacts (image,
kernel, etc) to flash/boot the board and the test suite expected to
execute.
Currently there is a functionality called testexport (not too
used/maintained) that allows you to export the test suite.

I continue to have mixed feelings about testexport. It adds complexity but
I'm not sure its actually worth it.
An alternative would be to specify a set of commit hashes for the
configuration under test (poky or oe-core+bitbake and any other layers),
then have LAVA obtain those pieces and run the tests directly.
Its worth considering that we already now have two difference pieces of
code trying to package up the build system/layers, eSDK and testexport.
Ideally if we had some kind of standardised layer setup/configuration
approach we'd then just have a config file to share, then the tools could
recreate the environment and allow the tests to be run there without
testexport. Layer-setup is itself a harder subject but for example the
layer setup code in autobuilder-helper could easily be reused as things
stand today...
In fact the more I think about it, the more I think we may want to do it that way...

Post by Anibal Limon
I created a simple LAVA test definition that allows run testimage
(oe-test runtime) in my own LAVA LAB, is very simplistic only has a
regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
get target and server IP addresses.
In this way the LAVA server handles all the testing and finally the
Yocto Autobuilder can get/poll an event to know what was the actual
result of the job and the job could be send to different LAVA LAB's.

That does sound useful and is likely a way we could end up doing this.
Its probably worth highlighting that we now have a way of summarising the
result of the test in the form of the json file the tests all generate.
Sharing that back to the Yocto autobuilder would give us the test results
we need.

Post by Anibal Limon
Some of the tasks, I identified, (if is accepted)
- Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
move the EXTRA_PLAIN_CMDS to a class.
- Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
templates across boards.
- Poky/OE: Review/fix test-export or provide other mechanism to export
the test suite.

I think some of these are also independent of each other and good things
to work on regardless...
I would like to hear feedback from those at Intel using LAVA who submitted
the existing code.
Cheers,
Richard