Kata Containers metrics CI Jenkins slave request

[grahamwhaley]

Please fill out the details below to file a request for access to the CNCF Community Infrastructure Lab. Please note that access is targeted to people working on specific open source projects; this is not designed just to get your feet wet. The most important answer is the URL of the project you'll be working with. If you're looking to learn Kubernetes and related technologies, please try out Katacoda.

First and Last Name

Graham Whaley

Email

graham.whaley@intel.com

Company/Organization

Intel

Job Title

Senior Software Engineer

Project Title

Kata Containers

Briefly describe the project

Open source multi architecture community collaboration to develop virtual machine based container runtimes and deliver their integration into common container infrastructures and orchestration (OCI, Docker, Kubernetes etc.)

Which members of the CNCF community and/or end-users would benefit from your work?

The obvious member is Kubernetes, who already work closely in conjunction with the Kata Containers community to ensure Kubernetes and virtual machine container runtimes are a natural and compatible fit.

Is the code that you’re going to run 100% open source? If so, what is the URL or URLs where it is located? What is your association with that project?

Yes, 100% open source and up on github:
https://github.com/kata-containers

What kind of machines and how many do you expect to use (see: https://www.packet.net/bare-metal/)?

Prediction is 2x t1.small.x86 machines, running 24/7-ish. Our current jenkins CI backlog across all the repositories pretty much consumes one whole machine (and it is not compute bound).

We can start/trial with just one t1.small.x86 (for PR CI), and later add another to support master branch merge regression checking.

What OS and networking are you planning to use (see: https://help.packet.net/technical/infrastructure/supported-operating-systems)?

I would expect Ubuntu 18.04

Please state your contributions to the open source community and any other relevant initiatives

Previously having worked on a new architecture addition to the Linux kernel which eventually made it into the upstream, for the last 2+ years I have been focussed on the open source Clear Containers (https://github.com/clearcontainers), now Kata Containers.

Any other relevant details we should know about?

I expect us to tie the machines as Jenkins slaves into our existing Jenkins master at http://jenkins.katacontainers.io/, and dedicate them to metrics CI builds only.

Kata Containers is umbrella'd under the OpenStackFoundation, but is not part of the OpenStack project.

[dankohn]

Could you please describe what you'd like to actually do? We're open to supporting you, but would like to confirm that Intel or OpenStack infrastructure cannot meet your needs.

[jacobsmith928]

@dankohn if this use case (e.g. ongoing CI infra) doesn't fit well into the CIL, we can work with @grahamwhaley separately on an arrangement.

[dankohn]

@grahamwhaley Thanks for the reference to kata-containers/ci#6

I checked with @jacobsmith928 and we have a thumbs up for you to go forward.

My request (both for Community Infrastructure Lab policy and for best practice) is that you make 100% of your continuous integration code open source (other than confidential tokens, obviously).

+1

[grahamwhaley]

That's fantastic news @dankohn @jacobsmith928
I know you've probably gotten the details from kata-containers/ci#6, but I'll drop a summary here for the record.

For Kata Containers we have a set of metrics tests that we'd like to run in a CI to both:

• check for regressions on each new PR request/update
• log the results for master branch merges so we can collect and plot/analyse historical data

Due to the nature of the majority of the tests, we need to run these in a reproducible manner (otherwise we cannot regression check or compare over time), and that thus mandates either bare metal machines or dedicated cloud servers (that support nested VMs), with no noisy neighbour effects etc.

We have struggled to find any suitable hardware so far, and hence the request here. OSF does not have access to such hardware.
We plan to run this all under Jenkins. These machines will be new dedicated metrics slaves controlled by the existing kata Jenkins CI master (that is hosted under the OSF resource kindly donated by vexxhost).

All of the code and configs will be fully open sourced. Almost all of it is already open:

• The script the slaves run
• the code for the program that checks for regressions
• docs for setting up our Jenkins configs

and we intend to publish all the Jenkins details/configs we can, like we already publish all the Jenkins configs (apart from the secrets ;-) ) for the parallel QA CI: https://github.com/kata-containers/ci/tree/master/jenkins

Thanks!

[taylorwaggoner]

@grahamwhaley - I have invited you to the Kata Containers project in Packet. Please let me know if you have any questions!

[grahamwhaley]

Thanks @taylorwaggoner I've accepted the invite, and can see the Kata Containers project within the CNCF org on packet.net.
It's my first time deploying on packet.net, so I need to go do a bit of readup on how to set up the deployment and then how we tie that into Jenkins (if we have a nodepool for instance etc.).
That'll take me a little bit of time. I'll post a status update here when things are up (or stuck ;-) ).

Many thanks everybody!

[jacobsmith928]

@grahamwhaley definitely ping @vielmetti or me in our community slack and we can help as needed.

[vielmetti]

A specific issue we are tracking is JCLOUDS-1219 , and its related Github issue jclouds/jclouds-labs#337

[grahamwhaley]

I just noticed on slack that the t1.small.x86 machine now come in two flavours (4 and 8 core?). Given I need repeatability of metrics tests in order for the CI to spot regressions, and the fact that right now I cannot get jenkins to deploy a packet machine on-demand via the jclouds plugin, I think the prudent way forwards is to deploy and assign a t1.small.x86 machine 24/7 to the task of Kata metrics CI.
Any objections to that on resource sharing/utilisation/cost grounds etc.? @dankohn @jacobsmith928 @vielmetti

[jacobsmith928]

@grahamwhaley yes that is fine.

[grahamwhaley]

Hi.
I'm seeing more variance (noise) in the metrics results than I expected on the t1.small.x86 machine. It also seems that the machine is running Kata itself very slowly (8s to get into a container, rather than the <1.5s I was expecting).
Can I request access to an x86 machine from the next tier up (I would grab the name, but I'm having difficulty getting to the packet web pages with that info right now - it might be the c1.small ?), the goals being:

• confirm if the variance and slowness is specific to that machine type/tier
• If not, then debug using the new machine, whilst leaving the t1.small running the CI builds still)
• if the problems do only manifest on the t1.small.x86, then it is likely I will request if we can move from the t1.small to the next tier.

Many thanks.

[dankohn]

Certainly. Please go ahead. -- Dan Kohn <dan@linuxfoundation.org> Executive Director, Cloud Native Computing Foundation https://www.cncf.io +1-415-233-1000 https://www.dankohn.com

…

On Thu, Oct 4, 2018 at 9:29 AM Graham Whaley ***@***.***> wrote: Hi. I'm seeing more variance (noise) in the metrics results than I expected on the t1.small.x86 machine. It also seems that the machine is running Kata itself very slowly (8s to get into a container, rather than the <1.5s I was expecting). Can I request access to an x86 machine from the next tier up (I would grab the name, but I'm having difficulty getting to the packet web pages with that info right now - it might be the c1.small ?), the goals being: - confirm if the variance and slowness is specific to that machine type/tier - If not, then debug using the new machine, whilst leaving the t1.small running the CI builds still) - if the problems do only manifest on the t1.small.x86, then it is likely I will request if we can move from the t1.small to the next tier. Many thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#83 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC8MBshVFgm9gLc_Qlntk65lBzTCaWb-ks5uhg1GgaJpZM4WT3j_> .