ccloudvm create as non-root user fails at "Building CIAO" step

[jdandrea]

It was recommended to only create the ciao ccloudvm instance as a non-root user.

For the non-root user on the ccloudvm host:

• Account was created with sudo access
• Appropriate proxy env vars are sourced at login-time
• User was added to kvm group via sudo gpasswd -a $USER kvm
• User was logged out and logged in again
• Account has a local go directory with ciao-project pkg, et. al
• User PATH includes /home/USERNAME/go/bin
• GOPATH is /home/USERNAME/go
• GOROOT is /usr/local/go

Next:

• ccloudvm was torn down under the previously used root account
• ccloudvm was setup under the non-root user account

See attached cloud-init-output.log from the ccloudvm instance. Operations were not permitted under /home/jdandrea/go (e.g., changing ownership from root to jdandrea, presumably), and ciao installation failed as a result.

Here's what appeared on the ccloudvm instance during the same time:

Booting VM : [OK]
Adding singlevm to /etc/hosts : [OK]
Mounting /home/jdandrea/go : [OK]
Add docker GPG key : [OK]
Adding docker repo : [OK]
Retrieving updated list of packages : [OK]
Install docker-ce : [OK]
Adding user jdandrea to the docker group : [OK]
Configuring docker proxies : [OK]
Downloading Go : [OK]
Unpacking Go : [OK]
Add Google GPG key : [OK]
Retrieving updated list of packages : [OK]
Installing kubectl : [OK]
Installing GCC : [OK]
Installing Make : [OK]
Installing QEMU : [OK]
Installing xorriso : [OK]
Installing ceph-common : [OK]
Auto removing unused components : [OK]
Building ciao : [FAIL]
Installing Go development utils : [OK]
Pulling ceph/demo : [OK]
Downloading Fedora-Cloud-Base-24-1.2.x86_64.qcow2 : [FAIL]
Downloading xenial-server-cloudimg-amd64-disk1.img : [OK]
Downloading CNCI image : [OK]
Downloading latest clear cloud image : [OK]
VM successfully created!

[jdandrea]

Here is the full ccloudvm create context.

[jdandrea]

Tried adding user account to the sudo and docker groups as well. No difference.

It looks like some of the chown commands run during cloud-init are not sudo:

root@singlevm:/var/lib/cloud# grep chown ./instances/e67a6687-5dc6-4021-80ed-88d7da3618ea/user-data.txt
- sudo chown jdandrea:jdandrea hostgo
- chown jdandrea:jdandrea -R /home/jdandrea/.docker
- chown jdandrea:jdandrea /home/jdandrea
- chown jdandrea:jdandrea -R /home/jdandrea/go
- chown jdandrea:jdandrea -R /home/jdandrea/local

... and the non-root account does indeed get sudo access:

jdandrea@singlevm:/var/lib/cloud$ sudo -l
Matching Defaults entries for jdandrea on singlevm:
    env_reset, mail_badpass,
    secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin\:/snap/bin

User jdandrea may run the following commands on singlevm:
    (ALL) NOPASSWD: ALL

[jdandrea]

The permission issue may be separate though, plus /home/USER/go is mounted in /etc/fstab and shows up as root:root, so there's no way to chown.

This is the part that appears to fail:

- curl -X PUT -d "Building ciao" 10.0.2.2:43040
- sudo -u jdandrea no_proxy=.localdomain.com,10.0.2.15,10.0.2.2,127.0.0.1,127.3.237.1,localaddress,localhost,redis-master,redis-slave,singlevm
  NO_PROXY=.localdomain.com,10.0.2.15,10.0.2.2,127.0.0.1,127.3.237.1,localaddress,localhost,redis-master,redis-slave,singlevm
  http_proxy=http://one.proxy.att.com:8080 HTTP_PROXY=http://one.proxy.att.com:8080
  https_proxy=http://one.proxy.att.com:8080 HTTPS_PROXY=http://one.proxy.att.com:8080
  DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true  GOPATH=/home/jdandrea/go
  /usr/local/go/bin/go get github.com/ciao-project/ciao/...
- if [ $? -eq 0 ] ; then ret="OK" ; else ret="FAIL" ; fi ; curl -X PUT -d $ret 10.0.2.2:43040

When I try this interactively as the non-root account:

jdandrea@singlevm:/var/lib/cloud$ sudo -u jdandrea no_proxy=.localdomain.com,10.0.2.15,10.0.2.2,127.0.0.1,127.3.237.1,localaddress,localhost,redis-master,redis-slave,singlevm NO_PROXY=.localdomain.com,10.0.2.15,10.0.2.2,127.0.0.1,127.3.237.1,localaddress,localhost,redis-master,redis-slave,singlevm http_proxy=http://one.proxy.att.com:8080 HTTP_PROXY=http://one.proxy.att.com:8080 https_proxy=http://one.proxy.att.com:8080 HTTPS_PROXY=http://one.proxy.att.com:8080 DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true GOPATH=/home/jdandrea/go /usr/local/go/bin/go get github.com/ciao-project/ciao/...
go install github.com/ciao-project/ciao/ciao-launcher: open /home/jdandrea/go/bin/ciao-launcher: operation not permitted

[markdryan]

@jdandrea This looks like a 9p issue. There could be a number of reasons for this.

1. When you created the new account, did you resync the ciao code as the new user? If your GOPATH, and the files it contains, are not owned by your current user you probably won't be able to access them from inside the VM.
2. Could you check the uid and gid of your current user matches the uid and gid of the user inside the guest. They should match ( and if they don't 9P won't work properly).

You could try changing the security model used to mount the go path by passing the following parameter to create

--mount hostgo,none,$HOME/go

If this doesn't work, edit the ciao.yaml workload locally and remove lines 7 to 12

https://github.com/intel/ccloudvm/blob/master/workloads/ciao.yaml#L7

[jdandrea]

@markdryan Thanks! Let's see... when I created the account I did not resync. What I did was:

• mv /home/ciao/go /home/jdandrea
• chown -R jdandrea:jdandrea /home/jdandrea/go
• Adjusted/re-sourced GOPATH to /home/jdandrea/go
• Adjusted/re-sourced PATH to include /home/jdandrea/go/bin

I've since destroyed the instance and can't check uid/gid for a match, but I will do that next go-round. (Groan.) I'll try the security model if the uid/gid match doesn't work.

[markdryan]

@jdandrea Thanks for the info. There was a bug a while back in which 9P file sharing did not work if you were not the first user on the host machine, i.e., you did not have uid and gid of 1000. This should be fixed, by forcing the guest user to have the same uid and gid as the host user. I just wanted to double check.

9P can be a bit tricky to get to work sometimes.

[jdandrea]

Update: My uid:gid is 1004:1005. On the singlevm instance it's 1004:1004, so it doesn't match. Turns out that $GOPATH/src/github.com/intel/ccloudvm/workloads/xenial.yaml references gid but cloud-init on Xenial actually uses primary_group instead, and requires the group name, not gid.

As a workaround, ensure the uid:gid are identical on the ccloudvm host. The gid will remain unparsed, and then the actual group will be set to uid, thereby matching the host. No security model change needed.

[markdryan]

There doesn't seem to be any way to set the gid of a user in cloud-init. It is always set to the uid. I had thought this was supported in cloud-init when I made the gid/uid changes but it seems I was wrong. I can't see any real way of fixing this. We could detect the case and issue a warning if the host user's uid and gid do not match and he launches a VM with 9p mounts. At least then he'd know that 9p file sharing was unlikely to work.

[markdryan]

Here's the cloud-init bug

https://bugs.launchpad.net/cloud-init/+bug/1396362

Perhaps we should change the title of this bug to be something like.

9p filesharing does not work if the host user's gid and uid differ.

[jdandrea]

In light of the situation, I think issuing a warning is a fine and thoughtful thing to do.