Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 88 additions & 28 deletions g3doc/user_guide/tutorials/docker-in-gvisor.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,41 +6,79 @@ applications.
In gVisor, all basic docker commands should function as expected. The host
network driver and the bridge network driver are tested and supported.

### Supported Docker Versions

<table>
<thead>
<tr>
<th>Docker Version</th>
<th>Support Status</th>
<th>Required gVisor Configuration</th>
</tr>
</thead>
<tbody>
<tr>
<td>Docker v27</td>
<td>Supported</td>
<td>
<ul>
<li><code>--net-raw</code></li>
</ul>
</td>
</tr>
<tr>
<td>Docker v28</td>
<td>Supported</td>
<td>
<ul>
<li><code>--net-raw</code></li>
<li><code>--allow-packet-socket-write</code></li>
</ul>
</td>
</tr>
<tr>
<td>Docker v29</td>
<td>Supported</td>
<td>
<ul>
<li><code>--net-raw</code></li>
<li><code>--allow-packet-socket-write</code></li>
<li>Either <code>tmpfs</code> mount at <code>/var/lib/docker</code> OR <code>--feature containerd-snapshotter=false</code></li>
</ul>
</td>
</tr>
</tbody>
</table>

### Limitations

- Since version 29, docker defaults to using the
[containerd image store](https://docs.docker.com/engine/storage/containerd/)
as its default storage backend. For this to work inside gVisor, we need to
mount a tmpfs at `/var/lib/docker` because gVisor currently only allows
tmpfs mounts as upper layers of an overlay filesystem. See
[images/basic/docker/start-dockerd.sh](https://github.com/google/gvisor/blob/master/images/basic/docker/start-dockerd.sh)
for reference. Alternately, one can chose to avoid the new storage backend
altogether by launching dockerd with `--feature
containerd-snapshotter=false`.
- `dockerd` inside gVisor needs to be executed with flags `--iptables=false
--ip6tables=false` and additional network setup is needed, check
[images/basic/docker/start-dockerd.sh](https://github.com/google/gvisor/blob/master/images/basic/docker/start-dockerd.sh)
for reference.
- With iptables disabled, `docker run --expose=` does not expose the port; if
a nested container needs to expose ports, inside gVisor use `docker run
--network=host`.
for reference. With iptables disabled, `docker run --expose=` does not
expose the port; if a nested container needs to expose ports, inside gVisor
use `docker run --network=host`.

### Configuration Reference

### NOTE on runsc setup
To run Docker inside gVisor, you need to configure `runsc` and potentially the
container storage depending on the Docker version.

To run docker within gvisor, runsc must be enabled to allow raw sockets. This is
not the default, `--net-raw` must be passed to runsc.
#### runsc Flags (in `/etc/docker/daemon.json`)

In addition, Docker versions 28 and beyond need the ability to write to
AF_PACKET sockets. This is because dockerd sends unsolicited ARP/NA requests
when bringing up interfaces. To allow this, the `--allow-packet-socket-write` is
also to be supplied (the default behavior is to disallow writes to AF_PACKET
sockets).
To update your `/etc/docker/daemon.json` with the necessary `runtimeArgs`:

To use the following tutorial, that means having the following runtimes
configuration in `/etc/docker/daemon.json`:
* **`--net-raw`** (Required for all Docker versions)
* Enables raw sockets inside the sandbox. This is required for Docker's
networking to function.
* **`--allow-packet-socket-write`** (Required for Docker v28 and later)
* Allows the sandbox to write to `AF_PACKET` sockets. This is necessary
because `dockerd` (v28+) sends unsolicited ARP/NA requests when bringing
up network interfaces.
* > **Note:** `--allow-packet-socket-write` allows sandboxed code to craft
arbitrary network packets.

> **Note:** `--allow-packet-socket-write` allows sandboxed code to craft
> arbitrary packets. It is only needed for Docker versions 28 and beyond.
##### Example `/etc/docker/daemon.json`

```json
{
Expand All @@ -56,9 +94,31 @@ configuration in `/etc/docker/daemon.json`:
}
```

If you have an existing entry for `runsc`, likely created by `runsc install`,
then edit the entry and add the `"runtimeArgs"` key and value to the existing
entry.
If you have an existing entry for `runsc` (likely created by `runsc install`),
edit the entry to add the `"runtimeArgs"` key and value.

#### Docker Storage Backend

Since version 29, Docker Engine defaults to using the containerd image store,
which uses `overlayfs` for container root filesystems.

In nested container and sandbox environments where `/var/lib/docker` is already
on an `overlayfs` mount, attempting to mount another `overlayfs` on top will
fail because Linux does not permit overlay-on-overlay mounts (and gVisor
currently requires `tmpfs` as an upper layer for `overlayfs`).

To run Docker in gVisor, you must use one of the following workarounds:

* **Mount `tmpfs` at `/var/lib/docker` (Recommended):** Mount a `tmpfs`
filesystem at `/var/lib/docker` before starting the Docker daemon inside the
sandbox so `overlayfs` can use `tmpfs` as its backing store.
* **Disable containerd image store:** Launch `dockerd` with `--feature
containerd-snapshotter=false` to fall back to classic storage drivers.

See
[images/basic/docker/start-dockerd.sh](https://github.com/google/gvisor/blob/master/images/basic/docker/start-dockerd.sh)
as a reference implementation of how to handle these configurations at container
startup.

## How to run Docker in a gVisor container

Expand Down
8 changes: 3 additions & 5 deletions images/basic/docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
FROM ubuntu:24.04

ARG DOCKER_VERSION="5:28.5.2-1~ubuntu.24.04~noble"
FROM ubuntu:latest

ENV DEBIAN_FRONTEND="noninteractive"
RUN apt-get update && apt-get -y install ca-certificates curl
Expand All @@ -11,8 +9,8 @@ RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" > /etc/apt/sources.list.d/docker.list && \
apt-get update && \
apt-get install -qqy \
docker-ce=${DOCKER_VERSION} \
docker-ce-cli=${DOCKER_VERSION} \
docker-ce \
docker-ce-cli \
containerd.io \
docker-buildx-plugin \
docker-compose-plugin \
Expand Down
1 change: 1 addition & 0 deletions pkg/sentry/fsimpl/tmpfs/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,7 @@ go_test(
"//pkg/context",
"//pkg/errors/linuxerr",
"//pkg/fspath",
"//pkg/hostarch",
"//pkg/sentry/contexttest",
"//pkg/sentry/fsimpl/lock",
"//pkg/sentry/kernel/auth",
Expand Down
201 changes: 110 additions & 91 deletions pkg/sentry/fsimpl/tmpfs/tmpfs.go
Original file line number Diff line number Diff line change
Expand Up @@ -256,109 +256,128 @@ func (fstype FilesystemType) GetFilesystem(ctx context.Context, vfsObj *vfs.Virt
rootMode = 01777
}

mopts := vfs.GenericParseMountOptions(opts.Data)
var printedOpts []string

maxSizeInPages := getDefaultSizeLimit(disableDefaultSizeLimit) / hostarch.PageSize
maxSizeStr, ok := mopts["size"]
if ok {
delete(mopts, "size")
maxSizeInBytes, _, err := parseSize(maxSizeStr)
if err != nil {
ctx.Debugf("tmpfs.FilesystemType.GetFilesystem: parseSize() failed: %v", err)
return nil, nil, linuxerr.EINVAL
}
maxInodes := getDefaultInodeLimit(disableDefaultSizeLimit)
rootKUID := creds.EffectiveKUID
rootKGID := creds.EffectiveKGID

var printedSize uint64
if maxSizeInBytes > 0 {
// If size > 0, convert size in bytes to nearest Page Size
// bytes as Linux allocates memory in terms of Page size.
maxSizeInPages, ok = hostarch.ToPagesRoundUp(maxSizeInBytes)
if !ok {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: Pages RoundUp Overflow error: %v", ok)
printedOptsMap := make(map[string]string)

for _, opt := range vfs.GenericParseMountOptionsOrdered(opts.Data) {
key := opt.Key
value := opt.Value

switch key {
case "size":
maxSizeInBytes, _, err := parseSize(value)
if err != nil {
ctx.Debugf("tmpfs.FilesystemType.GetFilesystem: parseSize() failed for size: %v", err)
return nil, nil, linuxerr.EINVAL
}
printedSize = (maxSizeInPages*hostarch.PageSize + 1023) / 1024
} else {
// size = 0 is a special case: no size limit
maxSizeInPages = math.MaxInt64
printedSize = 0
}
var printedSize uint64
if maxSizeInBytes > 0 {
var ok bool
maxSizeInPages, ok = hostarch.ToPagesRoundUp(maxSizeInBytes)
if !ok {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: Pages RoundUp Overflow error")
return nil, nil, linuxerr.EINVAL
}
printedSize = (maxSizeInPages*hostarch.PageSize + 1023) / 1024
} else {
maxSizeInPages = math.MaxInt64
printedSize = 0
}
printedOptsMap["size"] = fmt.Sprintf("size=%vk", printedSize)

printedOpts = append(printedOpts, fmt.Sprintf("size=%vk", printedSize))
}
maxInodes := getDefaultInodeLimit(disableDefaultSizeLimit)
maxInodesStr, ok := mopts["nr_inodes"]
if ok {
delete(mopts, "nr_inodes")
var err error
var percentageSpecified bool
maxInodes, percentageSpecified, err = parseSize(maxInodesStr)
if percentageSpecified {
// Percentage not allowed for specifying inode limit
return nil, nil, linuxerr.EINVAL
}
if err != nil {
ctx.Debugf("tmpfs.FilesystemType.GetFilesystem: parseSize() failed: %v", err)
return nil, nil, linuxerr.EINVAL
}
case "nr_blocks":
maxBlocks, percentageSpecified, err := parseSize(value)
if percentageSpecified {
ctx.Debugf("tmpfs.FilesystemType.GetFilesystem: percentage not allowed for nr_blocks")
return nil, nil, linuxerr.EINVAL
}
if err != nil {
ctx.Debugf("tmpfs.FilesystemType.GetFilesystem: parseSize() failed for nr_blocks: %v", err)
return nil, nil, linuxerr.EINVAL
}
maxSizeInPages = maxBlocks
printedSize := (maxSizeInPages*hostarch.PageSize + 1023) / 1024
printedOptsMap["size"] = fmt.Sprintf("size=%vk", printedSize)

case "nr_inodes":
var err error
var percentageSpecified bool
maxInodes, percentageSpecified, err = parseSize(value)
if percentageSpecified {
ctx.Debugf("tmpfs.FilesystemType.GetFilesystem: percentage not allowed for nr_inodes")
return nil, nil, linuxerr.EINVAL
}
if err != nil {
ctx.Debugf("tmpfs.FilesystemType.GetFilesystem: parseSize() failed for nr_inodes: %v", err)
return nil, nil, linuxerr.EINVAL
}
printedOptsMap["nr_inodes"] = fmt.Sprintf("nr_inodes=%v", maxInodes)

printedOpts = append(printedOpts, fmt.Sprintf("nr_inodes=%v", maxInodes))
}
modeStr, ok := mopts["mode"]
if ok {
delete(mopts, "mode")
mode, err := strconv.ParseUint(modeStr, 8, 32)
if err != nil {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: invalid mode: %q", modeStr)
return nil, nil, linuxerr.EINVAL
}
rootMode = linux.FileMode(mode & 07777)
case "mode":
mode, err := strconv.ParseUint(value, 8, 32)
if err != nil {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: invalid mode: %q", value)
return nil, nil, linuxerr.EINVAL
}
rootMode = linux.FileMode(mode & 07777)
printedOptsMap["mode"] = fmt.Sprintf("mode=%s", value)

printedOpts = append(printedOpts, fmt.Sprintf("mode=%s", modeStr))
}
rootKUID := creds.EffectiveKUID
uidStr, ok := mopts["uid"]
if ok {
delete(mopts, "uid")
uid, err := strconv.ParseUint(uidStr, 10, 32)
if err != nil {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: invalid uid: %q", uidStr)
return nil, nil, linuxerr.EINVAL
}
kuid := creds.UserNamespace.MapToKUID(auth.UID(uid))
if !kuid.Ok() {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: unmapped uid: %d", uid)
return nil, nil, linuxerr.EINVAL
}
rootKUID = kuid
case "uid":
uid, err := strconv.ParseUint(value, 10, 32)
if err != nil {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: invalid uid: %q", value)
return nil, nil, linuxerr.EINVAL
}
kuid := creds.UserNamespace.MapToKUID(auth.UID(uid))
if !kuid.Ok() {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: unmapped uid: %d", uid)
return nil, nil, linuxerr.EINVAL
}
rootKUID = kuid
printedOptsMap["uid"] = fmt.Sprintf("uid=%s", value)

printedOpts = append(printedOpts, fmt.Sprintf("uid=%s", uidStr))
}
rootKGID := creds.EffectiveKGID
gidStr, ok := mopts["gid"]
if ok {
delete(mopts, "gid")
gid, err := strconv.ParseUint(gidStr, 10, 32)
if err != nil {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: invalid gid: %q", gidStr)
return nil, nil, linuxerr.EINVAL
}
kgid := creds.UserNamespace.MapToKGID(auth.GID(gid))
if !kgid.Ok() {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: unmapped gid: %d", gid)
case "gid":
gid, err := strconv.ParseUint(value, 10, 32)
if err != nil {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: invalid gid: %q", value)
return nil, nil, linuxerr.EINVAL
}
kgid := creds.UserNamespace.MapToKGID(auth.GID(gid))
if !kgid.Ok() {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: unmapped gid: %d", gid)
return nil, nil, linuxerr.EINVAL
}
rootKGID = kgid
printedOptsMap["gid"] = fmt.Sprintf("gid=%s", value)

case "noswap":
// Accept, but ignore, noswap.

default:
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: unknown option: %s", key)
return nil, nil, linuxerr.EINVAL
}
rootKGID = kgid

printedOpts = append(printedOpts, fmt.Sprintf("gid=%s", gidStr))
}
// Accept, but ignore, noswap
delete(mopts, "noswap")

if len(mopts) != 0 {
ctx.Warningf("tmpfs.FilesystemType.GetFilesystem: unknown options: %v", mopts)
return nil, nil, linuxerr.EINVAL
var printedOpts []string
if val, ok := printedOptsMap["size"]; ok {
printedOpts = append(printedOpts, val)
}
if val, ok := printedOptsMap["nr_inodes"]; ok {
printedOpts = append(printedOpts, val)
}
if val, ok := printedOptsMap["mode"]; ok {
printedOpts = append(printedOpts, val)
}
if val, ok := printedOptsMap["uid"]; ok {
printedOpts = append(printedOpts, val)
}
if val, ok := printedOptsMap["gid"]; ok {
printedOpts = append(printedOpts, val)
}

devMinor, err := vfsObj.GetAnonBlockDevMinor()
Expand Down
Loading
Loading