Notes on docker swarm

[njordr]

Hey.

I'm trying to figure out if I can/how I can use sidecar with docker swarm

I created a cluster (single host) with 4 services:

• portainer agent
• portainer server
• sidecar
• nginx

Portainer server bind port 9000
nginx bind port 80 and 443

All the services are on the ingress network for two reason:

• first is the design I want to use (all in the ingress network except for envoy binded to the host network)
• second because a limitation of docker (I suppose, do not dig too much in it) where I cannot create a service bound to both ingress and host network

Expected result:

• sidecar recognizes 3 containers, two of them with ports bound
• health check for nginx works

Desiderata:

• sidecar recognizes the containers with the right ports also if the ports are not bound but only exposed or defined by ServicePort var (in this way I can have all the services without ports bound to host to avoid to handle overlapping ports). This is similar to rancher lb where an haproxy container bind to 80 and 443 and forward the requests to the different backend cointainers and none of them has ports bind)

Actual result:

• sidecar cannot retrieve service ports
• if I use this label for checking the service HealthCheckArgs="http://{{ host }}:{{ tcp 80 }}/", host variable retrieves a different IP from the one assigned (in this case it tries to check the service on IP 10.255.0.5 but the service is on IP 10.255.0.10). you can see that in the log below time="2018-09-12T19:29:39Z" level=info msg="Adding health check: HttpGet (ID: 59844989cd37), Args: http://10.255.0.5:-1/",

I could have misunderstood sidecar architecture/swarm functionalities so this probably is not a bug :)

Here are sidecar debug logs

time="2018-09-12T19:29:38Z" level=info msg="Sidecar ------------------------------------------",
time="2018-09-12T19:29:38Z" level=info msg="  * Sidecar:",
time="2018-09-12T19:29:38Z" level=info msg="    * ExcludeIPs: [192.168.168.168]",
time="2018-09-12T19:29:38Z" level=info msg="    * Discovery: [docker]",
time="2018-09-12T19:29:38Z" level=info msg="    * StatsAddr: ",
time="2018-09-12T19:29:38Z" level=info msg="    * PushPullInterval: 20s",
time="2018-09-12T19:29:38Z" level=info msg="    * GossipMessages: 15",
time="2018-09-12T19:29:38Z" level=info msg="    * LoggingFormat: ",
time="2018-09-12T19:29:38Z" level=info msg="    * LoggingLevel: debug",
time="2018-09-12T19:29:38Z" level=info msg="    * DefaultCheckEndpoint: /version",
time="2018-09-12T19:29:38Z" level=info msg="    * Seeds: [127.0.0.1]",
time="2018-09-12T19:29:38Z" level=info msg="    * ClusterName: syshell",
time="2018-09-12T19:29:38Z" level=info msg="    * AdvertiseIP: ",
time="2018-09-12T19:29:38Z" level=info msg="    * BindPort: 7950",
time="2018-09-12T19:29:38Z" level=info msg="  * DockerDiscovery:",
time="2018-09-12T19:29:38Z" level=info msg="    * DockerURL: unix:///var/run/docker.sock",
time="2018-09-12T19:29:38Z" level=info msg="  * StaticDiscovery:",
time="2018-09-12T19:29:38Z" level=info msg="    * ConfigFile: static.json",
time="2018-09-12T19:29:38Z" level=info msg="  * Services:",
time="2018-09-12T19:29:38Z" level=info msg="    * NameMatch: ",
time="2018-09-12T19:29:38Z" level=info msg="    * ServiceNamer: docker_label",
time="2018-09-12T19:29:38Z" level=info msg="    * NameLabel: ServiceName",
time="2018-09-12T19:29:38Z" level=info msg="  * HAproxy:",
time="2018-09-12T19:29:38Z" level=info msg="    * ReloadCmd: ",
time="2018-09-12T19:29:38Z" level=info msg="    * VerifyCmd: ",
time="2018-09-12T19:29:38Z" level=info msg="    * BindIP: 192.168.168.168",
time="2018-09-12T19:29:38Z" level=info msg="    * TemplateFile: views/haproxy.cfg",
time="2018-09-12T19:29:38Z" level=info msg="    * ConfigFile: /etc/haproxy.cfg",
time="2018-09-12T19:29:38Z" level=info msg="    * PidFile: /var/run/haproxy.pid",
time="2018-09-12T19:29:38Z" level=info msg="    * Disable: false",
time="2018-09-12T19:29:38Z" level=info msg="    * User: haproxy",
time="2018-09-12T19:29:38Z" level=info msg="    * Group: haproxy",
time="2018-09-12T19:29:38Z" level=info msg="    * UseHostnames: false",
time="2018-09-12T19:29:38Z" level=info msg="  * Listeners:",
time="2018-09-12T19:29:38Z" level=info msg="    * Urls: []",
time="2018-09-12T19:29:38Z" level=info msg=--------------------------------------------------,
time="2018-09-12T19:29:38Z" level=debug msg="NodeMeta(): 512",
time="2018-09-12T19:29:38Z" level=debug msg="NotifyJoin(): c0ab1194b32f {\"ClusterName\":\"syshell\",\"State\":\"Running\"}",
time="2018-09-12T19:29:38Z" level=debug msg="memberlist: Initiating push/pull sync with: 127.0.0.1:7950",
time="2018-09-12T19:29:38Z" level=debug msg="LocalState(): %!b(bool=true)",
time="2018-09-12T19:29:38Z" level=debug msg="memberlist: Stream connection from=127.0.0.1:56876",
time="2018-09-12T19:29:38Z" level=debug msg="LocalState(): %!b(bool=true)",
time="2018-09-12T19:29:38Z" level=debug msg="MergeRemoteState(): {\"Servers\":{},\"LastChanged\":\"1970-01-01T00:00:00Z\",\"ClusterName\":\"\",\"Hostname\":\"c0ab1194b32f\"} true",
time="2018-09-12T19:29:38Z" level=debug msg="Merging state: Services ------------------------------\n",
time="2018-09-12T19:29:38Z" level=debug msg="MergeRemoteState(): {\"Servers\":{},\"LastChanged\":\"1970-01-01T00:00:00Z\",\"ClusterName\":\"\",\"Hostname\":\"c0ab1194b32f\"} true",
time="2018-09-12T19:29:38Z" level=debug msg="Merging state: Services ------------------------------\n",
time="2018-09-12T19:29:38Z" level=info msg="Setting up signal handlers",
time="2018-09-12T19:29:38Z" level=debug msg="Member: c0ab1194b32f 10.255.0.5",
time="2018-09-12T19:29:38Z" level=debug msg="Meta: {\"ClusterName\":\"syshell\",\"State\":\"Running\"}",
time="2018-09-12T19:29:38Z" level=debug msg="AddListener(): added HAproxy, new count 1",
time="2018-09-12T19:29:38Z" level=debug msg="Services ------------------------------\n\nCluster Hosts -------------------------\n    c0ab1194b32f\n---------------------------------------",
time="2018-09-12T19:29:38Z" level=debug msg="Event: &docker.APIEvents{Action:\"start\", Type:\"container\", Actor:docker.APIActor{ID:\"c0ab1194b32f60b2e73c0dfdda666cd10d6ad7aac490c7e3d5a19c3e2f108b33\", Attributes:map[string]string{\"com.docker.swarm.service.name\":\"sidecar\", \"image\":\"gonitro/sidecar:f56be36@sha256:195709bed8256d705f6c0143826401c0c3620bfb580873ddc19c82c77c6c9e40\", \"com.docker.swarm.service.id\":\"2za4vy219t6lhb0xaex305no3\", \"com.docker.swarm.task\":\"\", \"com.docker.swarm.task.id\":\"1yfl8rrxwbxordriacufb3m86\", \"com.docker.swarm.task.name\":\"sidecar.1.1yfl8rrxwbxordriacufb3m86\", \"name\":\"sidecar.1.1yfl8rrxwbxordriacufb3m86\", \"SidecarDiscover\":\"false\", \"com.docker.swarm.node.id\":\"lp7uebppc477us10ec3fkxo82\"}}, Status:\"start\", ID:\"c0ab1194b32f60b2e73c0dfdda666cd10d6ad7aac490c7e3d5a19c3e2f108b33\", From:\"gonitro/sidecar:f56be36@sha256:195709bed8256d705f6c0143826401c0c3620bfb580873ddc19c82c77c6c9e40\", Time:1536780578, TimeNano:1536780578497886751}\n",
time="2018-09-12T19:29:38Z" level=debug msg="GetBroadcasts(): 3 1263",
time="2018-09-12T19:29:38Z" level=debug msg="GetBroadcasts(): 3 1263",
time="2018-09-12T19:29:38Z" level=debug msg="GetBroadcasts(): 3 1263",
time="2018-09-12T19:29:38Z" level=debug msg="GetBroadcasts(): 3 1263",
time="2018-09-12T19:29:38Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:38Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:39Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:39Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:39Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:39Z" level=debug msg="Found container with no 'ServiceName' label: 59844989cd37932f999f8e5085c0d77b37813698212b005daed5c1b8c6bcca01 (/nginx.1.nsocaf2zihu8vq7d6ajkg5n1r), returning 'nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3'",
time="2018-09-12T19:29:39Z" level=debug msg="Found container with no 'ServiceName' label: b6932c9d02ab3abea93488d2ef53dcaae6cafaf73236946db9451f74e8714eac (/portainer_portainer.1.ssku2iw886ilhq4d7btlvuulk), returning 'portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910'",
time="2018-09-12T19:29:39Z" level=debug msg="Found container with no 'ServiceName' label: d8fd8f63de007c2756f0a271b01ac84deaa6ca84e7a9fc4f8f4c46112fe9a482 (/portainer_agent.lp7uebppc477us10ec3fkxo82.af26lbc28lryaut4d7hvxx7gv), returning 'portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be'",
time="2018-09-12T19:29:39Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:39Z" level=debug msg="Note: Not enough space to fit any messages or message was nil",
time="2018-09-12T19:29:39Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:39Z" level=warning msg="Unable to find ServicePort 80 for service 59844989cd37",
time="2018-09-12T19:29:39Z" level=info msg="Adding health check: HttpGet (ID: 59844989cd37), Args: http://10.255.0.5:-1/",
time="2018-09-12T19:29:39Z" level=warning msg="Adding check for service portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910 (id: b6932c9d02ab) failed. Got check type: , check args: .",
time="2018-09-12T19:29:39Z" level=warning msg="Using default check for service portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910 (id: b6932c9d02ab).",
time="2018-09-12T19:29:39Z" level=info msg="Adding health check:  (ID: b6932c9d02ab), Args: ",
time="2018-09-12T19:29:39Z" level=warning msg="Adding check for service portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be (id: d8fd8f63de00) failed. Got check type: , check args: .",
time="2018-09-12T19:29:39Z" level=warning msg="Using default check for service portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be (id: d8fd8f63de00).",
time="2018-09-12T19:29:39Z" level=info msg="Adding health check:  (ID: d8fd8f63de00), Args: ",
time="2018-09-12T19:29:40Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:40Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:40Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:40Z" level=debug msg="Notifying listeners of change at 2018-09-12 19:29:39.503168671 +0000 UTC",
time="2018-09-12T19:29:40Z" level=debug msg="Notifying listeners of change at 2018-09-12 19:29:39.503230986 +0000 UTC",
time="2018-09-12T19:29:40Z" level=debug msg="Notifying listeners of change at 2018-09-12 19:29:39.503259669 +0000 UTC",
time="2018-09-12T19:29:40Z" level=info msg="State change event from c0ab1194b32f",
time="2018-09-12T19:29:40Z" level=debug msg="Member: c0ab1194b32f 10.255.0.5",
time="2018-09-12T19:29:40Z" level=debug msg="Meta: {\"ClusterName\":\"syshell\",\"State\":\"Running\"}",
time="2018-09-12T19:29:40Z" level=debug msg="Found container with no 'ServiceName' label: 59844989cd37932f999f8e5085c0d77b37813698212b005daed5c1b8c6bcca01 (/nginx.1.nsocaf2zihu8vq7d6ajkg5n1r), returning 'nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3'",
time="2018-09-12T19:29:40Z" level=debug msg="Found container with no 'ServiceName' label: b6932c9d02ab3abea93488d2ef53dcaae6cafaf73236946db9451f74e8714eac (/portainer_portainer.1.ssku2iw886ilhq4d7btlvuulk), returning 'portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910'",
time="2018-09-12T19:29:40Z" level=debug msg="Found container with no 'ServiceName' label: d8fd8f63de007c2756f0a271b01ac84deaa6ca84e7a9fc4f8f4c46112fe9a482 (/portainer_agent.lp7uebppc477us10ec3fkxo82.af26lbc28lryaut4d7hvxx7gv), returning 'portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be'",
time="2018-09-12T19:29:40Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:40Z" level=debug msg="Note: Not enough space to fit any messages or message was nil",
time="2018-09-12T19:29:40Z" level=debug msg="Starting to broadcast",
time="2018-09-12T19:29:40Z" level=debug msg="Completing broadcast",
time="2018-09-12T19:29:40Z" level=debug msg="Services ------------------------------\n  c0ab1194b32f: (1.0 sec ago)\n      59844989cd37 nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3                 nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3  1.0 sec ago     Unhealthy\n      d8fd8f63de00 portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be                 portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be  1.0 sec ago     Alive    \n      b6932c9d02ab portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910                 portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910  1.0 sec ago     Alive    \n\n\nCluster Hosts -------------------------\n    c0ab1194b32f\n---------------------------------------",
time="2018-09-12T19:29:40Z" level=info msg="State change event from c0ab1194b32f",
time="2018-09-12T19:29:40Z" level=info msg="State change event from c0ab1194b32f",
time="2018-09-12T19:29:40Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:41Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:41Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:41Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:41Z" level=debug msg="Running checks",
time="2018-09-12T19:29:41Z" level=debug msg="Error executing check, status UNKNOWN: (id 59844989cd37)",
time="2018-09-12T19:29:41Z" level=debug msg="Found container with no 'ServiceName' label: 59844989cd37932f999f8e5085c0d77b37813698212b005daed5c1b8c6bcca01 (/nginx.1.nsocaf2zihu8vq7d6ajkg5n1r), returning 'nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3'",
time="2018-09-12T19:29:41Z" level=debug msg="Found container with no 'ServiceName' label: b6932c9d02ab3abea93488d2ef53dcaae6cafaf73236946db9451f74e8714eac (/portainer_portainer.1.ssku2iw886ilhq4d7btlvuulk), returning 'portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910'",
time="2018-09-12T19:29:41Z" level=debug msg="Found container with no 'ServiceName' label: d8fd8f63de007c2756f0a271b01ac84deaa6ca84e7a9fc4f8f4c46112fe9a482 (/portainer_agent.lp7uebppc477us10ec3fkxo82.af26lbc28lryaut4d7hvxx7gv), returning 'portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be'",
time="2018-09-12T19:29:41Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:41Z" level=debug msg="Note: Not enough space to fit any messages or message was nil",
time="2018-09-12T19:29:41Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:41Z" level=debug msg="Sending broadcast 3 msgs 353 1st length",
time="2018-09-12T19:29:41Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:42Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:42Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:42Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:42Z" level=debug msg="Found container with no 'ServiceName' label: 59844989cd37932f999f8e5085c0d77b37813698212b005daed5c1b8c6bcca01 (/nginx.1.nsocaf2zihu8vq7d6ajkg5n1r), returning 'nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3'",
time="2018-09-12T19:29:42Z" level=debug msg="Found container with no 'ServiceName' label: b6932c9d02ab3abea93488d2ef53dcaae6cafaf73236946db9451f74e8714eac (/portainer_portainer.1.ssku2iw886ilhq4d7btlvuulk), returning 'portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910'",
time="2018-09-12T19:29:42Z" level=debug msg="Found container with no 'ServiceName' label: d8fd8f63de007c2756f0a271b01ac84deaa6ca84e7a9fc4f8f4c46112fe9a482 (/portainer_agent.lp7uebppc477us10ec3fkxo82.af26lbc28lryaut4d7hvxx7gv), returning 'portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be'",
time="2018-09-12T19:29:42Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:42Z" level=debug msg="Note: Not enough space to fit any messages or message was nil",
time="2018-09-12T19:29:42Z" level=debug msg="Member: c0ab1194b32f 10.255.0.5",
time="2018-09-12T19:29:42Z" level=debug msg="Meta: {\"ClusterName\":\"syshell\",\"State\":\"Running\"}",
time="2018-09-12T19:29:42Z" level=debug msg="Services ------------------------------\n  c0ab1194b32f: (1.0 sec ago)\n      59844989cd37 nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3                 nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3  1.0 sec ago     Unhealthy\n      d8fd8f63de00 portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be                 portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be  1.0 sec ago     Alive    \n      b6932c9d02ab portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910                 portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910  1.0 sec ago     Alive    \n\n\nCluster Hosts -------------------------\n    c0ab1194b32f\n---------------------------------------",
time="2018-09-12T19:29:42Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:42Z" level=debug msg="Note: Not enough space to fit any messages or message was nil",
time="2018-09-12T19:29:43Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:43Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:43Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:43Z" level=debug msg="Found container with no 'ServiceName' label: 59844989cd37932f999f8e5085c0d77b37813698212b005daed5c1b8c6bcca01 (/nginx.1.nsocaf2zihu8vq7d6ajkg5n1r), returning 'nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3'",
time="2018-09-12T19:29:43Z" level=debug msg="Found container with no 'ServiceName' label: b6932c9d02ab3abea93488d2ef53dcaae6cafaf73236946db9451f74e8714eac (/portainer_portainer.1.ssku2iw886ilhq4d7btlvuulk), returning 'portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910'",
time="2018-09-12T19:29:43Z" level=debug msg="Found container with no 'ServiceName' label: d8fd8f63de007c2756f0a271b01ac84deaa6ca84e7a9fc4f8f4c46112fe9a482 (/portainer_agent.lp7uebppc477us10ec3fkxo82.af26lbc28lryaut4d7hvxx7gv), returning 'portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be'",
time="2018-09-12T19:29:43Z" level=debug msg="Event: &docker.APIEvents{Action:\"update\", Type:\"service\", Actor:docker.APIActor{ID:\"2za4vy219t6lhb0xaex305no3\", Attributes:map[string]string{\"updatestate.old\":\"updating\", \"name\":\"sidecar\", \"updatestate.new\":\"completed\"}}, Status:\"service:update\", ID:\"2za4vy219t6lhb0xaex305no3\", From:\"\", Time:1536780583, TimeNano:1536780583628683303}\n",
time="2018-09-12T19:29:43Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:43Z" level=debug msg="Note: Not enough space to fit any messages or message was nil",
time="2018-09-12T19:29:43Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:44Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:44Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:44Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:44Z" level=debug msg="Running checks",
time="2018-09-12T19:29:44Z" level=debug msg="Error executing check, status UNKNOWN: (id 59844989cd37)",
time="2018-09-12T19:29:44Z" level=debug msg="Found container with no 'ServiceName' label: 59844989cd37932f999f8e5085c0d77b37813698212b005daed5c1b8c6bcca01 (/nginx.1.nsocaf2zihu8vq7d6ajkg5n1r), returning 'nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3'",
time="2018-09-12T19:29:44Z" level=debug msg="Found container with no 'ServiceName' label: b6932c9d02ab3abea93488d2ef53dcaae6cafaf73236946db9451f74e8714eac (/portainer_portainer.1.ssku2iw886ilhq4d7btlvuulk), returning 'portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910'",
time="2018-09-12T19:29:44Z" level=debug msg="Found container with no 'ServiceName' label: d8fd8f63de007c2756f0a271b01ac84deaa6ca84e7a9fc4f8f4c46112fe9a482 (/portainer_agent.lp7uebppc477us10ec3fkxo82.af26lbc28lryaut4d7hvxx7gv), returning 'portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be'",
time="2018-09-12T19:29:44Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:44Z" level=debug msg="Note: Not enough space to fit any messages or message was nil",
time="2018-09-12T19:29:44Z" level=debug msg="Member: c0ab1194b32f 10.255.0.5",
time="2018-09-12T19:29:44Z" level=debug msg="Meta: {\"ClusterName\":\"syshell\",\"State\":\"Running\"}",
time="2018-09-12T19:29:44Z" level=debug msg="Services ------------------------------\n  c0ab1194b32f: (1.0 sec ago)\n      59844989cd37 nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3                 nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3  1.0 sec ago     Unhealthy\n      d8fd8f63de00 portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be                 portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be  1.0 sec ago     Alive    \n      b6932c9d02ab portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910                 portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910  1.0 sec ago     Alive    \n\n\nCluster Hosts -------------------------\n    c0ab1194b32f\n---------------------------------------",
time="2018-09-12T19:29:44Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:44Z" level=debug msg="Note: Not enough space to fit any messages or message was nil",
time="2018-09-12T19:29:45Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:45Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:45Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:45Z" level=debug msg="Found container with no 'ServiceName' label: 59844989cd37932f999f8e5085c0d77b37813698212b005daed5c1b8c6bcca01 (/nginx.1.nsocaf2zihu8vq7d6ajkg5n1r), returning 'nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3'",
time="2018-09-12T19:29:45Z" level=debug msg="Found container with no 'ServiceName' label: b6932c9d02ab3abea93488d2ef53dcaae6cafaf73236946db9451f74e8714eac (/portainer_portainer.1.ssku2iw886ilhq4d7btlvuulk), returning 'portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910'",
time="2018-09-12T19:29:45Z" level=debug msg="Found container with no 'ServiceName' label: d8fd8f63de007c2756f0a271b01ac84deaa6ca84e7a9fc4f8f4c46112fe9a482 (/portainer_agent.lp7uebppc477us10ec3fkxo82.af26lbc28lryaut4d7hvxx7gv), returning 'portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be'",
time="2018-09-12T19:29:45Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:45Z" level=debug msg="Note: Not enough space to fit any messages or message was nil",
time="2018-09-12T19:29:45Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:46Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:46Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:46Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:46Z" level=debug msg="Found container with no 'ServiceName' label: 59844989cd37932f999f8e5085c0d77b37813698212b005daed5c1b8c6bcca01 (/nginx.1.nsocaf2zihu8vq7d6ajkg5n1r), returning 'nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3'",
time="2018-09-12T19:29:46Z" level=debug msg="Found container with no 'ServiceName' label: b6932c9d02ab3abea93488d2ef53dcaae6cafaf73236946db9451f74e8714eac (/portainer_portainer.1.ssku2iw886ilhq4d7btlvuulk), returning 'portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910'",
time="2018-09-12T19:29:46Z" level=debug msg="Found container with no 'ServiceName' label: d8fd8f63de007c2756f0a271b01ac84deaa6ca84e7a9fc4f8f4c46112fe9a482 (/portainer_agent.lp7uebppc477us10ec3fkxo82.af26lbc28lryaut4d7hvxx7gv), returning 'portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be'",
time="2018-09-12T19:29:46Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:46Z" level=debug msg="Note: Not enough space to fit any messages or message was nil",
time="2018-09-12T19:29:46Z" level=debug msg="Member: c0ab1194b32f 10.255.0.5",
time="2018-09-12T19:29:46Z" level=debug msg="Meta: {\"ClusterName\":\"syshell\",\"State\":\"Running\"}",
time="2018-09-12T19:29:46Z" level=debug msg="Services ------------------------------\n  c0ab1194b32f: (1.0 sec ago)\n      59844989cd37 nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3                 nginx:latest@sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3  1.0 sec ago     Unhealthy\n      d8fd8f63de00 portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be                 portainer/agent:latest@sha256:468f58ed0ae81cdae7dd377d813efbce6011709d1fe3b539f515d2e0623c40be  1.0 sec ago     Alive    \n      b6932c9d02ab portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910                 portainer/portainer:latest@sha256:ab096b92ed177b47adfa8a9a99e304d36596efa557b9627c066cee164cc39910  1.0 sec ago     Alive    \n\n\nCluster Hosts -------------------------\n    c0ab1194b32f\n---------------------------------------",
time="2018-09-12T19:29:46Z" level=debug msg="GetBroadcasts(): 3 1398",
time="2018-09-12T19:29:46Z" level=debug msg="Note: Not enough space to fit any messages or message was nil",
time="2018-09-12T19:29:47Z" level=debug msg="GetBroadcasts(): 3 1398",

[mihaitodor]

I'll let @relistan comment on the general architecture, but, from a code perspective:

• Here is the function which parses the HealthCheckArgs for your service. You will notice that the host resolves to m.DefaultCheckHost, which, unless specified explicitly via SIDECAR_ADVERTISE_IP, comes from here, which is determined in here to be the first private IPv4 address (that is not excluded via SIDECAR_EXCLUDE_IPS) returned by net.InterfaceAddrs().
• If you look closer at templateCheckArgs, you'll see that, in the case of tcp, it calls svc.PortForServicePort(p, "tcp"), which means that it will inspect the ServicePort_80=xxxx label on your service container, so the health check will be directed at host:xxxx, where host is the host which runs Envoy and xxxx is the fixed port which Envoy will redirect to the dynamic port that Docker assigns to the published port 80 when you run your service with docker run -p80 ....
• Here we fetch an instance of HttpGetCmd (for HTTP health checks), which simply calls http.Get(args) when invoked from here, where args contains the instantiation of the HealthCheckArgs template, as described above.
• Since you're using Envoy as a proxy, why is HAPROXY_DISABLE set to false in the log you pasted above?
• Like I said here, you might need to fiddle with HAPROXY_BIND_IP (unfortunately, the variable name has not been refactored when we added the Envoy integration, but it's being used by Envoy as well. See here and here). Its default value is 192.168.168.168, but you can set it to another private routable IP. Note that when Sidecar is running in host networking mode, it calls ip addr add $HAPROXY_BIND_IP/32 dev lo here ONLY if you set this variable explicitly when running the container because s6 doesn't know about the default value set in the sidecar executable. @relistan is there a reason why we don't set ENV HAPROXY_BIND_IP="192.168.168.168" in the Sidecar Dockerfile?

[relistan]

@njordr What I think you are saying is that you want to have the ServicePort be the same/native port but bound on a different address so that you don't have port conflicts? If so:

1. You still need to bind your ports in Docker, but you can configure Docker not to bind them on all addresses (defaults to 0.0.0.0). Generally you'd have them bound on localhost or some other internal network. You can either do this in the daemon configuration, for all containers, or specify it per container. My guess is setting the default is what you will want.

2. You can tell Envoy or HAproxy to bind on an IP specifically (not 0.0.0.0). In this way you can use the same ports and not conflict between the proxy and Docker.

Did I understand correctly what you wanted?

Also, as @mihaitodor says, the health check is going to refuse to run against a port that is not exposed/bound.

[relistan]

@mihaitodor I don't know why it's not in the Dockerfile

[njordr]

@relistan Thanks for the info. I think what you suggest could solve my problems, merged with the advises from @mihaitodor

I'll try in the week-end and let you know :)

[relistan]

@njordr it looks like you are in Dublin. If you want, I'd be happy to grab a coffee and talk through what you're trying to work out.

[njordr]

@relistan I'm happy to grab a coffee too