Promtimer should support alternate address (was: promtimer issue working with remote AWS instances)

[stevewatanabe]

I was encountering an issue when starting promtimer against a remote AWS instance. I put a printf in urllib/request.py in do_open

1347             try:
1348                 print(f"{req.get_method()} {req.selector} {req.data} {headers}")
1349                 h.request(req.get_method(), req.selector, req.data, headers,
1350                           encode_chunked=req.has_header('Transfer-encoding'))
1351             except OSError as err: # timeout error
1352                 raise URLError(err)
1353             r = h.getresponse()

and see this crash. Note the /pools/default/buckets is being done against the localhost (127.0.0.1) which is my mac. This is a single node, just created cluster so maybe that's a reason

steve.watanabe @ ~/promtimer/bin (master) $ ./promtimer --cluster ec2-52-23-193-116.compute-1.amazonaws.com:8091 --user Administrator --password asdasd
GET /pools/default/nodeServices None {'Host': 'ec2-52-23-193-116.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
GET /pools/default/nodeServices None {'Host': 'ec2-52-23-193-116.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Connection': 'close'}
GET /pools/default/buckets None {'Host': '127.0.0.1:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1349, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 975, in send
    self.connect()
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 941, in connect
    self.sock = self._create_connection(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/steve.watanabe/promtimer/bin/../promtimer/promtimer.py", line 336, in <module>
    main()
  File "/Users/steve.watanabe/promtimer/bin/../promtimer/promtimer.py", line 302, in main
    buckets = stats_sources[0].get_buckets()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/steve.watanabe/promtimer/promtimer/cbstats.py", line 412, in get_buckets
    response = util.execute_request('{}:{}'.format(self.host(), self.port()),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/steve.watanabe/promtimer/promtimer/util.py", line 139, in execute_request
    response = opener.open(request)
               ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 519, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1378, in http_open
    return self.do_open(http.client.HTTPConnection, req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1352, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 61] Connection refused>

I then add a second node to the cluster and try again. Note it uses a non-local IP address (172.31.15.151) but not the external one and so the connection times out.

steve.watanabe @ ~/promtimer/bin (master) $ ./promtimer --cluster ec2-52-23-193-116.compute-1.amazonaws.com:8091 --user Administrator --password asdasd
GET /pools/default/nodeServices None {'Host': 'ec2-52-23-193-116.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
GET /pools/default/nodeServices None {'Host': 'ec2-52-23-193-116.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Connection': 'close'}
GET /pools/default/buckets None {'Host': '172.31.15.151:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1349, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 975, in send
    self.connect()
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 941, in connect
    self.sock = self._create_connection(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

I skimmed the promtimer code and see it is using /pools/default/nodeServices

Running on my node

      "thisNode": true,
      "hostname": "172.31.15.151"

      "hostname": "ec2-54-88-70-0.compute-1.amazonaws.com"

and

      "hostname": "172.31.15.151"

      "thisNode": true,
      "hostname": "ec2-54-88-70-0.compute-1.amazonaws.com"

Having entered this issue perhaps there's two issues: 1) remote promtimer doesn't work against a node that doesn't have a "hostname" in the results from /pools/default/nodeServices and 2) ns_server may not be returning the right hostname...though I'm not sure how ns_server would know the IP isn't the external IP.

[stevewatanabe]

Thinking that it might work against the second node it too used the non-external hostname and thus timed out.

steve.watanabe @ ~/promtimer/bin (master) $ ./promtimer --cluster ec2-54-88-70-0.compute-1.amazonaws.com:8091 --user Administrator --password asdasd
GET /pools/default/nodeServices None {'Host': 'ec2-54-88-70-0.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
GET /pools/default/nodeServices None {'Host': 'ec2-54-88-70-0.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Connection': 'close'}
GET /pools/default/buckets None {'Host': '172.31.15.151:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}

[dave-finlay]

Hey Steve

pools/default/nodeServices only returns a hostname when the node has a hostname which is something other than cb.local(which is a logical name for either 127.0.0.1 or ::1 depending on the address family. The reasons for not including the hostname as 127.0.0.1 should probably be clear.)

When you bring up a standalone node and don't configure it with a hostname explicitly, it will pick up the name cb.local which is why you're not getting a hostname with 1 node, I guess. We can improve this behavior in Promtimer to use the cluster address instead of 127.0.0.1. This would be a small patch.

When you add a second node, you did so using the internal addresses - and so these are returned in the pools/default/nodeServices response. If you want this use case to work, you should use the external addresses when setting up the cluster and adding the nodes. Or you should use "alternate address" and we'll need to add support in Promtimer to use that. (This would be an improvement).

[stevewatanabe]

Hi Dave,
I started from scratch and....
For the first node I installed bits and went to the UI http://ec2-52-23-193-116.compute-1.amazonaws.com:8091/ and "setup the cluster" with defaults. I then added "ec2-54-88-70-0.compute-1.amazonaws.com" as the second node and did a rebalance.

Both nodes show the Servers as

I'm not sure how to setup the first node and tell it to use ec2-52-23-193-116.compute-1.amazonaws.com as the hostname.

[dave-finlay]

I'm not sure how to setup the first node and tell it to use ec2-52-23-193-116.compute-1.amazonaws.com as the hostname.

Enter it in the hostname box here:

or of course you can use the CLI.

[stevewatanabe]

That did the trick! I am able to start promtimer against either of the two nodes. Thanks for the help Dave!

[dave-finlay]

Hey Steve - I've created a PR that will allow Promtimer to connect to a remote single node instance running on localhost: #51. Would you be able to give it a try? It doesn't solve the problem of having Promtimer be able to connect via alternate address - but it should work for single nodes, which can be convenient.

[stevewatanabe]

Tested on AWS linux instance. Thanks for the quick fix.