I currently have a fully async https server that runs on a dedicated thread. When I make requests to the server that don't require an additional http hop, I get just under 20,000 requests per second:
$ wrk https://127.0.0.1:18080/0/0/1 -s post.lua -d 25 -t 8 -c 500
Running 25s test @ https://127.0.0.1:18080/0/0/1
8 threads and 500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 29.66ms 3.04ms 36.52ms 94.96%
Req/Sec 2.05k 100.42 2.54k 85.15%
408727 requests in 25.02s, 74.06MB read
Requests/sec: 16338.76
Transfer/sec: 2.96MB
However, I have an endpoint that forwards the request to an additional server (localhost server running http, so no tls handshaking required). When I hit this endpoint, my requests per second completely tanks:
$ wrk https://127.0.0.1:18080/0/0/2 -s post.lua -d 25 -t 8 -c 500
Running 25s test @ https://127.0.0.1:18080/0/0/2
8 threads and 500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 158.85ms 13.58ms 211.73ms 85.50%
Req/Sec 389.13 50.99 510.00 74.25%
77619 requests in 25.06s, 3.92MB read
Non-2xx or 3xx responses: 77619
Requests/sec: 3097.48
Transfer/sec: 160.32KB
Keep in mind that in the above test, the additional server isn't even running, so the connections are refused. There's no work being done on the server and no responses to parse.
My current architecture has a client pool that maintains an array of available clients so that I can reuse sockets. This client pool creates an asio::strand using an executor from an asio::thread_pool of 8 threads. I have a few questions:
-
What is the expected performance hit of a server making requests to an additional server? What % of the baseline performance should I expect to maintain? In the first test where it's just my server hitting the DB directly and returning, I get 16,000 requests per second. In the test where the server fires off an additional http request, I get 3,000 requests per second. That's only 18% of the original performance and that seems to be low.
-
What is the overhead of posting to strands? And maybe more specifically, what is the overhead of posting to a strand that's executing across a thread pool? Should I move my client pool to a dedicated thread instead of allowing it to execute across a thread pool?
-
What is the recommended architecture for something like this? Both the server and the embedded client are fully async, but I still don't think I'd want them on the same thread. The client needs to process json and I'd like to keep that off the server thread.
I'm not really sure where to start with investigating. If the latency of posting to strands is low, then there's an issue with my client pool logic. If the latency of posting to strands is high, then it's an issue with how I'm leveraging them. Any input on this would be appreciated as it feels like I'm at a loss on how to proceed.
I currently have a fully async https server that runs on a dedicated thread. When I make requests to the server that don't require an additional http hop, I get just under 20,000 requests per second:
However, I have an endpoint that forwards the request to an additional server (localhost server running http, so no tls handshaking required). When I hit this endpoint, my requests per second completely tanks:
Keep in mind that in the above test, the additional server isn't even running, so the connections are refused. There's no work being done on the server and no responses to parse.
My current architecture has a client pool that maintains an array of available clients so that I can reuse sockets. This client pool creates an asio::strand using an executor from an asio::thread_pool of 8 threads. I have a few questions:
What is the expected performance hit of a server making requests to an additional server? What % of the baseline performance should I expect to maintain? In the first test where it's just my server hitting the DB directly and returning, I get 16,000 requests per second. In the test where the server fires off an additional http request, I get 3,000 requests per second. That's only 18% of the original performance and that seems to be low.
What is the overhead of posting to strands? And maybe more specifically, what is the overhead of posting to a strand that's executing across a thread pool? Should I move my client pool to a dedicated thread instead of allowing it to execute across a thread pool?
What is the recommended architecture for something like this? Both the server and the embedded client are fully async, but I still don't think I'd want them on the same thread. The client needs to process json and I'd like to keep that off the server thread.
I'm not really sure where to start with investigating. If the latency of posting to strands is low, then there's an issue with my client pool logic. If the latency of posting to strands is high, then it's an issue with how I'm leveraging them. Any input on this would be appreciated as it feels like I'm at a loss on how to proceed.