Multiple load balancing strategies#1361

zlm2012 · 2017-07-09T10:36:09Z

In this PR, several optimizations are done for load balancing:

Now load balancing will run each time starting a new request, instead of run when new socket needed to be created. t/50reverse-proxy-round-robin.t has been modified to prove it.
Added h2o_socketpool_target_status_t for recording requests on the fly & returned socket for the specific target. Added h2o_socketpool_target_status_vector_t as an argument for load balancing selector function so balancer can get status of each target.
Added void *data_for_balancer in h2o_socketpool_target_t so proxy handler can add more information for load balancing each target.
Moved round robin related code outside lib/common/socketpool.c.
Modified function h2o_socketpool_init_by_targets() so proxy handler can designate the balancer when initializing socketpool now.

kazuho · 2017-07-10T04:43:27Z

Thank you for the PR. This would be a nice step to improve the proxying capability of H2O.

Regarding the design, I see two issues.

I would prefer it if each balancer backend was able to define it's own structure of pooled sockets, instead of being enforced to use a linked-list of pools. In case of the round-robin selector, the issue is restraining us to using an O(N) method in h2o_balancer_rr_selector.

I believe that locks should be defined by each balancer backend (if necessary). Or to clarify, is it possible to remove h2o_socket_pool_t::_shared.mutex?

zlm2012 · 2017-07-10T05:23:53Z

For the first issue, pooled sockets are divided from the old h2o_socketpool_t::_shared.sockets into socket list in h2o_socketpool_target_status_t for each target depending on the socket remote peer. And a vector for status of each target is located in h2o_socketpool_t::_shared. The selector of load balancer does the same as before. The difference from before is we run the selector then consider using pooled sockets or create a new one now. So I think h2o_balancer_rr_selector would not use O(N) method.

For the second issue, I was actually planning to do so. But when I considered about destroy_expired, we need to lock all the mutex each time when recycling timeouted sockets. I was not sure if benefits from defining locks each backend could be great enough to neutralize that so I remained it there.

zlm2012 · 2017-07-10T13:56:59Z

And I just realized that we also need to add an approach for balancer to get extra information per request.

…tor.

zlm2012 · 2017-08-11T02:17:45Z

Looks like this PR has been finally completed. Some new feature has been appended:

per request extra data could be passed via override->req_extra.
least connection and weighted round robin supports are added, where weighted round robin is combined with normal round robin.
support per-pathconf balancer configuration.

Now value of proxy.reverse.backends should be either a scalar or a sequence, where the elements of the sequence could be either a scalar or a mapping with at least an entry with key url. For example:

proxy.reverse.backends:
  - url: https://your.treasure.net
    weight: 2
  - http://local.your.treasure.net

proxy.reverse.balancer is added for configure which balancer is to be used and some overall configuration of the balancer. Value should be either a scalar or a mapping, where at least one entry with key type must exists if using mapping. For example, usage of the directive could either be

proxy.reverse.balancer: least-conn

or

proxy.reverse.balancer:
  - type: round-robin
    other-key: other value (though currently no overall configurations for round robin)

Currently we only support least-conn and round-robin for proxy.reverse.balancer.

Integration test has been updated so weighted round robin is also been tested. For least connection, there's only a common load balancer integration test though. I cannot come up with a good solution for testing least connection.

…tor.

…mization

…lized, initialize `result` to zero to suppress compiler warning, from kazuho ab86e47

…...`

…specific to the global pool

…be a bit array

… is a violation of C99, as well as avoiding copy of h2o_vector

…s (the value is inceremented in `h2o_socketpool_connect` and decremented when the application is notified of a connection failure

…we would wonder how _weight_ is being defined (i.e, is `H2O_SOCKETPOOL_TARGET_MAX_WEIGHT` pre-subtracted value?)

kazuho · 2018-01-11T11:18:44Z

lib/handler/configurator/proxy.c

@@ -265,11 +266,131 @@ static int on_config_ssl_session_cache(h2o_configurator_command_t *cmd, h2o_conf
    return 0;
 }

+static int parse_balancer(h2o_configurator_command_t *cmd, yoml_t *node, h2o_socketpool_target_t **targets, size_t target_len)
+{


Do we need to define this as a separate function now?

I believe that none of the balancers being defined take an argument (except the per-target weight). Therefore, the type of the balancer attribute could be restricted to YOML_TYPE_SCALAR. All we need to do is to call strcasecmp to determine the correct balancer constructor.

kazuho · 2018-01-11T11:27:23Z

lib/handler/configurator/proxy.c

+    }
+
+    if (strcmp(lb_type_node->data.scalar, "round-robin") == 0) {
+        self->vars->conf.balancer = h2o_balancer_create_rr();


Relating to my previous comment, I believe that the selected balancer cannot be stored as a value of the context considering the fact that it is an argument to proxy.reverse.url, instead of being a separate configuration directive.

Consider the case where multiple mappings are made, in which only the first one has a balancer attribute.

paths: '/foo': proxy.reverse.url: backends: ... balancer: least-conn '/foo': proxy.reverse.url: backends: ...

The mapping for /foo will incorrectly use an instance of a least-conn balancer (shouldn't it be round-robin?). That single instance will be shared with the two proxy definitions (i.e. one for /abc and the other for /foo). I believe that it is also a bug.

kazuho · 2018-01-11T11:29:46Z

Doing my final review, I have gathered some cosmetic changes that I would like to be made in https://github.com/h2o/h2o/tree/kazuho/pr/1361.

Could you please take a look at them? I believe that the only change to code is dd88f55; others are style adjustments and comments.

zlm2012 · 2018-01-11T11:54:14Z

Doing my final review, I have gathered some cosmetic changes that I would like to be made in https://github.com/h2o/h2o/tree/kazuho/pr/1361.

Thank you! Those changes look good to me. I'll merge that branch and fix the issues mentioned.

…zation

zlm2012 · 2018-01-12T03:36:07Z

Configurator is fixed & added integration test for testing multiple balancers.

Multiple load balancing strategies

kazuho · 2018-01-14T02:38:56Z

Thank you for your efforts on the PR. Merged to master.

zlm2012 added 4 commits July 9, 2017 15:38

load balance over targets including active sockets.

785bd28

modify test for round robin to support keep-alive

c5943b5

move round robin outside lib/common/socketpool.c & some tune.

f2b0d2c

now proxy handler can set balancer when initializing socketpool

0442e4c

zlm2012 added 5 commits July 23, 2017 19:40

Merge remote-tracking branch 'upstream/master' into lb-optimization

f8a28ac

now socketpool support per request extra data for load balancer selec…

8d9bc64

…tor.

support least connection balancer.

69812d6

support weighted round robin, with integration test.

787095e

add support for overall load balancer configuration.

08029ec

zlm2012 added 17 commits September 3, 2017 17:14

support bounded consistent hashing load balancer.

0273e4f

load balance over targets including active sockets.

6835ac9

modify test for round robin to support keep-alive

3a6e010

move round robin outside lib/common/socketpool.c & some tune.

cda691e

now proxy handler can set balancer when initializing socketpool

d9433df

now socketpool support per request extra data for load balancer selec…

19715b2

…tor.

support least connection balancer.

9e4cf8c

support weighted round robin, with integration test.

0d889d9

add support for overall load balancer configuration.

ed05386

support bounded consistent hashing load balancer.

326689a

Merge branch 'lb-optimization' of github.com:zlm2012/h2o into lb-opti…

734fd8b

…mization

Merge branch 'master' of https://github.com/h2o/h2o into lb-opt-try-php7

5d03fbe

remove unnecessary assert & uncomment c++ guards

14abc5e

fix broken fuzzer driver due to new param appended to arg list

47943a6

Merge branch 'master' into lb-optimization

caea994

add assertion to trap the path that could have left result uninitia…

cd1b87d

…lized, initialize `result` to zero to suppress compiler warning, from kazuho ab86e47

Merge branch 'master' into lb-optimization

68c0c0f

kazuho added 15 commits January 11, 2018 16:19

add balancer source files to server

8caa8bf

function names should follow the convention: `h2o_<module_name>_verb_…

af3ee7b

…...`

merge add_target into h2o_socketpool_connect now that the logic is …

188a2c9

…specific to the global pool

some doc-comment

26be4bf

change type of tried from int [] to char [] (ideally they should …

507f82d

…be a bit array

get rid of struct initialization without any member (i.e. {}) which…

d2aebef

… is a violation of C99, as well as avoiding copy of h2o_vector

revert addition of an unused variable

0d52e38

be consistent in the way referring to a value

b298758

clang-format

17f382d

revert changes that make the value of pool->_shared.count errorneou…

dd88f55

…s (the value is inceremented in `h2o_socketpool_connect` and decremented when the application is notified of a connection failure

adjust doc-comment to clarify the differences

4238718

add files to xcodeproj

be8ee99

fix compiler and C conformance issues

031db75

let the name indicate that the value is subtracted by one (otherwise …

d45cd2c

…we would wonder how _weight_ is being defined (i.e, is `H2O_SOCKETPOOL_TARGET_MAX_WEIGHT` pre-subtracted value?)

fix fuzzer build error

9983284

kazuho reviewed Jan 11, 2018

View reviewed changes

zlm2012 added 4 commits January 11, 2018 21:45

Merge remote-tracking branch 'upstream/master' into lb-optimization

537e110

Merge remote-tracking branch 'upstream/kazuho/pr/1361' into lb-optimi…

6598725

…zation

fix proxy configurator according to last review.

7f87afd

add test for multiple balancer usage

e34cb05

correct the comment.

92e0d21

kazuho changed the title ~~Optimization for Load Balancing~~ Multiple load balancing strategies Jan 14, 2018

kazuho merged commit 92e0d21 into h2o:master Jan 14, 2018

kazuho added a commit that referenced this pull request Jan 14, 2018

Merge pull request #1361

8584b6d

Multiple load balancing strategies

markmaker mentioned this pull request Nov 29, 2023

Feature Request : Load balancing - Proxy - Cluster #775

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple load balancing strategies#1361

Multiple load balancing strategies #1361

zlm2012 commented Jul 9, 2017

kazuho commented Jul 10, 2017

zlm2012 commented Jul 10, 2017

zlm2012 commented Jul 10, 2017

zlm2012 commented Aug 11, 2017

kazuho Jan 11, 2018

kazuho Jan 11, 2018 •
edited

Loading

kazuho commented Jan 11, 2018

zlm2012 commented Jan 11, 2018 •
edited

Loading

zlm2012 commented Jan 12, 2018

kazuho commented Jan 14, 2018

Multiple load balancing strategies#1361

Multiple load balancing strategies #1361

Conversation

zlm2012 commented Jul 9, 2017

kazuho commented Jul 10, 2017

zlm2012 commented Jul 10, 2017

zlm2012 commented Jul 10, 2017

zlm2012 commented Aug 11, 2017

kazuho Jan 11, 2018

Choose a reason for hiding this comment

kazuho Jan 11, 2018 • edited Loading

Choose a reason for hiding this comment

kazuho commented Jan 11, 2018

zlm2012 commented Jan 11, 2018 • edited Loading

zlm2012 commented Jan 12, 2018

kazuho commented Jan 14, 2018

kazuho Jan 11, 2018 •
edited

Loading

zlm2012 commented Jan 11, 2018 •
edited

Loading