As discussed in T361950#9934152, we don't want to go through LVS for internal traffic. Solution proposed by @Vgutierrez is to use local envoy to do the routing / load balancing.
AC:
- federation traffic in WDQS does not go through LVS
Gehel | |
Jul 1 2024, 6:50 PM |
F56307972: wdqs-graph-split-traffic.png | |
Jul 9 2024, 10:02 AM |
As discussed in T361950#9934152, we don't want to go through LVS for internal traffic. Solution proposed by @Vgutierrez is to use local envoy to do the routing / load balancing.
AC:
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Open | None | T335067 Epic: Wikidata Query Service stabilization | |||
Open | None | T337013 [Epic] Splitting the graph in WDQS | |||
Open | None | T364363 [Epic] Productionize federated wdqs graph-split endpoints | |||
Invalid | Gehel | T368972 Use Envoy instead of LVS to route internal federation traffic for WDQS |
Discussed this at pairing with @RKemper today, we were curious how to implement in envoy.
I'm far from being an envoy expert, but it looks like we could use envoy as a load balancer by defining a static envoy config similar to cluster.yaml.erb and listerner.yaml.erb . So envoy definitely supports this, but it's not a feature that's in use anywhere at WMF as far as I can tell. Tagging serviceops to comment on whether or not configuring this way is recommended.
ServiceOps, most of the context you need is in the parent ticket. Thanks for looking!
In the below post I'm trying to aggregate info from the previous phab ticket and in-meeting discussions to give service-ops a jumping-off point to start from
(Some of these details might be wrong, but I think the main points are correct)
Previous way: user submits a query at query.wikidata.org-> ATS -> LVS -> nginx -> blazegraph
New way: (if a federated query) user submits a query at query-(main|scholarly).wikidata.org -> ATS -> LVS -> nginx -> blazegraph -> http client inside blazegraph reaches out to endpoint specified in federation request
(Non-federated queries are equivalent to the previous non-graph-split way of doing things so are irrelevant for our purposes here)
Previous way: throttling done via token-bucket algorithm on 2-tuple of (IP addr, user agent)
New way: Still same token-bucket algorithm, however we need to identify internal federation queries and disable throttling in that use case (throttling will still occur at the external level; ie the first wdqs-(main|scholarly) endpoint contacted by the user). Current proposal is to use envoy for internal (i.e. emanating from blazegraph http client) federation queries, but that raises some questions, most pressingly: is there a mechanism whereby we can query etcd state so that hosts depooled from pybal will similarly be excluded from the envoy pool?
Note that the throttling question is orthogonal to the LVS vs Envoy question. As long as we are able to identify which requests are coming from the outside (internet) vs internal, we can put in place throttling exceptions.
It is not clear to me what solutions we have in place to route / load balance internal traffic. If LVS isn't the solution to load balance internal traffic, we need another solution. I don't think that Data-Platform-SRE is the right team to put in place a new load balancing solution.
Whatever solution we use for internal load balancing should provide at least:
Note that we should not have traffic loops, at least not in the sense that I understand loops. There are multiple blazegraph pools with different datasets. Each pool will be able to federate with another pool, but should never federate with itself. As an example:
Queries to wdqs-main, federated with wdqs-scholarly:
query.wikidata.org-> ATS -> LVS -> nginx -> blazegraph (main) -> LVS -> nginx -> blazegraph (scholarly)
After discussion with @Vgutierrez and @Joe, given that we have separate LVS pools for the different blazegraph pools, we don't create traffic loops and we should be good using LVS.
I'm closing this as invalid, feel free to re-open if I misunderstood something.