[go: nahoru, domu]
Page MenuHomePhabricator
Create Task
Maniphest T337013

[Epic] Splitting the graph in WDQS
Open, HighPublic
Actions

Description

In order to stabilize the Wikidata Query Service we are looking into splitting the graph inside Blazegraph into 2 (or potentially more) subgraphs. This ticket is for tracking the investigation into what a sensible split would be, what the consequences are and then making it happen.

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT335067 Epic: Wikidata Query Service stabilization
 OpenNoneT337013 [Epic] Splitting the graph in WDQS
 ResolvedGehelT337018 Refine the definitions of the potential splits
 Resolved• ManuelT337020 [Analytics] Understand the size and connectedness of Wikimedia-internal concepts in the Wikidata graph
 ResolvedAndrewTavis_WMDET337021 [Analytics] Find out size of term subgraph
 DeclinedNoneT337407 1st experiment in splitting the WDQS graph
 DeclinedNoneT337406 Setting up a test server exposing an experimental split of the WDQS graph
 Resolveddr0ptp4ktT347989 Adapt rdf-spark-tools to split the wikidata graph based on a set of rules
 ResolveddcausseT349519 Determine if IGUANA and TFT would fit our query analysis needs
 ResolvedLydia_PintscherT349829 List and document all known public SPARQL endpoints serving wikidata
 ResolvedGehelT350464 Expose SPARQL endpoints with full wikidata data set and with split graph to enable experimentation on federation with a split graph
 ResolveddcausseT349069 Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers
 ResolvedbkingT347504 WDQS graph split: load data from dumps into new hosts
 ResolvedbkingT349011 Improve data-reload cookbook based on graph split needs
 ResolvedbkingT347505 Prepare new WDQS hosts for graph splitting
 ResolvedRKemperT351650 Expose 3 new dedicated WDQS endpoints
 ResolvedRKemperT350465 Load Wikidata split graphs into test servers
 Resolveddr0ptp4ktT350106 Implement a spark job that converts a RDF triples table into a RDF file format
 ResolvedRKemperT354658 Create 3 microsites for wdqs full graph, main graph, & scholarly articles
 ResolvedDzahnT355273 ProbeDown - *_experimental_wikidata_org
 ResolvedRKemperT354661 Generate TLS certs for new WDQS endpoints
 ResolvedRKemperT354662 Create DNS records for 3 new WDQS endpoints
 OpenNoneT354043 Decide the name, domain and logo of WDQS for scholarly articles
 ResolvedRKemperT355593 Re-generate webserver-misc-apps.discovery.wmnet cergen certificate
 ResolvedGehelT355888 Enable cross federation between experimental WDQS endpoints
 ResolvedbkingT352878 Troubleshoot recurring systemd unit failures and availability issues for wdqs1022-24
 OpenNoneT352538 [EPIC] Evaluate the impact of the graph split
 ResolvedAndrewTavis_WMDET349512 [Analytics] Collect multiple sets of SPARQL queries
 DuplicateAndrewTavis_WMDET350157 [Analytics] Extract a representative sample of SPARQL queries from the query logs
 OpenNoneT337015 review sample of queries to get a better sense of impact of various graph split options
 ResolvedGehelT349911 Explore the feasibility of using SPARQL federation for scholia queries
 ResolveddcausseT351819 Create a tool that records and compares a set of sparql query results
 Resolveddr0ptp4ktT355037 Compare the performance of sparql queries between the full graph and the subgraphs
 ResolveddcausseT355040 Compare the results of sparql queries between the fullgraph and the subgraphs
 InvalidSannitaT356773 [tracking] Community feedback for the WDQS Split the Graph project
 Resolved• Lucas_Werkmeister_WMDET355298 Investigate the impact of the WDQS graph split on constraints checks
 ResolveddcausseT357966 Document limitations of blazegraph federation
 ResolveddcausseT357980 Compile a set of queries rewritten with federation across the two graph splits
 ResolvedpfischerT361935 Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs
 ResolvedEBernhardsonT361950 Ensure that WDQS query throttling does not interfere with federation
 ResolveddcausseT362060 Generalize ScholarlyArticleSplitter
 OpenNoneT364363 [Epic] Productionize federated wdqs graph-split endpoints
 OpenRKemperT364364 Provision DNS and certificates for wdqs graph split domains
ResolvedStevemuneneT364366 Implement puppet logic for wdqs graph split
 ResolvedStevemuneneT364367 Create dedicated UIs for wdqs graph split endpoints
 ResolvedStevemuneneT364368 Create separate pybal pools for wdqs graph split (main vs scholarly)
 ResolvedStevemuneneT372919 Bring wqds2024 back into service
 OpenRKemperT373145 Create new service catalog entries for wdqs-main and wdqs-scholarly
 OpenRKemperT364077 Adapt the wdqs data-transfer cookbook to operate with federated subgraphs
 InvalidGehelT368972 Use Envoy instead of LVS to route internal federation traffic for WDQS
 OpenRKemperT370754 Import WDQS subgraphs to production nodes
 OpenNoneT371346 Add a banner to query.wikidata.org to inform users about the scholarly articles split
 ResolvedStevemuneneT371833 Tear down wdqs graph split experimental endpoints
 OpenNoneT372816 Add federated queries in the test queries used to validate deployment
 OpenNoneT372818 Verify and correct WDQS Grafana dashboard with graph split
 OpenNoneT373391 Create wdqs-main and wdqs-scholarly specific test queries
 OpenbkingT373791 Transfer a sane journal (subgraph:main) to wdqs2021 from wdqs2022
 OpenNoneT373812 Internal federation sometimes fail with HttpConnectionOverHTTP
 OpenRKemperT367409 Measure the time to recovery of WDQS after the graph split
 ResolvedGehelT367510 Request permission to create 4 kafka topics in kafka-main (WDQS graph split)
 OpenAudreyPenven_WMDET369079 Update `UniqueValueChecker` to query a list of endpoints
 ResolvedpfischerT369729 RDF Updater Consumer: Allow "empty" patches

Event Timeline

Lydia_Pintscher created this task.May 19 2023, 9:35 AM
Lydia_Pintscher mentioned this in T335067: Epic: Wikidata Query Service stabilization.May 19 2023, 10:04 AM
Gehel triaged this task as High priority.May 22 2023, 12:58 PM
Gehel moved this task from Incoming to Scaling on the Wikidata-Query-Service board.
Gehel moved this task from Scaling to Epics on the Wikidata-Query-Service board.
karapayneWMDE subscribed.May 24 2023, 2:43 PM
Gehel removed a subtask: T337015: review sample of queries to get a better sense of impact of various graph split options.May 24 2023, 3:25 PM
Gehel removed a subtask: T337406: Setting up a test server exposing an experimental split of the WDQS graph.
elal subscribed.May 24 2023, 3:28 PM
• Manuel added a project: Wikidata Analytics.May 30 2023, 6:38 PM
• Manuel moved this task from Incoming to Waiting Tasks on the Wikidata Analytics board.
• Manuel mentioned this in T337799: [EPIC] Analytics support around splitting the WDQS graph [up to milestone 3].May 30 2023, 8:54 PM
• Manuel removed a project: Wikidata Analytics.Jul 7 2023, 9:12 PM
tfmorris subscribed.Jul 7 2023, 9:29 PM
Gehel closed subtask T337407: 1st experiment in splitting the WDQS graph as Declined.Jul 17 2023, 3:37 PM
• Manuel closed subtask T337020: [Analytics] Understand the size and connectedness of Wikimedia-internal concepts in the Wikidata graph as Resolved.Jul 18 2023, 1:16 PM
• Manuel closed subtask T337021: [Analytics] Find out size of term subgraph as Resolved.Jul 20 2023, 9:32 AM
bking subscribed.Sep 15 2023, 1:37 PM
bking added a subscriber: RKemper.EditedSep 15 2023, 2:02 PM

When T345475 is done, we should have 3 new WDQS hosts in CODFW that could be used for the graph splitting experiment. @RKemper let us know if you have any objections to this plan.

We'll use eqiad hosts instead, see T347505

bking added a subtask: T345475: Service implementation for wdqs202[3-5].codfw.wmnet.Sep 15 2023, 2:02 PM
dr0ptp4kt subscribed.Sep 22 2023, 4:07 PM
bking mentioned this in T347504: WDQS graph split: load data from dumps into new hosts.Sep 27 2023, 3:27 PM
bking added a subtask: T347504: WDQS graph split: load data from dumps into new hosts.
bking removed a subtask: T345475: Service implementation for wdqs202[3-5].codfw.wmnet.Sep 27 2023, 3:36 PM
bking added a subtask: T347505: Prepare new WDQS hosts for graph splitting.Sep 27 2023, 3:39 PM
Gehel added a project: Discovery-Search (Current work).Oct 17 2023, 9:18 AM
Gehel moved this task from Incoming to Epics on the Discovery-Search (Current work) board.
bking closed subtask T347505: Prepare new WDQS hosts for graph splitting as Resolved.Oct 20 2023, 1:58 PM
dcausse added a subtask: T349512: [Analytics] Collect multiple sets of SPARQL queries.Oct 23 2023, 1:41 PM
dcausse added a subtask: T349519: Determine if IGUANA and TFT would fit our query analysis needs.Oct 23 2023, 2:51 PM
dcausse added a subtask: T349829: List and document all known public SPARQL endpoints serving wikidata.Oct 26 2023, 2:59 PM
dcausse added a subtask: T349911: Explore the feasibility of using SPARQL federation for scholia queries.Oct 27 2023, 2:19 PM
Gehel closed subtask T337018: Refine the definitions of the potential splits as Resolved.Nov 3 2023, 9:55 AM
Gehel removed a subtask: T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers.Nov 3 2023, 10:21 AM
Gehel removed a subtask: T347504: WDQS graph split: load data from dumps into new hosts.
Gehel removed a subtask: T347505: Prepare new WDQS hosts for graph splitting.
dr0ptp4kt closed subtask T347989: Adapt rdf-spark-tools to split the wikidata graph based on a set of rules as Resolved.Nov 20 2023, 5:45 PM
Gehel removed a subtask: T351650: Expose 3 new dedicated WDQS endpoints.Nov 20 2023, 7:41 PM
dcausse added a subtask: T351819: Create a tool that records and compares a set of sparql query results.Nov 22 2023, 2:01 PM
Gehel closed subtask T349519: Determine if IGUANA and TFT would fit our query analysis needs as Resolved.Dec 1 2023, 9:20 AM
Gehel removed a subtask: T349512: [Analytics] Collect multiple sets of SPARQL queries.Dec 1 2023, 2:49 PM
Gehel removed a subtask: T349911: Explore the feasibility of using SPARQL federation for scholia queries.
Gehel removed a subtask: T351819: Create a tool that records and compares a set of sparql query results.
SEgt-WMF subscribed.Jan 8 2024, 6:58 PM
Gehel closed subtask T350464: Expose SPARQL endpoints with full wikidata data set and with split graph to enable experimentation on federation with a split graph as Resolved.Jan 26 2024, 8:41 AM
dcausse added a subtask: T357966: Document limitations of blazegraph federation.Feb 20 2024, 11:03 AM
dcausse added a subtask: T357980: Compile a set of queries rewritten with federation across the two graph splits.Feb 20 2024, 2:00 PM
Lydia_Pintscher closed subtask T349829: List and document all known public SPARQL endpoints serving wikidata as Resolved.Mar 4 2024, 7:36 PM
Stashbot added a comment.Mar 5 2024, 10:37 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-05T22:37:10Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 60 days, 0:00:00 on wdqs[1022-1024].eqiad.wmnet with reason: T337013

Stashbot added a comment.Mar 5 2024, 10:37 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-05T22:37:14Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 60 days, 0:00:00 on wdqs[1022-1024].eqiad.wmnet with reason: T337013

Stashbot added a comment.Mar 7 2024, 6:22 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-07T18:22:18Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 60 days, 0:00:00 on wdqs[1022-1025].eqiad.wmnet with reason: T337013

Stashbot added a comment.Mar 7 2024, 6:22 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-07T18:22:38Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 60 days, 0:00:00 on wdqs[1022-1025].eqiad.wmnet with reason: T337013

Kanashimi subscribed.Mar 7 2024, 10:37 PM
Gehel closed subtask T357980: Compile a set of queries rewritten with federation across the two graph splits as Resolved.Mar 8 2024, 10:19 AM
dcausse added a subtask: T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs.Apr 5 2024, 12:53 PM
Gehel closed subtask T357966: Document limitations of blazegraph federation as Resolved.Apr 5 2024, 1:01 PM
dcausse added a subtask: T361950: Ensure that WDQS query throttling does not interfere with federation.Apr 5 2024, 3:29 PM
Daniel_Mietchen subscribed.Apr 6 2024, 1:04 AM
dcausse added a subtask: T362060: Generalize ScholarlyArticleSplitter.Apr 8 2024, 11:48 AM
Gehel added a subtask: T364077: Adapt the wdqs data-transfer cookbook to operate with federated subgraphs.May 10 2024, 8:27 AM
Gehel closed subtask T362060: Generalize ScholarlyArticleSplitter as Resolved.May 24 2024, 7:28 AM
Gehel closed subtask T367510: Request permission to create 4 kafka topics in kafka-main (WDQS graph split) as Resolved.Jun 21 2024, 1:17 PM
Gehel reopened subtask T367510: Request permission to create 4 kafka topics in kafka-main (WDQS graph split) as Open.Jun 21 2024, 1:26 PM
Nosferattus subscribed.EditedJun 25 2024, 5:56 PM

Will this also solve T261764? or is the graph used by the Query Service different than the one used by the API?

Lydia_Pintscher added a subtask: T369079: Update `UniqueValueChecker` to query a list of endpoints.Jul 3 2024, 6:52 AM
Lydia_Pintscher added a comment.Jul 3 2024, 7:43 AM
In T337013#9923203, @Nosferattus wrote:

Will this also solve T261764? or is the graph used by the Query Service different than the one used by the API?

I don't believe this is using the Query Service. This means it would not be affected.

diegodlh subscribed.Jul 8 2024, 12:34 AM
pfischer changed the status of subtask T369729: RDF Updater Consumer: Allow "empty" patches from Open to In Progress.Jul 11 2024, 8:28 AM
• Lucas_Werkmeister_WMDE closed subtask T355298: Investigate the impact of the WDQS graph split on constraints checks as Resolved.Jul 16 2024, 8:28 AM
Gehel closed subtask T367510: Request permission to create 4 kafka topics in kafka-main (WDQS graph split) as Resolved.Jul 19 2024, 12:42 PM
Gehel closed subtask T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs as Resolved.
Gehel closed subtask T369729: RDF Updater Consumer: Allow "empty" patches as Resolved.
Gehel closed subtask T361950: Ensure that WDQS query throttling does not interfere with federation as Resolved.Jul 19 2024, 12:46 PM
Stashbot added a comment.Fri, Aug 23, 3:52 PM

Mentioned in SAL (#wikimedia-operations) [2024-08-23T15:52:57Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 17:00:00 on wdqs[1023-1024].eqiad.wmnet with reason: noisy alerts related to graph split T337013

Stashbot added a comment.Fri, Aug 23, 3:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-08-23T15:53:13Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 17:00:00 on wdqs[1023-1024].eqiad.wmnet with reason: noisy alerts related to graph split T337013

Nosferattus unsubscribed.Fri, Aug 23, 4:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits