While discussing T330686, we figured that our delta algorithm assumes that the previous run was successful.
In this task we want to change this so that the delta is made againt whichever is the latest snapshot available in Hive. This will make the job robust against intermittent failures.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Make previous_weekly be the last successful DAG run of image_suggestions | repos/data-engineering/airflow-dags!287 | xcollazo | redo-robust-deltas | main | |
| Hotfix: Make previous_weekly a varprop on image_suggestions DAG. | repos/data-engineering/airflow-dags!262 | xcollazo | hotfix-image-suggestions | main | |
| Calculate previous snapshots for deltas automatically. | repos/structured-data/image-suggestions!11 | xcollazo | T330688-robust-deltas | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T311814 [EPIC] Section-level image suggestions data pipeline | |||
| Resolved | xcollazo | T330688 Make sure our delta algorithm doesn't depend on successful past runs |
xcollazo updated https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/11
Calculate previous snapshots for deltas automatically.
xcollazo merged https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/11
Calculate previous snapshots for deltas automatically.
One issue we have is that the pipeline ran for snapshot=2023-02-20 while we were working on this task.
This means that now the data on image_suggestions_search_index_delta is corrupted.
After discussions with @Cparle and @mfossati, we decided to do the following to fix this:
image_suggestions_search_index_full image_suggestions_search_index_delta image_suggestions_lead_image_data image_suggestions_wikidata_data
Mentioned in SAL (#wikimedia-analytics) [2023-03-03T16:48:12Z] <xcollazo> Deleted snapshot=2023-02-20 for tables image_suggestions_search_index_full, image_suggestions_search_index_delta, image_suggestions_lead_image_data and image_suggestions_wikidata_data from the analytics_platform_eng schema. This data will be regenerated. See https://phabricator.wikimedia.org/T330688.
xcollazo updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/262
Hotfix: Make previous_weekly a varprop on image_suggestions DAG.
cparle merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/262
Hotfix: Make previous_weekly a varprop on image_suggestions DAG.
xcollazo updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/287
Make previous_weekly be the last successful DAG run of image_suggestions
xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/287
Make previous_weekly be the last successful DAG run of image_suggestions
Deployment is now blocked by the Airflow 2.5.1 upgrade (See T332031). We could just branch out for this deployment, but since the upgrade is slated for Thu Mar 16, it doesn't make much sense to pay the branching penalty for just one deployment.