As an event platform engineer, I need to understand how I can restart a Flink application from the point in time that it failed
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Gehel | T317045 [Epic] Re-architect the Search Update Pipeline | |||
| Resolved | Gehel | T340548 [EPIC] Deployment of the Search Update Pipeline on Flink / k8s | |||
| Open | lbowmaker | T328561 [Event Platform] Flink Operations | |||
| Resolved | gmodena | T328563 [Flink Operations] How to handle restarting a Flink application |
gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/11
Draft: Enable s3-fs-presto plugin
I have a working setup on minikube that manages restarts and HA using the flink k8s operator, minio (for checkpointing) and the helm template.
As a next step, I'd like to scale up experiments to DSE. @Ottomata need to touch base and validate some assumptions of our current deployment.
Flink HA services require _to start the JobManager and TaskManager pods with a service account which has the permissions to create, edit, delete ConfigMaps._ (https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/kubernetes/).
Does our service account meet this requirements?
We'll also need storage for application checkpointing. This is required for HA services (e.g. restart strategy), and we won't able to fallback to the Job Manager heap. I reached out to SRE Data Persistence for info re onboarding to Swift.
permissions to create, edit, delete ConfigMaps
Yes, we got it!
FWIW, we MAYYYBYE will want to use Zookeeper for the HA state. This would make k8s cluster restarts easier on Service Ops. Modified ConfigMap state is not restored on k8s cluster restart, so if we need to persist it, we have to do so manually when they want to restart k8s.
@lbowmaker @Ottomata @dcausse I documented today's application restart discussion at https://www.mediawiki.org/wiki/Platform_Engineering_Team/Event_Platform_Value_Stream/Pyflink_Enrichment_Service_Deployment#Application_restarts.
otto merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/11
Enable s3-fs-presto plugin