Apache DataFusion SQL Query Engine

Apache Beam is a unified programming model for Batch and Streaming data processing.

Publicly available real-time data sets on Kafka, Redpanda, RabbitMQ & Apache Pulsar

Scalable datastore for metrics, events, and real-time analytics

the portable Python dataframe library

Materials and IPython notebooks for "Python for Data Analysis" by Wes McKinney, published by O'Reilly Media

SQL stream processing, analytics, and management. We decouple storage and compute to offer instant failover, dynamic scaling, speedy bootstrapping, and efficient joins.

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

Metriport is an open-source universal API for healthcare data.

An Awesome List of Open-Source Data Engineering Projects

TFX is an end-to-end platform for deploying production ML pipelines

Fluentd: Unified Logging Layer (project under CNCF)

🚀 A curated list of awesome articles, videos, and other resources to learn and practice software architecture, patterns, and principles.

Lean and mean distributed stream processing system written in rust and web assembly.

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

A production ready example Django app that's using Docker and Docker Compose.

Open Source Event Monitoring

Testcontainers that start Kubernetes in Docker.

Flower: A Friendly Federated Learning Framework

Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required

This project contains examples which demonstrate how to deploy analytic models to mission-critical, scalable production environments leveraging Apache Kafka and its Streams API. Models are built wi…

A dedicated scratchpad for developers

Understanding Deep Learning - Simon J.D. Prince

EventStoreDB, the event-native database. Designed for Event Sourcing, Event-Driven, and Microservices architectures

Data-Centric Pipelines and Data Versioning

High-Performance server for NATS.io, the cloud and edge native messaging system.

Python library for Wit.ai

Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth.