1 Release published by 1 person

• v2.57.0 Beam 2.57.0 Release

  published Jun 26, 2024

31 Pull requests merged by 19 people

• Dont read from cache in sensitive workflows

  #31734 merged Jul 1, 2024

• Add a unit test for MakePipelineOptionsFileAndEnvVar

  #31732 merged Jul 1, 2024

• [Dataflow Streaming] Enabled Heartbeat by Default

  #31689 merged Jul 1, 2024

• [CsvIO] Move CsvIOParseResult

  #31722 merged Jun 29, 2024

• Remove excessive logging in test.

  #31715 merged Jun 28, 2024

• [CsvIO] Create CsvIOParseConfiguration

  #31714 merged Jun 28, 2024

• [#31403] Python wrapper to download, use, or build and run prism.

  #31583 merged Jun 28, 2024

• Create CsvIOParseResult

  #31706 merged Jun 28, 2024

• Properly close Storage API batch connections

  #31710 merged Jun 28, 2024

• Eliminate the use of testRuntimeMigration for sdks:java:io:common

  #31693 merged Jun 28, 2024

• Allow override beam version for PythonExternalTransform via pipeline option

  #31691 merged Jun 28, 2024

• Add spark mapstate

  #31669 merged Jun 28, 2024

• move heartbeat processor to where it is being used

  #31298 merged Jun 28, 2024

• Bump braces from 3.0.2 to 3.0.3 in /sdks/typescript

  #31664 merged Jun 27, 2024

• [#28187][Prism] Relax or fix issues in Prism to allow Python pipelines to execute.

  #31694 merged Jun 27, 2024

• Create CsvIOParseError data class

  #31700 merged Jun 27, 2024

• Bump github.com/tetratelabs/wazero from 1.7.0 to 1.7.3 in /sdks

  #31672 merged Jun 27, 2024

• Bump github.com/spf13/cobra from 1.8.0 to 1.8.1 in /sdks

  #31612 merged Jun 27, 2024

• Bump cloud.google.com/go/datastore from 1.17.0 to 1.17.1 in /sdks

  #31695 merged Jun 27, 2024

• Try mkdir for final destination before in WriteFiles

  #31690 merged Jun 26, 2024

• Blog and site updates for Beam 2.57.0 release

  #31667 merged Jun 26, 2024

• Bump github.com/go-sql-driver/mysql from 1.8.0 to 1.8.1 in /sdks

  #31688 merged Jun 26, 2024

• Add Histogram combiner

  #31379 merged Jun 26, 2024

• Add Storage API streaming max retries parameter for BigQueryOptions

  #31683 merged Jun 26, 2024

• Fixes a regression related to BQ read transform upgrade via the TransformService

  #31685 merged Jun 26, 2024

• Publish a blog post - deploy-python-pipeline-on-flink-runner

  #31655 merged Jun 26, 2024

• Fix nullable array issue 31674 in AvroGenericRecordToStorageApiProto

  #31675 merged Jun 25, 2024

• Expose GroupIntoBatches parameters for WriteFiles auto-sharding transform

  #31617 merged Jun 25, 2024

• Solace Read connector: RetryCallable mechanism

  #31539 merged Jun 25, 2024

• Bump cloud.google.com/go/bigtable from 1.22.0 to 1.25.0 in /sdks

  #31681 merged Jun 25, 2024

• Adjust JVM heap size for extremely large memory machine

  #31567 merged Jun 24, 2024

20 Pull requests opened by 10 people

• Remove expensive shuffle of read data in KafkaIO when using sdf and commit offsets

  #31682 opened Jun 25, 2024

• Basic yaml-defined provider.

  #31684 opened Jun 25, 2024

• Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.13.8 to 1.17.1 in /sdks

  #31687 opened Jun 26, 2024

• Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.13.8 to 1.17.2 in /sdks

  #31696 opened Jun 27, 2024

• Poll for backlog in background thread instead of inline

  #31697 opened Jun 27, 2024

• Add jobLabelsMap parameter to BigQueryOptions

  #31698 opened Jun 27, 2024

• Unit Testing in Beam Blog Post

  #31701 opened Jun 27, 2024

• Support custom JdbcReadWithPartitionsHelper for Jdbc.ReadWithPartitions

  #31702 opened Jun 27, 2024

• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.42.2 to 1.57.0 in /sdks

  #31707 opened Jun 28, 2024

• Bump cloud.google.com/go/storage from 1.41.0 to 1.42.0 in /sdks

  #31708 opened Jun 28, 2024

• Bump github.com/nats-io/nats-server/v2 from 2.10.12 to 2.10.17 in /sdks

  #31709 opened Jun 28, 2024

• Set SchemaCoder for key in WithKeys transform

  #31711 opened Jun 28, 2024

• Remove testRuntimeMigration configuration for test-utils dependencies

  #31713 opened Jun 28, 2024

• [CsvIO] Scaffold CsvIOParseHelpers

  #31720 opened Jun 28, 2024

• Add options to control number of Storage API connections when using multiplexing

  #31721 opened Jun 28, 2024

• Pass-through IcebergIO catalog properties

  #31726 opened Jun 29, 2024

• implementation equals and hashCode of FlinkOrderedListState

  #31727 opened Jun 30, 2024

• Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.13.8 to 1.17.3 in /sdks

  #31728 opened Jul 1, 2024

• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.42.2 to 1.57.1 in /sdks

  #31729 opened Jul 1, 2024

• Support custom JdbcReadWithPartitionsHelper

  #31733 opened Jul 1, 2024

14 Issues closed by 8 people

• [Bug]: The Documentation of `@SchemaFieldNumber` has the wrong type

  #30273 closed Jul 1, 2024

• [CsvIO]: Create a CsvIOParseConfiguration Class

  #31704 closed Jun 28, 2024

• [Task][prism] Have python wrapper check and download released prism binary if available.

  #31403 closed Jun 28, 2024

• [CsvIO]: Create CsvIOParseResult

  #31705 closed Jun 28, 2024

• [Task]: Support custom Python SDK for sdks/java/extensions/python PythonService

  #31680 closed Jun 28, 2024

• [Feature Request]: Use BigQuery emulator

  #28841 closed Jun 28, 2024

• [Feature Request]: Add MapState in SparkRunner

  #31668 closed Jun 28, 2024

• [CsvIO] Create CsvIOParseError data class

  #31699 closed Jun 27, 2024

• [Feature Request]: Making number of retries configurable in BigQuery Storage Write connector

  #25382 closed Jun 26, 2024

• [Bug]: Issue with AvroGenericRecordToStorageApiProto.java handling nullable arrays

  #31674 closed Jun 25, 2024

• [Feature Request]: Expose GroupIntoBatches Parameters for WriteFiles Autosharding transform

  #30131 closed Jun 25, 2024

• Performance Regression or Improvement: gbk_python_batch_load_test_2gb_of_100KB_records:runtime

  #31660 closed Jun 25, 2024

• [Bug]: Asynchronous callback FutureT is broken from beam 2.54.0

  #31445 closed Jun 25, 2024

• Load dataflow jobs using java 11 in Java 11 Dataflow tests

  #20021 closed Jun 24, 2024

15 Issues opened by 9 people

• [Feature Request]: Increase the test coverage of pipieline_options.go

  #31731 opened Jul 1, 2024

• The PostCommit Java ValidatesRunner Samza job is flaky

  #31725 opened Jun 29, 2024

• [Feature Request]: Implement equals, hashCode in FlinkRunner's OrderedListState.

  #31724 opened Jun 29, 2024

• [Feature Request]: Add OrderedListState test in StateInternalsTest

  #31723 opened Jun 29, 2024

• [CsvIO]: Create CsvIOParseHelpers::parseCell(String, Schema.Field)

  #31719 opened Jun 28, 2024

• [CsvIO]: Create CsvIOParseHelpers::mapFieldPositions(CSVFormat, Schema)

  #31718 opened Jun 28, 2024

• [Feature Request]:[Go SDK] Improve "bad return type" messaging

  #31717 opened Jun 28, 2024

• [CsvIO]: Create CsvIOParseHelpers::validate(CSVFormat, Schema)

  #31716 opened Jun 28, 2024

• [CsvIO]: Create CsvIOParseHelpers::validate(CSVFormat)

  #31712 opened Jun 28, 2024

• [CsvIO]: Create CsvIOParseHelpers utility class

  #31703 opened Jun 27, 2024

• [Feature Request]: Develop a way to evolve the construction schema used by transforms safely

  #31686 opened Jun 25, 2024

• [Bug]: PubsubMessageWithTopicCoder.of() returns PubsubMessageWithAttributesAndMessageIdCoder

  #31679 opened Jun 24, 2024

• [Task]: Deprecate Java8

  #31678 opened Jun 24, 2024

• [Task]: Migrate to build with Java11

  #31677 opened Jun 24, 2024

• [Bug]: Dependency resolution is happening when installing latest version of apache-beam[gcp]

  #31676 opened Jun 24, 2024

36 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Integrate direct path/fan out logic

  #31504 commented on Jul 1, 2024 • 56 new comments

• Solace Read connector: adding Basic Authentication support

  #31541 commented on Jun 28, 2024 • 17 new comments

• Enable MapState and SetState for dataflow streaming engine pipelines with legacy runner by building on top of MultimapState.

  #31453 commented on Jul 1, 2024 • 6 new comments

• PubsubMessageWithTopicCoder.of() is returning wrong coder

  #31619 commented on Jun 27, 2024 • 4 new comments

• [Bug]: BigQuery Write step timeout since version 2.54.0

  #31095 commented on Jun 28, 2024 • 4 new comments

• [Bug]: Using string in PartitionColumn throws error, tries to convert it to string

  #31419 commented on Jul 1, 2024 • 4 new comments

• [prism] Programmatic Cancel, and Drain

  #29669 commented on Jun 29, 2024 • 3 new comments

• Add support for schema providers for generic classes

  #31648 commented on Jun 24, 2024 • 2 new comments

• [Task][prism] Have java wrapper check and download released prism binary if available.

  #31402 commented on Jun 26, 2024 • 2 new comments

• Bump ws from 6.2.2 to 6.2.3 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel

  #31625 commented on Jun 27, 2024 • 2 new comments

• [Bug]: WriteToFiles in python leave few records in temp directory when writing to large number (100+) of files

  #29515 commented on Jun 26, 2024 • 2 new comments

• Support writing to Pubsub with ordering key; Add PubsubMessage SchemaCoder

  #31608 commented on Jun 29, 2024 • 2 new comments

• Avoid length-prefix-bytes substitutions for Flink boundaries.

  #31579 commented on Jun 24, 2024 • 2 new comments

• [Feature Request]: Allow setting BigQuery endpoint, for example to use bigquery emulator

  #28149 commented on Jun 28, 2024 • 2 new comments

• [Bug]: Python TypeError when converting Avro `logicalType` `timestamp-millis` to Beam Schema

  #31656 commented on Jul 1, 2024 • 2 new comments

• Pass original message down through conversion for storage write api

  #31106 commented on Jun 29, 2024 • 1 new comment

• Implement a `Top` partitioner

  #29106 commented on Jun 29, 2024 • 1 new comment

• [Python] Managed Transforms API

  #31495 commented on Jun 27, 2024 • 1 new comment

• Add SessionPoolOptions to SpannerConfig

  #31663 commented on Jun 29, 2024 • 1 new comment

• Performance Regression or Improvement: sideinpts_python_batch_10gb_1kb_10workers_1000window_first_iterable:runtime

  #31661 commented on Jun 25, 2024 • 1 new comment

• Replace StorageV1 client with GCS client - Draft

  #28733 commented on Jul 1, 2024 • 1 new comment

• [Feature Request]: Support Reading from Solace message broker

  #31440 commented on Jul 1, 2024 • 1 new comment

• [Feature Request]: allow users to provide their own JdbcReadWithPartitionsHelper

  #27120 commented on Jul 1, 2024 • 1 new comment

• [Bug]: ignoreUnknownValues not working when using CreateDisposition.CREATE_IF_NEEDED

  #27892 commented on Jun 29, 2024 • 1 new comment

• [Bug]: Coder cannot be resolved after `WithKeys.of().withKeyType()` for @DefaultSchema(JavaBeanSchema.class) types

  #29577 commented on Jun 28, 2024 • 1 new comment

• [Task]: Single dockerhub repository for Flink job server container

  #31631 commented on Jun 25, 2024 • 1 new comment

• [Feature Request]: UDF supports lambda & RecordType parameter

  #27465 commented on Jun 26, 2024 • 1 new comment

• Use a provider to create JMS ConnectionFactory and avoid Serialization issues

  #18110 commented on Jun 26, 2024 • 1 new comment

• Support withFormatRecordOnFailureFunction() for BigQuery STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods

  #31659 commented on Jun 26, 2024 • 0 new comments

• Added insertion and enrichment pipeline

  #31657 commented on Jun 27, 2024 • 0 new comments

• Bump github.com/nats-io/nats-server/v2 from 2.10.12 to 2.10.16 in /sdks

  #31611 commented on Jun 27, 2024 • 0 new comments

• Document Beam Python on Databricks

  #20382 commented on Jun 27, 2024 • 0 new comments

• [Bug]: Pickling error in save_main_session with STORAGE_WRITE_API in Dataflow Python pipeline

  #31587 commented on Jun 27, 2024 • 0 new comments

• Solace Read connector: adding implementations of SempClient and SempClientFactory

  #31542 commented on Jun 24, 2024 • 0 new comments

• [Task][prism]: Be able to execute non-Go SDKs on Prism.

  #28187 commented on Jun 28, 2024 • 0 new comments

• added custom watermark for kinesis reader

  #28763 commented on Jun 24, 2024 • 0 new comments