• [CHANGE] Change default tempo port from 3100 to 3200 #770 (@MurzNN)
• [ENHANCEMENT] Added hedged request metric tempodb_backend_hedged_roundtrips_total and a new storage agnostic tempodb_backend_request_duration_seconds metric that supersedes the soon-to-be deprecated storage specific metrics (tempodb_azure_request_duration_seconds, tempodb_s3_request_duration_seconds and tempodb_gcs_request_duration_seconds). #790 (@JosephWoodward)
• [CHANGE] Jsonnet: use dedicated configmaps for distributors and ingesters #775 (@kvrhdn)
• [FEATURE] Added the ability to hedge requests with all backends #750 (@joe-elliott)
• [ENHANCEMENT] Performance: improve compaction speed with concurrent reads and writes #754 (@mdisibio)
• [ENHANCEMENT] Improve readability of cpu and memory metrics on operational dashboard #764 (@bboreham)
• [ENHANCEMENT] Add azure_request_duration_seconds metric. #767 (@JosephWoodward)
• [ENHANCEMENT] Add s3_request_duration_seconds metric. #776 (@JosephWoodward)
• [ENHANCEMENT] Add tempo_ingester_flush_size_bytes metric. #777 (@bboreham)
• [ENHANCEMENT] Microservices jsonnet: resource requests and limits can be set in $._config. #793 (@kvrhdn)
• [ENHANCEMENT] Add -config.expand-env cli flag to support environment variables expansion in config file. #796 (@Ashmita152)
• [ENHANCEMENT] Emit traces for ingester flush operations. #812 (@bboreham)
• [ENHANCEMENT] Add retry middleware in query-frontend. #814 (@kvrhdn)

• [BUGFIX] Guard against negative dataLength #763 (@joe-elliott)

• [CHANGE] Mark -auth.enabled as deprecated. New flag is -multitenancy.enabled and is set to false by default. This is a breaking change if you were relying on auth/multitenancy being enabled by default. #646
• [ENHANCEMENT] Performance: Improve Ingester Record Insertion. #681
• [ENHANCEMENT] Improve WAL Replay by not rebuilding the WAL. #668
• [ENHANCEMENT] Add config option to disable write extension to the ingesters. #677
• [ENHANCEMENT] Preallocate byte slices on ingester request unmarshal. #679
• [ENHANCEMENT] Reduce marshalling in the ingesters to improve performance. #694 This change requires a specific rollout process to prevent dropped spans. First, rollout everything except distributors. After all ingesters have updated you can then rollout distributors to the latest version. This is due to changes in the communication between ingesters <-> distributors.
• [ENHANCEMENT] Allow setting the bloom filter shard size with support dynamic shard count.#644
• [ENHANCEMENT] GCS SDK update v1.12.0 => v.15.0, ReadAllWithEstimate used in GCS/S3 backends. #693
• [ENHANCEMENT] Add a new endpoint /api/echo to test the query frontend is reachable. #714
• [BUGFIX] Fix Query Frontend grpc settings to avoid noisy error log. #690
• [BUGFIX] Zipkin Support - CombineTraces. #688
• [BUGFIX] Zipkin support - Dedupe span IDs based on span.Kind (client/server) in Query Frontend. #687
• [BUGFIX] Azure Backend - Fix an issue with the append method on the Azure backend. #736

License Change v0.7.0 and future versions are licensed under AGPLv3 #660

• [CHANGE] Add json struct tags to overrides' Limits struct in addition to yaml tags. #656
• [CHANGE] Update to Go 1.16, latest OpenTelemetry proto definition and collector #546
• [CHANGE] max_spans_per_trace limit override has been removed in favour of max_bytes_per_trace. This is a breaking change to the overrides config section. #612
• [CHANGE] Add new flag -ingester.lifecycler.ID to manually override the ingester ID with which to register in the ring. #625
• [CHANGE] ingestion_rate_limit limit override has been removed in favour of ingestion_rate_limit_bytes. ingestion_burst_size limit override has been removed in favour of ingestion_burst_size_bytes. This is a breaking change to the overrides config section. #630
• [FEATURE] Add page based access to the index file. #557
• [FEATURE] (Experimental) WAL Compression/checksums. #638
• [ENHANCEMENT] Add a Shutdown handler to flush data to backend, at "/shutdown". #526
• [ENHANCEMENT] Queriers now query all (healthy) ingesters for a trace to mitigate 404s on ingester rollouts/scaleups. This is a breaking change and will likely result in query errors on rollout as the query signature b/n QueryFrontend & Querier has changed. #557
• [ENHANCEMENT] Add list compaction-summary command to tempo-cli #588
• [ENHANCEMENT] Add list and view index commands to tempo-cli #611
• [ENHANCEMENT] Add a configurable prefix for HTTP endpoints. #631
• [ENHANCEMENT] Add kafka receiver. #613
• [ENHANCEMENT] Upgrade OTel collector to v0.21.0. #613
• [ENHANCEMENT] Add support for Cortex Background Cache. #640
• [BUGFIX] Fixes permissions errors on startup in GCS. #554
• [BUGFIX] Fixes error where Dell ECS cannot list objects. #561
• [BUGFIX] Fixes listing blocks in S3 when the list is truncated. #567
• [BUGFIX] Fixes where ingester may leave file open #570
• [BUGFIX] Fixes a bug where some blocks were not searched due to query sharding and randomness in blocklist poll. #583
• [BUGFIX] Fixes issue where wal was deleted before successful flush and adds exponential backoff for flush errors #593
• [BUGFIX] Fixes issue where Tempo would not parse odd length trace ids #605
• [BUGFIX] Sort traces on flush to reduce unexpected recombination work by compactors #606
• [BUGFIX] Ingester fully persists blocks locally to reduce amount of work done after restart #628

• [CHANGE] Fixed ingester latency spikes on read #461
• [CHANGE] Ingester cut blocks based on size instead of trace count. Replace ingester traces_per_block setting with max_block_bytes. This is a breaking change. #474
• [CHANGE] Refactor cache section in tempodb. This is a breaking change b/c the cache config section has changed. #485
• [CHANGE] New compactor setting for max block size data instead of traces. #520
• [CHANGE] Change default ingester_client compression from gzip to snappy. #522
• [CHANGE/BUGFIX] Rename tempodb_compaction_objects_written and tempodb_compaction_bytes_written metrics to tempodb_compaction_objects_written_total and tempodb_compaction_bytes_written_total. #524
• [CHANGE] Replace tempo-cli list block --check-dupes option with --scan and collect additional stats #534
• [FEATURE] Added block compression. This is a breaking change b/c some configuration fields moved. #504
• [CHANGE] Drop Vulture Loki dependency. This is a breaking change. #509
• [ENHANCEMENT] Serve config at the "/config" endpoint. #446
• [ENHANCEMENT] Switch blocklist polling and retention to different concurrency mechanism, add configuration options. #475
• [ENHANCEMENT] Add S3 options region and forcepathstyle #431
• [ENHANCEMENT] Add exhaustive search to combine traces from all blocks in the backend. #489
• [ENHANCEMENT] Add per-tenant block retention #77
• [ENHANCEMENT] Change index-downsample to index-downsample-bytes. This is a breaking change #519
• [BUGFIX] Upgrade cortex dependency to v1.7.0-rc.0+ to address issue with forgetting ring membership #442 #512
• [BUGFIX] No longer raise the tempodb_blocklist_poll_errors_total metric if a block doesn't have meta or compacted meta. #481]
• [BUGFIX] Replay wal completely before ingesting new spans. #525

• [CHANGE] Redo tempo-cli with basic command structure and improvements #385
• [CHANGE] Add content negotiation support and sharding parameters to Querier #375
• [CHANGE] Remove S3 automatic bucket creation #404
• [CHANGE] Compactors should round robin tenants instead of choosing randomly #420
• [CHANGE] Switch distributor->ingester communication to more efficient PushBytes method. This is a breaking change when running in microservices mode with separate distributors and ingesters. To prevent errors ingesters must be fully upgraded first, then distributors.
• [CHANGE] Removed disk_cache. This is a breaking change b/c there is no disk cache. Please use redis or memcached. #441
• [CHANGE] Rename IngestionMaxBatchSize to IngestionBurstSize. This is a breaking change. #445
• [ENHANCEMENT] Add docker-compose example for GCS along with new backend options #397
• [ENHANCEMENT] tempo-cli list blocks usability improvements #403
• [ENHANCEMENT] Reduce active traces locking time. #449
• [ENHANCEMENT] Added tempo_distributor_bytes_received_total as a per tenant counter of uncompressed bytes received. #453
• [BUGFIX] Compactor without GCS permissions fail silently #379
• [BUGFIX] Prevent race conditions between querier polling and ingesters clearing complete blocks #421
• [BUGFIX] Exclude blocks in last active window from compaction #411
• [BUGFIX] Mixin: Ignore metrics and query-frontend route when checking for TempoRequestLatency alert. #440
• [FEATURE] Add support for Azure Blob Storage backend #340
• [FEATURE] Add Query Frontend module to allow scaling the query path #400

• [CHANGE] From path.Join to filepath.Join #338
• [CHANGE] Upgrade Cortex from v1.3.0 to v.1.4.0 #341
• [CHANGE] Compact more than 2 blocks at a time #348
• [CHANGE] Remove tempodb_compaction_duration_seconds metric. #360
• [ENHANCEMENT] Add tempodb_compaction_objects_combined metric. #339
• [ENHANCEMENT] Added OpenMetrics exemplar support. #359
• [ENHANCEMENT] Add tempodb_compaction_objects_written metric. #360
• [ENHANCEMENT] Add tempodb_compaction_bytes_written metric. #360
• [ENHANCEMENT] Add tempodb_compaction_blocks_total metric. #360
• [ENHANCEMENT] Add support for S3 V2 signatures. #352
• [ENHANCEMENT] Add support for Redis caching. #354
• [BUGFIX] Frequent errors logged by compactor regarding meta not found #327
• [BUGFIX] Fix distributors panicking on rollout #343
• [BUGFIX] Fix ingesters occassionally double flushing #364
• [BUGFIX] Fix S3 backend logs "unsupported value type" #381

• [CHANGE] Bloom filters are now sharded to reduce size and improve caching, as blocks grow. This is a breaking change and all data stored before this change will not be queryable. #192
• [CHANGE] Rename maintenance cycle to blocklist poll. #315
• [ENHANCEMENT] CI checks for vendored dependencies using make vendor-check. Update CONTRIBUTING.md to reflect the same before checking in files in a PR. #274
• [ENHANCEMENT] Add warnings for suspect configs. #294
• [ENHANCEMENT] Add command line flags for s3 credentials. #308
• [ENHANCEMENT] Support multiple authentication methods for S3 (IRSA, IAM role, static). #320
• [ENHANCEMENT] Add per tenant bytes counter. #331
• [BUGFIX] S3 multi-part upload errors #306
• [BUGFIX] Increase Prometheus notfound metric on tempo-vulture. #301
• [BUGFIX] Return 404 if searching for a tenant id that does not exist in the backend. #321
• [BUGFIX] Prune in-memory blocks from missing tenants. #314