[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HandlingForCancelledExceptions #31882

Merged
merged 13 commits into from
Nov 19, 2022
Merged

HandlingForCancelledExceptions #31882

merged 13 commits into from
Nov 19, 2022

Conversation

xinlian12
Copy link
Member
@xinlian12 xinlian12 commented Nov 2, 2022 •
edited
Loading

Issue
In our recent CI pipeline, local test & spark jobs, the following exception has been logged. It does not really impact the prod requests, but could cause confusion.

00:09:48.702 [cosmos-rntbd-nio-3-4] ERROR c.a.c.i.d.RntbdTransportClient - Report this classes issue to ensure it is addressed:
[RntbdServiceEndpoint({"id":16,"closed":false,"concurrentRequests":2,"remoteAddress":"cdb-ms-prod-eastus1-be3.documents.azure.com:14407","channelPool":{"remoteAddress":"cdb-ms-prod-eastus1-be3.documents.azure.com:14407","isClosed":false,"configuration":{"maxChannels":130,"maxRequestsPerChannel":30,"idleConnectionTimeout":0,"readDelayLimit":65000000000,"writeDelayLimit":10000000000},"state":{"channelsAcquired":1,"channelsAvailable":0,"requestQueueLength":0}},"transportClient":{"id":2,"closed":false,"endpointCount":13,"endpointEvictionCount":0}})]
[com.azure.cosmos.implementation.directconnectivity.RntbdTransportClient.lambda$invokeStoreAsync$2(RntbdTransportClient.java:266)]
[request completed with an unexpected class java.util.concurrent.CancellationException: \{"record":AsyncRntbdRequestRecord({"args":{"transportRequestId":93,"activityId":"567bd093-5a7d-11ed-b69a-93bb60c2c445","origin":"rntbd://cdb-ms-prod-eastus1-be3.documents.azure.com:14407","replicaPath":"/apps/d5b32dbb-78fd-49a0-889f-fc16669a462e/services/53a011c3-d2ba-4188-988b-c3094eb60310/partitions/6d262d09-4798-487d-9048-9d94ea00d8c7/replicas/133118463084888432s","timeCreated":"2022-11-02T07:09:48.673339Z","lifetime":"PT0.022387672S"},"requestLength":-1,"responseLength":-1,"status":{"done":false,"cancelled":false,"completedExceptionally":false},"timeline":[{"eventName":"created","startTimeUTC":"2022-11-02T07:09:48.673339Z","durationInMilliSecs":0.031},{"eventName":"queued","startTimeUTC":"2022-11-02T07:09:48.673370Z","durationInMilliSecs":0.001},{"eventName":"channelAcquisitionStarted","startTimeUTC":"2022-11-02T07:09:48.673371Z","durationInMilliSecs":22.418},{"eventName":"pipelined","startTimeUTC":null,"durationInMilliSecs":0.0},{"eventName":"transitTime","startTimeUTC":null,"durationInMilliSecs":0.0},{"eventName":"decodeTime","startTimeUTC":null,"durationInMilliSecs":0.0},{"eventName":"received","startTimeUTC":null,"durationInMilliSecs":0.0},{"eventName":"completed","startTimeUTC":null,"durationInMilliSecs":0.0}]}),"error":{"cause":null,"stackTrace":[{"classLoaderName":null,"moduleName":"java.base","moduleVersion":"11.0.9","methodName":"cancel","fileName":"CompletableFuture.java","lineNumber":2396,"className":"java.util.concurrent.CompletableFuture","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"MonoCompletionStage.java","lineNumber":56,"className":"reactor.core.publisher.MonoCompletionStage$1","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drainLoop","fileName":"Operators.java","lineNumber":2252,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drain","fileName":"Operators.java","lineNumber":2220,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2032,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxDoFinally.java","lineNumber":134,"className":"reactor.core.publisher.FluxDoFinally$DoFinallySubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxContextWrite.java","lineNumber":141,"className":"reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxFlatMap.java","lineNumber":359,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drainLoop","fileName":"Operators.java","lineNumber":2252,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drain","fileName":"Operators.java","lineNumber":2220,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2032,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"terminate","fileName":"Operators.java","lineNumber":1240,"className":"reactor.core.publisher.Operators","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxFlatMap.java","lineNumber":1022,"className":"reactor.core.publisher.FluxFlatMap$FlatMapInner","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"unsubscribeEntry","fileName":"FluxFlatMap.java","lineNumber":340,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"unsubscribeEntry","fileName":"FluxFlatMap.java","lineNumber":219,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"unsubscribe","fileName":"FluxFlatMap.java","lineNumber":1083,"className":"reactor.core.publisher.FlatMapTracker","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxFlatMap.java","lineNumber":360,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"MonoCollectList.java","lineNumber":144,"className":"reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drainLoop","fileName":"Operators.java","lineNumber":2252,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drain","fileName":"Operators.java","lineNumber":2220,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2032,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxMap.java","lineNumber":169,"className":"reactor.core.publisher.FluxMap$MapSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drainLoop","fileName":"Operators.java","lineNumber":2252,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drain","fileName":"Operators.java","lineNumber":2220,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2032,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxTakeUntil.java","lineNumber":146,"className":"reactor.core.publisher.FluxTakeUntil$TakeUntilPredicateSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"MonoIgnoreElements.java","lineNumber":104,"className":"reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxConcatArray.java","lineNumber":286,"className":"reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drainLoop","fileName":"Operators.java","lineNumber":2252,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drain","fileName":"Operators.java","lineNumber":2220,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2032,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"terminate","fileName":"Operators.java","lineNumber":1240,"className":"reactor.core.publisher.Operators","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxFlatMap.java","lineNumber":1022,"className":"reactor.core.publisher.FluxFlatMap$FlatMapInner","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"unsubscribeEntry","fileName":"FluxFlatMap.java","lineNumber":340,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"unsubscribeEntry","fileName":"FluxFlatMap.java","lineNumber":219,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"unsubscribe","fileName":"FluxFlatMap.java","lineNumber":1083,"className":"reactor.core.publisher.FlatMapTracker","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxFlatMap.java","lineNumber":360,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"doOnCancel","fileName":"MonoSingle.java","lineNumber":108,"className":"reactor.core.publisher.MonoSingle$SingleSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2784,"className":"reactor.core.publisher.Operators$MonoInnerProducerBase","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"terminate","fileName":"Operators.java","lineNumber":1240,"className":"reactor.core.publisher.Operators","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"MonoFlatMap.java","lineNumber":187,"className":"reactor.core.publisher.MonoFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxPeekFuseable.java","lineNumber":159,"className":"reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"doOnCancel","fileName":"MonoSingle.java","lineNumber":108,"className":"reactor.core.publisher.MonoSingle$SingleSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2784,"className":"reactor.core.publisher.Operators$MonoInnerProducerBase","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"terminate","fileName":"Operators.java","lineNumber":1240,"className":"reactor.core.publisher.Operators","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"MonoFlatMap.java","lineNumber":187,"className":"reactor.core.publisher.MonoFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"terminate","fileName":"Operators.java","lineNumber":1240,"className":"reactor.core.publisher.Operators","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"MonoFlatMap.java","lineNumber":187,"className":"reactor.core.publisher.MonoFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxFlatMap.java","lineNumber":359,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drainLoop","fileName":"Operators.java","lineNumber":2252,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drain","fileName":"Operators.java","lineNumber":2220,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2032,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxTakeUntil.java","lineNumber":146,"className":"reactor.core.publisher.FluxTakeUntil$TakeUntilPredicateSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drainLoop","fileName":"Operators.java","lineNumber":2252,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drain","fileName":"Operators.java","lineNumber":2220,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2032,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxTake.java","lineNumber":172,"className":"reactor.core.publisher.FluxTake$TakeSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"doOnCancel","fileName":"MonoSingle.java","lineNumber":108,"className":"reactor.core.publisher.MonoSingle$SingleSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2784,"className":"reactor.core.publisher.Operators$MonoInnerProducerBase","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drainLoop","fileName":"Operators.java","lineNumber":2252,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drain","fileName":"Operators.java","lineNumber":2220,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2032,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"MonoPeekTerminal.java","lineNumber":144,"className":"reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"terminate","fileName":"Operators.java","lineNumber":1240,"className":"reactor.core.publisher.Operators","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"MonoFlatMap.java","lineNumber":187,"className":"reactor.core.publisher.MonoFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxMapFuseable.java","lineNumber":176,"className":"reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"terminate","fileName":"Operators.java","lineNumber":1240,"className":"reactor.core.publisher.Operators","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"MonoFlatMap.java","lineNumber":272,"className":"reactor.core.publisher.MonoFlatMap$FlatMapInner","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"MonoFlatMap.java","lineNumber":188,"className":"reactor.core.publisher.MonoFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"doOnCancel","fileName":"MonoSingle.java","lineNumber":108,"className":"reactor.core.publisher.MonoSingle$SingleSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2784,"className":"reactor.core.publisher.Operators$MonoInnerProducerBase","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxMap.java","lineNumber":169,"className":"reactor.core.publisher.FluxMap$MapSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drainLoop","fileName":"Operators.java","lineNumber":2252,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drain","fileName":"Operators.java","lineNumber":2220,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"Operators.java","lineNumber":2032,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxRetryWhen.java","lineNumber":163,"className":"reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"SerializedSubscriber.java","lineNumber":157,"className":"reactor.core.publisher.SerializedSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxMap.java","lineNumber":169,"className":"reactor.core.publisher.FluxMap$MapSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"set","fileName":"Operators.java","lineNumber":1160,"className":"reactor.core.publisher.Operators","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"cancel","fileName":"FluxMergeSequential.java","lineNumber":601,"className":"reactor.core.publisher.FluxMergeSequential$MergeSequentialInner","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"FluxMergeSequential.java","lineNumber":240,"className":"reactor.core.publisher.FluxMergeSequential$MergeSequentialMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"next","fileName":"FluxGenerate.java","lineNumber":178,"className":"reactor.core.publisher.FluxGenerate$GenerateSubscription","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"lambda$getPaginatedQueryResultAsObservable$1","fileName":"Paginator.java","lineNumber":108,"className":"com.azure.cosmos.implementation.query.Paginator","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"slowPath","fileName":"FluxGenerate.java","lineNumber":271,"className":"reactor.core.publisher.FluxGenerate$GenerateSubscription","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"request","fileName":"FluxGenerate.java","lineNumber":213,"className":"reactor.core.publisher.FluxGenerate$GenerateSubscription","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drain","fileName":"FluxMergeSequential.java","lineNumber":423,"className":"reactor.core.publisher.FluxMergeSequential$MergeSequentialMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"innerComplete","fileName":"FluxMergeSequential.java","lineNumber":328,"className":"reactor.core.publisher.FluxMergeSequential$MergeSequentialMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxMergeSequential.java","lineNumber":584,"className":"reactor.core.publisher.FluxMergeSequential$MergeSequentialInner","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxMap.java","lineNumber":144,"className":"reactor.core.publisher.FluxMap$MapSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"SerializedSubscriber.java","lineNumber":146,"className":"reactor.core.publisher.SerializedSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxRetryWhen.java","lineNumber":200,"className":"reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxMap.java","lineNumber":144,"className":"reactor.core.publisher.FluxMap$MapSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"complete","fileName":"Operators.java","lineNumber":2665,"className":"reactor.core.publisher.Operators$MonoInnerProducerBase","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"MonoSingle.java","lineNumber":180,"className":"reactor.core.publisher.MonoSingle$SingleSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"complete","fileName":"Operators.java","lineNumber":1817,"className":"reactor.core.publisher.Operators$MonoSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"MonoFlatMap.java","lineNumber":249,"className":"reactor.core.publisher.MonoFlatMap$FlatMapInner","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"FluxMapFuseable.java","lineNumber":129,"className":"reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"complete","fileName":"Operators.java","lineNumber":1816,"className":"reactor.core.publisher.Operators$MonoSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"MonoFlatMap.java","lineNumber":151,"className":"reactor.core.publisher.MonoFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"MonoPeekTerminal.java","lineNumber":180,"className":"reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"FluxOnErrorResume.java","lineNumber":79,"className":"reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"complete","fileName":"Operators.java","lineNumber":2664,"className":"reactor.core.publisher.Operators$MonoInnerProducerBase","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"MonoSingle.java","lineNumber":180,"className":"reactor.core.publisher.MonoSingle$SingleSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxTake.java","lineNumber":153,"className":"reactor.core.publisher.FluxTake$TakeSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"FluxTake.java","lineNumber":133,"className":"reactor.core.publisher.FluxTake$TakeSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"FluxSwitchIfEmpty.java","lineNumber":74,"className":"reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"FluxTakeUntil.java","lineNumber":84,"className":"reactor.core.publisher.FluxTakeUntil$TakeUntilPredicateSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"FluxRepeat.java","lineNumber":90,"className":"reactor.core.publisher.FluxRepeat$RepeatSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"tryEmitScalar","fileName":"FluxFlatMap.java","lineNumber":488,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"FluxFlatMap.java","lineNumber":421,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"complete","fileName":"Operators.java","lineNumber":1816,"className":"reactor.core.publisher.Operators$MonoSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"MonoFlatMap.java","lineNumber":151,"className":"reactor.core.publisher.MonoFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"complete","fileName":"Operators.java","lineNumber":1816,"className":"reactor.core.publisher.Operators$MonoSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"MonoFlatMap.java","lineNumber":151,"className":"reactor.core.publisher.MonoFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"complete","fileName":"Operators.java","lineNumber":2664,"className":"reactor.core.publisher.Operators$MonoInnerProducerBase","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"MonoSingle.java","lineNumber":180,"className":"reactor.core.publisher.MonoSingle$SingleSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxPeekFuseable.java","lineNumber":277,"className":"reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"complete","fileName":"Operators.java","lineNumber":1817,"className":"reactor.core.publisher.Operators$MonoSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"MonoFlatMap.java","lineNumber":151,"className":"reactor.core.publisher.MonoFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"complete","fileName":"Operators.java","lineNumber":2664,"className":"reactor.core.publisher.Operators$MonoInnerProducerBase","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"MonoSingle.java","lineNumber":180,"className":"reactor.core.publisher.MonoSingle$SingleSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"checkTerminated","fileName":"FluxFlatMap.java","lineNumber":846,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drainLoop","fileName":"FluxFlatMap.java","lineNumber":608,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"innerComplete","fileName":"FluxFlatMap.java","lineNumber":894,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxFlatMap.java","lineNumber":997,"className":"reactor.core.publisher.FluxFlatMap$FlatMapInner","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxSwitchIfEmpty.java","lineNumber":85,"className":"reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxConcatArray.java","lineNumber":230,"className":"reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"MonoIgnoreElements.java","lineNumber":89,"className":"reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxTakeUntil.java","lineNumber":121,"className":"reactor.core.publisher.FluxTakeUntil$TakeUntilPredicateSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"FluxTakeUntil.java","lineNumber":99,"className":"reactor.core.publisher.FluxTakeUntil$TakeUntilPredicateSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"FluxRepeatPredicate.java","lineNumber":86,"className":"reactor.core.publisher.FluxRepeatPredicate$RepeatPredicateSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"FluxMap.java","lineNumber":122,"className":"reactor.core.publisher.FluxMap$MapSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onNext","fileName":"FluxOnErrorResume.java","lineNumber":79,"className":"reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"complete","fileName":"Operators.java","lineNumber":1816,"className":"reactor.core.publisher.Operators$MonoSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"MonoCollectList.java","lineNumber":129,"className":"reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"checkTerminated","fileName":"FluxFlatMap.java","lineNumber":846,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drainLoop","fileName":"FluxFlatMap.java","lineNumber":608,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"innerComplete","fileName":"FluxFlatMap.java","lineNumber":894,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxFlatMap.java","lineNumber":997,"className":"reactor.core.publisher.FluxFlatMap$FlatMapInner","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"Operators.java","lineNumber":2058,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"checkTerminated","fileName":"FluxFlatMap.java","lineNumber":846,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drainLoop","fileName":"FluxFlatMap.java","lineNumber":608,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"drain","fileName":"FluxFlatMap.java","lineNumber":588,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxFlatMap.java","lineNumber":465,"className":"reactor.core.publisher.FluxFlatMap$FlatMapMain","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxContextWrite.java","lineNumber":126,"className":"reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"FluxDoFinally.java","lineNumber":128,"className":"reactor.core.publisher.FluxDoFinally$DoFinallySubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"onComplete","fileName":"Operators.java","lineNumber":2058,"className":"reactor.core.publisher.Operators$MultiSubscriptionSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"complete","fileName":"Operators.java","lineNumber":1817,"className":"reactor.core.publisher.Operators$MonoSubscriber","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"lambda$subscribe$0","fileName":"MonoCompletionStage.java","lineNumber":92,"className":"reactor.core.publisher.MonoCompletionStage","nativeMethod":false},{"classLoaderName":null,"moduleName":"java.base","moduleVersion":"11.0.9","methodName":"uniWhenComplete","fileName":"CompletableFuture.java","lineNumber":859,"className":"java.util.concurrent.CompletableFuture","nativeMethod":false},{"classLoaderName":null,"moduleName":"java.base","moduleVersion":"11.0.9","methodName":"tryFire","fileName":"CompletableFuture.java","lineNumber":837,"className":"java.util.concurrent.CompletableFuture$UniWhenComplete","nativeMethod":false},{"classLoaderName":null,"moduleName":"java.base","moduleVersion":"11.0.9","methodName":"postComplete","fileName":"CompletableFuture.java","lineNumber":506,"className":"java.util.concurrent.CompletableFuture","nativeMethod":false},{"classLoaderName":null,"moduleName":"java.base","moduleVersion":"11.0.9","methodName":"complete","fileName":"CompletableFuture.java","lineNumber":2073,"className":"java.util.concurrent.CompletableFuture","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"messageReceived","fileName":"RntbdRequestManager.java","lineNumber":819,"className":"com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdRequestManager","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"channelRead","fileName":"RntbdRequestManager.java","lineNumber":194,"className":"com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdRequestManager","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"invokeChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":379,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"invokeChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":365,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"fireChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":357,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"fireChannelRead","fileName":"ByteToMessageDecoder.java","lineNumber":336,"className":"io.netty.handler.codec.ByteToMessageDecoder","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"channelRead","fileName":"ByteToMessageDecoder.java","lineNumber":308,"className":"io.netty.handler.codec.ByteToMessageDecoder","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"invokeChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":379,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"invokeChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":365,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"fireChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":357,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"fireChannelRead","fileName":"CombinedChannelDuplexHandler.java","lineNumber":436,"className":"io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"channelRead","fileName":"CombinedChannelDuplexHandler.java","lineNumber":253,"className":"io.netty.channel.CombinedChannelDuplexHandler","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"invokeChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":379,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"invokeChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":365,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"fireChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":357,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"channelRead","fileName":"IdleStateHandler.java","lineNumber":286,"className":"io.netty.handler.timeout.IdleStateHandler","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"invokeChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":379,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"invokeChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":365,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"fireChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":357,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"unwrap","fileName":"SslHandler.java","lineNumber":1373,"className":"io.netty.handler.ssl.SslHandler","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"decodeJdkCompatible","fileName":"SslHandler.java","lineNumber":1236,"className":"io.netty.handler.ssl.SslHandler","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"decode","fileName":"SslHandler.java","lineNumber":1285,"className":"io.netty.handler.ssl.SslHandler","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"decodeRemovalReentryProtection","fileName":"ByteToMessageDecoder.java","lineNumber":519,"className":"io.netty.handler.codec.ByteToMessageDecoder","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"callDecode","fileName":"ByteToMessageDecoder.java","lineNumber":458,"className":"io.netty.handler.codec.ByteToMessageDecoder","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"channelRead","fileName":"ByteToMessageDecoder.java","lineNumber":280,"className":"io.netty.handler.codec.ByteToMessageDecoder","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"invokeChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":379,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"invokeChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":365,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"fireChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":357,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"channelRead","fileName":"DefaultChannelPipeline.java","lineNumber":1410,"className":"io.netty.channel.DefaultChannelPipeline$HeadContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"invokeChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":379,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"invokeChannelRead","fileName":"AbstractChannelHandlerContext.java","lineNumber":365,"className":"io.netty.channel.AbstractChannelHandlerContext","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"fireChannelRead","fileName":"DefaultChannelPipeline.java","lineNumber":919,"className":"io.netty.channel.DefaultChannelPipeline","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"read","fileName":"AbstractNioByteChannel.java","lineNumber":166,"className":"io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"processSelectedKey","fileName":"NioEventLoop.java","lineNumber":788,"className":"io.netty.channel.nio.NioEventLoop","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"processSelectedKeysOptimized","fileName":"NioEventLoop.java","lineNumber":724,"className":"io.netty.channel.nio.NioEventLoop","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"processSelectedKeys","fileName":"NioEventLoop.java","lineNumber":650,"className":"io.netty.channel.nio.NioEventLoop","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"run","fileName":"NioEventLoop.java","lineNumber":562,"className":"io.netty.channel.nio.NioEventLoop","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"run","fileName":"SingleThreadEventExecutor.java","lineNumber":997,"className":"io.netty.util.concurrent.SingleThreadEventExecutor$4","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"run","fileName":"ThreadExecutorMap.java","lineNumber":74,"className":"io.netty.util.internal.ThreadExecutorMap$2","nativeMethod":false},{"classLoaderName":"app","moduleName":null,"moduleVersion":null,"methodName":"run","fileName":"FastThreadLocalRunnable.java","lineNumber":30,"className":"io.netty.util.concurrent.FastThreadLocalRunnable","nativeMethod":false},{"classLoaderName":null,"moduleName":"java.base","moduleVersion":"11.0.9","methodName":"run","fileName":"Thread.java","lineNumber":834,"className":"java.lang.Thread","nativeMethod":false}],"message":null,"suppressed":[],"localizedMessage":null}}: com.azure.core.exception.AzureException
	at com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdReporter.doReportIssue(RntbdReporter.java:66)
	at com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdReporter.reportIssue(RntbdReporter.java:43)
	at com.azure.cosmos.implementation.directconnectivity.RntbdTransportClient.lambda$invokeStoreAsync$2(RntbdTransportClient.java:266)
	at reactor.core.publisher.Mono.lambda$onErrorMap$31(Mono.java:3776)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
	at reactor.core.publisher.MonoCompletionStage.lambda$subscribe$0(MonoCompletionStage.java:89)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
	at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
	at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:143)
	at reactor.core.publisher.MonoCompletionStage.subscribe(MonoCompletionStage.java:67)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	at reactor.core.publisher.Mono.subscribeWith(Mono.java:4570)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4422)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4358)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4330)
	at com.azure.cosmos.implementation.directconnectivity.RntbdTransportClient.lambda$invokeStoreAsync$5(RntbdTransportClient.java:329)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.runFinally(FluxDoFinally.java:146)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.cancel(FluxDoFinally.java:135)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.cancel(FluxContextWrite.java:141)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.cancel(FluxFlatMap.java:359)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2252)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2220)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2032)
	at reactor.core.publisher.Operators.terminate(Operators.java:1240)
	at reactor.core.publisher.FluxFlatMap$FlatMapInner.cancel(FluxFlatMap.java:1022)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.unsubscribeEntry(FluxFlatMap.java:340)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.unsubscribeEntry(FluxFlatMap.java:219)
	at reactor.core.publisher.FlatMapTracker.unsubscribe(FluxFlatMap.java:1083)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.cancel(FluxFlatMap.java:360)
	at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.cancel(MonoCollectList.java:144)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2252)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2220)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2032)
	at reactor.core.publisher.FluxMap$MapSubscriber.cancel(FluxMap.java:169)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2252)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2220)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2032)
	at reactor.core.publisher.FluxTakeUntil$TakeUntilPredicateSubscriber.cancel(FluxTakeUntil.java:146)
	at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.cancel(MonoIgnoreElements.java:104)
	at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.cancel(FluxConcatArray.java:286)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2252)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2220)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2032)
	at reactor.core.publisher.Operators.terminate(Operators.java:1240)
	at reactor.core.publisher.FluxFlatMap$FlatMapInner.cancel(FluxFlatMap.java:1022)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.unsubscribeEntry(FluxFlatMap.java:340)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.unsubscribeEntry(FluxFlatMap.java:219)
	at reactor.core.publisher.FlatMapTracker.unsubscribe(FluxFlatMap.java:1083)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.cancel(FluxFlatMap.java:360)
	at reactor.core.publisher.MonoSingle$SingleSubscriber.doOnCancel(MonoSingle.java:108)
	at reactor.core.publisher.Operators$MonoInnerProducerBase.cancel(Operators.java:2784)
	at reactor.core.publisher.Operators.terminate(Operators.java:1240)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:187)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.cancel(FluxPeekFuseable.java:159)
	at reactor.core.publisher.MonoSingle$SingleSubscriber.doOnCancel(MonoSingle.java:108)
	at reactor.core.publisher.Operators$MonoInnerProducerBase.cancel(Operators.java:2784)
	at reactor.core.publisher.Operators.terminate(Operators.java:1240)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:187)
	at reactor.core.publisher.Operators.terminate(Operators.java:1240)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.cancel(MonoFlatMap.java:187)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.cancel(FluxFlatMap.java:359)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2252)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2220)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.cancel(Operators.java:2032)
	at reactor.core.publisher.FluxTakeUntil$TakeUntilPredicateSubscriber.cancel(FluxTakeUntil.java:146)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drainLoop(Operators.java:2252)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.drain(Operators.java:2220)

Root cause:
The exception has only started to show up since 4.37.1. Since 4.37.1, the azure-core dependency has upgraded from 1.32.0 to 1.33.0, which introduced the reactor-core upgrade from 3.4.22 to 3.4.23.

In reactor-core 3.4.23, the Mono.fromCompletionStage behavior has changed when it is got cancelled because of change in this PR: reactor/reactor-core#3146. Before it will not cancel the future, since 3.4.23, it will also cancel the future. Here is a discussion about the behavior change: reactor/reactor-core#3235

The reason we are seeing the exception mentioned above is because we subscribed again internally of the cancelled future, which caused the CancellationException

Spark query tests:
Master:
image

PR:
image

@ghost ghost added the Cosmos label Nov 2, 2022
@azure-sdk
Copy link
Collaborator

API change check

API changes are not detected in this pull request.

@xinlian12
Copy link
Member Author

/azp run java - cosmos - tests

@azure-pipelines
Copy link
Azure Pipelines successfully started running 1 pipeline(s).

@xinlian12
Copy link
Member Author

/azp run java - cosmos - tests

@azure-pipelines
Copy link
Azure Pipelines successfully started running 1 pipeline(s).

@xinlian12
Copy link
Member Author

/azp run java - cosmos - tests

@azure-pipelines
Copy link
Azure Pipelines successfully started running 1 pipeline(s).

@xinlian12 xinlian12 changed the title HandlingForCancelledExceptions -- [NO REVIEW YET] HandlingForCancelledExceptions Nov 3, 2022
@FabianMeiswinkel FabianMeiswinkel mentioned this pull request Nov 15, 2022
Merged
6 tasks
FabianMeiswinkel
FabianMeiswinkel approved these changes Nov 17, 2022
View reviewed changes
Copy link
Member
@FabianMeiswinkel FabianMeiswinkel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - LGTM

kushagraThapar
kushagraThapar approved these changes Nov 17, 2022
View reviewed changes
Copy link
Member
@kushagraThapar kushagraThapar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, great work @xinlian12 !

.../main/java/com/azure/cosmos/implementation/directconnectivity/rntbd/RntbdRequestManager.java Show resolved Hide resolved
@xinlian12
Copy link
Member Author

/azp run java - cosmos - tests

@azure-pipelines
Copy link
Azure Pipelines successfully started running 1 pipeline(s).

@xinlian12
Copy link
Member Author

/azp run java - cosmos - tests

@azure-pipelines
Copy link
Azure Pipelines successfully started running 1 pipeline(s).

@xinlian12 xinlian12 merged commit 350c352 into Azure:main Nov 19, 2022
@FabianMeiswinkel FabianMeiswinkel mentioned this pull request Dec 9, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kushagraThapar kushagraThapar kushagraThapar approved these changes

@FabianMeiswinkel FabianMeiswinkel FabianMeiswinkel approved these changes

@kirankumarkolli kirankumarkolli Awaiting requested review from kirankumarkolli kirankumarkolli is a code owner

@milismsft milismsft Awaiting requested review from milismsft

@aayush3011 aayush3011 Awaiting requested review from aayush3011

@simorenoh simorenoh Awaiting requested review from simorenoh

@jeet1995 jeet1995 Awaiting requested review from jeet1995

@Pilchie Pilchie Awaiting requested review from Pilchie

Assignees
No one assigned
Labels
Cosmos
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@xinlian12 @azure-sdk @kushagraThapar @FabianMeiswinkel