[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MediaItems textures rendered not in the order they provided to Composition #1505

Closed
1 task
mikekudzin opened this issue Jun 27, 2024 · 2 comments
Closed
1 task

MediaItems textures rendered not in the order they provided to Composition #1505

mikekudzin opened this issue Jun 27, 2024 · 2 comments
Assignees
@droid-girl
Labels
bug editing

Comments

@mikekudzin
Copy link
mikekudzin commented Jun 27, 2024 •
edited
Loading

Version

Media3 1.3.1

More version details

Devices that reproduce the issue

Pixel 4a running Android 13
Pixel 6a running 14

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Not tested

Reproduction steps

I am trying to generate output video using following inputs:

  1. Overlay with a transparent bitmap holding drawings, texts.
  2. Video item transformed in some way (just rotation in the example)
  3. Background which is a simple non-transparent PNG

Code is pretty straightforward

        val bitmapOverlayWithText = BitmapOverlay.createStaticBitmapOverlay(overlayBitmap)
        val overlayFrameEMI = EditedMediaItem.Builder(MediaItem.fromUri(frameFile.toUri()))
            .setFrameRate(D_F_RATE)
            .setDurationUs(D_DURATION)
            .setRemoveAudio(true)
            .setEffects(
                Effects(
                    listOf(),
                    listOf(
                        OverlayEffect(ImmutableList.of(bitmapOverlayWithText)),
                    )
                )
            )
            .build()

        val backgroundMI = MediaItem.fromUri(backgroundBitmapFile.toUri())
        val backgroundEMI = EditedMediaItem.Builder(backgroundMI)
            .setFrameRate(D_F_RATE)
            .setDurationUs(D_DURATION)
            .setRemoveAudio(true)
            .build()

        val videoMI = MediaItem.Builder().setUri(baseMediaUri)
            .setClippingConfiguration(
                MediaItem.ClippingConfiguration.Builder()
                    .setStartPositionUs(0)
                    .setEndPositionUs(D_DURATION)
                    .build()
            )
            .build()

        val videoEMI = EditedMediaItem.Builder(videoMI)
            .setEffects(
                Effects(
                    listOf(),
                    listOf(ScaleAndRotateTransformation.Builder().setRotationDegrees(45f).build())
                )
            )
            .build()

        val composition = Composition.Builder(
            EditedMediaItemSequence(overlayFrameEMI),
            EditedMediaItemSequence(videoEMI),
            EditedMediaItemSequence(backgroundEMI),            
        )
            .build()

        val transformer: Transformer =
            Transformer.Builder(context)
                .setVideoMimeType(MimeTypes.VIDEO_H264)
                .setAudioMimeType(MimeTypes.AUDIO_AAC)
                .build()
        transformer.start(composition, outputFilePath)

Output of the code above:

27.Jun.2024.16_40_45.GMTout_.bitmap.overlay-video-background.mp4

There is no original video in output. BUT SOMETIMES (rare) it produces expected output.

If I remove background from the composition

        val composition = Composition.Builder(
            EditedMediaItemSequence(overlayFrameEMI),
            EditedMediaItemSequence(videoEMI),
        )
            .build()

The output is nearly what I am expecting except the background (sic!)

27.Jun.2024.16.36.12.GMTout_.bitmap.overlay-video.mp4

And finally, setting alphaScale to 0.9

            ...
            .setVideoCompositorSettings(object : VideoCompositorSettings {
                override fun getOutputSize(inputSizes: MutableList<Size>): Size {
                    return inputSizes[0]
                }

                override fun getOverlaySettings(inputId: Int, presentationTimeUs: Long): OverlaySettings {
                    return OverlaySettings.Builder()
                        .setAlphaScale(0.9f)
                        .build()
                }
            })
            .build()
27.Jun.2024.17_06_09.GMTout._.3.layers.with.09.alphaScale.mp4

It looks like the video EditedMediaItemSequence get's overlaid by background EditedMediaItemSequence at some point.

Documentation for DefaultVideoCompositor states:

When composited, textures are overlaid over one another in the reverse order of their
registration order, so that the first registered source is on the very top. The way the textures
are overlaid can be customized using the {@link OverlaySettings} output by {@link
VideoCompositorSettings}.

I can be mistaken here, though, since Composition class itself doesn't provide explicit statement about rendering order. If so, what API/approach I can use to force rendering order?

Expected result

Textures are drawn in the reverse order they provided to Composition

Actual result

Order is not preserved, some medias got overlapped. Unpredictable output.

Media

27 Jun 2024 17_06_09 GMT_overlays
27 Jun 2024 17:41:20 GMT_bImage
27 Jun 2024 17:41:20 GMT_frame

Bug Report
@droid-girl droid-girl self-assigned this Jun 28, 2024
@droid-girl
Copy link
Contributor

Hi @mikekudzin ,
The issue #1029 with ordering of EditedMediaItems was fixed in this commit and it is a part of the next 1.4.0 release. This should provide a fix for the bug you see.

I quickly looked through the code you have provided. You can also use TextOverlay as an effect and apply it to your video. This might be another approach on how to do it.

@mikekudzin
Copy link
Author

Thank you! Switching to 1.4.0-beta1 containing #1029 fixed the issue.

@androidx androidx locked and limited conversation to collaborators Aug 28, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@droid-girl droid-girl

Labels
bug editing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@droid-girl @mikekudzin