In conjunction with issue #3897 , I see it as a bug that we are not rescaling the bbox coordinates with respect to padded batch images.

Possible solution
in case of batch_size=1 return unpadded image and bbox
for batch_size > 1 rescale bbox coordinates to new_padded_image_size/old_image size

Other alternatives
Rightnow the workaround is using map and lambda functions as shown in the above link i.e save old image shape in the dict and use it to rescale or create a entirely new map function which resizes each image not pad and rescales bbox accordingly