Do you happen to know how to calculate mp3 sample size of a file using for example ffprobe? Not sure if it makes sense to seek by audio frames, especially for certain formats like .wav, because there's 44k samples every second so I don't know what would be the need to seek by such a small amount

MP3 frames are always 1152 samples, no matter what sample rate or bit rate. Other codecs are different. It would be pointless to implement for uncompressed codecs, slightly less so for losslessly-compressed codecs, since there you can just re-encode. But for splitting lossy audio this would be great since I could directly visualize where my cutting points are as opposed to using an external editor to guess and also having to figure out time differerences between them as well. Of course, for some codecs this is unnecesary anyway, for example m4acut can add gapless playback tags so exact-sample splitting is possible (as long as your player supports them as well). So I suppose being able to visualize exact samples would also be good, considering the decoder time difference problem that I mentioned.
But if there's no way to do this using ffprobe then I guess it's impossible for now.

Ok I will hardcode frame size of 1152 and base the FPS and "short step" off that value for mp3. for aac it seems the frame size is 1024 so I will do similarily for that https://stackoverflow.com/questions/59173435/aac-packet-size