Hi @SantiagoMoreno-UdeA

I was able to reproduce this issue in TF Nightly as well. Please find the gist here.

A similar issue is being tracked in #59716

Does dynamic range quantization works for your case?

Thanks.

Hi @pjpratik!

Thanks for answering

I need all model in Int8 'cause I'm attempting to run whisper inference in a NPU and this only support int8 data type.
So Dynamic quantization is not an option for me :/.

Looking in advance for your answer.

Cheers!

@SantiagoMoreno-UdeA Thanks for the information.

@pkgoogle Could you please look into this issue?

Thanks.

I was able to reproduce from @pjpratik's gist.

@abattery can you please take a look

Hello @abattery have you had time to take a look on this?

@SantiagoMoreno-UdeA I guess MUL op used in this model requires 16bit activation in order to preserve its accuracy. I am still not sure what is going on with TFLiteconverter
Here is the same issue I raised so long time back and no one addresses it
#58451

When I have analyzed, observed seg fault here ...
#0 0x00007f623573d7d3 in mlir::quant::QuantizedType::getExpressedType() const () from /usr/local/lib/python3.9/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#1 0x00007f623573e1ac in mlir::quant::QuantizedType::castFromExpressedType(mlir::Type) ()
from /usr/local/lib/python3.9/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so

@nyadla-sys It seems that so far quantize Whisper it's very tricky. Thank you for your information I'll take a look.

Related to #29829

Hi there, I am facing the same issue when trying to convert whisper into int8 for running on TPU, is there any update please? Thank you.

Hi @James-Shared-Studios, No the error remains. It seems that it's very low level error.

I found that tflite versions of Whisper generate NaN values when processing the -float("inf") values that are used in one part of the transformer codebase (specifically, the logits processor that kicks in when you call generate with forced tokens). Perhaps those NaNs make the int8 quantization crash too. I made a crude patch here, which has worked for me to stop the NaNs happening: nyadla-sys/whisper.tflite#15