Hi @RageshAntonyHM,

I am trying to reproduce the issue while I had another error ModuleNotFoundError: No module named 'keras_nlp.backend, could you please confirm the version of it.

Thank You

@LakshmiKalaKadali

it is keras 3.0.5 and installed keras-nlp via pip install git+https://github.com/keras-team/keras-nlp (0.8.1)

@LakshmiKalaKadali

first install pip install git+https://github.com/keras-team/keras-nlp and then update Keras (pip install -U keras)

Then install tensorflow-datasets also @LakshmiKalaKadali

Also crashing in Colab with or without quantization.

@farmaker47

This conversion pipeline needs lot of Vram. At Least 24 GB.

@LakshmiKalaKadali any updates on this please?

@RageshAntonyHM
I got same crash in colab A100(40GB GPU RAM).

@urim85

Yeah. Actually, till it is crashing for me in 48 GB RTX 6000.

(What I told to @farmaker47 was, it will crash prematurally if VRAM is low. But also crashes in final step even if you have enough VRAM)

I saw that training is working OK having installed first TensorFlow nightly version (2.17.0-dev20240223). @RageshAntonyHM can you try with nightly version and check again the conversion?

@farmaker47

How to install TensorFlow nightly version? I tried pip install tf-nightly, but I am getting error

File "/usr/local/lib/python3.10/dist-packages/keras/src/backend/tensorflow/core.py", line 5, in
from tensorflow.compiler.tf2xla.python.xla import dynamic_update_slice
ModuleNotFoundError: No module named 'tensorflow.compiler.tf2xla'

Name: tf-nightly
Version: 2.17.0.dev20240223

I work with Colab.
So it is

!pip install tf-nightly
!pip install -q --upgrade keras-nlp
!pip install -q -U keras>=3

@farmaker47

Now, again I am getting that first mentioned error

could you please share your notebook link ?

The colab is from this example

https://ai.google.dev/gemma/docs/lora_tuning

I have changed nothing. So the idea is if you install tf-nightly the error for conversion disappears? I don't understand from your previous answer if the error is during tf-nightly installation or during conversion.

@farmaker47

I hope some package conflicts ,like some packages reinstall 'stable' version of tensorflow. Let me check

@farmaker47

I able to ran inference already. my problem is, i need to create a TFlite model for Gemma 2B. I think there is some problem still in conversion

i am very new to AI and even python.

@farmaker47

I able to ran inference already. my problem is, i need to create a TFlite model for Gemma 2B. I think there is some problem still in conversion

i am very new to AI and even python.

Then we have to wait a little bit so the TF team solves this and provide us the tf-nightly version we can use to convert it.

@LakshmiKalaKadali

import keras_nlp.backend import ops
is not needed. Sorry

But when using all nightly versions, I got some "GraphDef" issue

a minimal script to reproduce the issue

import keras
import keras_nlp
import tensorflow as tf

os.environ["KAGGLE_USERNAME"] = '....'
os.environ["KAGGLE_KEY"] = '...'
os.environ["KERAS_BACKEND"] = "tensorflow" 

gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset("gemma_2b_en")
tf.saved_model.save(gemma_lm.backbone, '/tmp/gemma_saved_model/')

f = tf.lite.TFLiteConverter.from_saved_model('/tmp/gemma_saved_model/').convert()

I tested with tf-2.15, 2.16, and 2.17 nightly and their corresponding packages. None of them works.

Hi @pkgoogle,

I have reproduced the issue in Colab with TF 2.15, the session crashed at the step generator = keras_nlp.models.GemmaCausalLM.from_preset("gemma_2b_en") . Please take a look.

Thank You

Adding @advaitjain and @paulinesho for visibility.

Use the below code snippet to generate gemma2 quantized model and it would be around 2.33gb

# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model('gemma_2/')
converter.target_spec.supported_ops = [
  tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
  tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()

# Save the model
with open('gemma2-quantized.tflite', 'wb') as f:
    f.write(tflite_model)

Will post the inference results soon

@nyadla-sys
Did you run the inference in Android?

@farmaker47

You told that you able to run it in Android

I generated the quantized model as per @nyadla-sys code and tried running in Android:

   private lateinit var interpreter: Interpreter
    private  val OUTPUT_BUFFER_SIZE = 800
    private val outputBuffer = ByteBuffer.allocateDirect(OUTPUT_BUFFER_SIZE)

    @WorkerThread
    private fun runInterpreterOn(input: String): String {

        interpreter = Interpreter(gemmaTfLiteFile)

        outputBuffer.clear()

        // Run interpreter, which will generate text into outputBuffer
        interpreter.run(input, outputBuffer)

        // Set output buffer limit to current position & position to 0
        outputBuffer.flip()

        // Get bytes from output buffer
        val bytes = ByteArray(outputBuffer.remaining())
        outputBuffer.get(bytes)

        outputBuffer.clear()

        // Return bytes converted to String
        return String(bytes, Charsets.UTF_8)
    }

But I get this error:

Cannot convert between a TensorFlowLite tensor with type FLOAT32 and a Java object of type java.lang.String (which is compatible with the TensorFlowLite type STRING).

How to fix this?

@farmaker47 @nyadla-sys @freedomtan

How to create Gemma Tokenizer for input and outputs in Android ? I think interpreter.run(input, outputBuffer) needs the specific format of Gemma.

The inputs formats:

0 serving_default_inputs_1:0 FLOAT32 [1, 3][-1, 3]
1 serving_default_inputs:0 FLOAT32 [1, 3][-1, 3]

output:
1872 StatefulPartitionedCall_1:0 FLOAT32 [1, 3, 2048][-1, 3, 2048]

After Gemma release they have presented also this for converting Gemma models:
https://github.com/googlesamples/mediapipe/tree/main/examples/llm_inference/conversion
and the android app to use the converted files:
https://github.com/googlesamples/mediapipe/tree/main/examples/llm_inference/android

You can also dig in there for the tokenizer I suppose.

@farmaker47 @nyadla-sys @freedomtan
I was able to perform fine-tuning using LoRA and conversion to Tensorflow lite of Gemma2 using A100 GPU based on comments in this forum. I am interested in performing inference using the exported tflite in Linux (Colab). However, we need to perform tokenization ourselves (for which I used Gemma Tokenizer from keras_nlp), but the backbone takes only fixed-length input. Thus, how to generate a response from a text prompt using the tflite on Linux (Colab) is not clear to me. Any suggestions on this are welcome.

Hi All, @RageshAntonyHM, if the media-pipe workflow is unideal for your use case we have another option, AI-Edge-Torch, our PyTorch conversion library, you can find more information here: googleblog.

We actually have examples of converting and quantizing decoder-only LLMs here: https://github.com/google-ai-edge/ai-edge-torch/tree/main/ai_edge_torch/generative/examples.

If the conversion is successful, you can also try visualizing the result in model-explorer as well.

Please try them out and let us know if this resolves your issue. If you still need further help, feel free to open a new issue at the respective repo. Thanks for your help.

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

Hello Everyone, I followed all steps in this issue to create a llm in mobile using gemma-2b-it, but I have a problem when I try to run the llm with mediapipe.

E0000 00:00:1719161438.366071   20510 calculator_graph.cc:887] INTERNAL: CalculatorGraph::Run() failed: 
                                                                                                    Calculator::Open() for node "odml.infra.TfLitePrefillDecodeRunnerCalculator" failed: ; RET_CHECK failure (external/odml/odml/infra/genai/inference/calculators/tflite_prefill_decode_runner_calculator.cc:157) (prefill_runner_)!=(nullptr)

the step that i followed were:

1. trained the llm with keras.
2. Convert from keras to tflite.
3. Convert tflite to mediaPipe. (using colab
4. load .task in android with mediapipe

does Anybody know something about that error?

Hi @jigmam, if you have your converted model and android studio project please share it. We need the context around the call which is doing the error for us to debug it. Thanks for your help!

Thanks for your response.
I am using react native, this is the repository: https://github.com/jigmam/reactllm.git
the .task is in this link: https://drive.google.com/file/d/1iJW75azdzf1o6LC0-ohAW8g8brohqb7P/view?usp=drive_link

and if you need to know how I created .task, you can see:

1. https://github.com/jigmam/reactllm/blob/main/Copia_de_lora_tuning.ipynb
2. https://github.com/jigmam/reactllm/blob/main/convertidor_de_modelos_funciona_conTPU.ipynb
   3.https://colab.research.google.com/github/googlesamples/mediapipe/blob/main/examples/llm_inference/bundling/llm_bundling.ipynb (Note: like a tokenizer.model a download .model from https://huggingface.co/google/gemma-2b-it/tree/main, I know is not a better way but I didnt know how get that tokenizer.model
   I am bloking, i dont know what i can do.

if you have any questions or suggestions, please let me know.

Hi @jigmam, while there is probably a way to make mediapipe work with reactnative, I suspect we will keep running into different issues if we go that route (as it's not a well traveled road) -- Would you be open to switching to an Android Studio project? If you are targeting Android: https://ai.google.dev/edge/mediapipe/solutions/setup_android

It's okey for my, I just need MVP to present a prototype. Following your recommendation I created a new project using example that link, but not working .task, could it be a issue in converter between keras to tflite?

Hi @jigmam, currently Keras3 does not work well with tflite primarily due to saved model format issues. What I'm hearing from you is that you were able to follow https://github.com/google-ai-edge/ai-edge-torch/tree/main/ai_edge_torch/generative/examples successfully? If so, great! If not, let us know if you are stuck.

No yet, I am reading the example, several questions: do I need to make fine-tuning again? Is That tutorial just using to convert to pytorch to tflite? Remember I Made fine-tuning in keras