MultiOutput Keras Model Evaluation Issue #111
Comments
|
Can you provide more of the stack trace? The config won't work as is because Precision and Recall are binary classification metrics so they need to either have a top_k parameter set or be binarized using BinarizationOptions, but it is failing at inference before it gets to evaluation so need to see what is causing that failure first. |
|
Hi @mdreves , I've ran it again, see below the full stack trace. Thanks |
|
Hi @mdreves , Just checking if you've had the chance to look further into the above yet? Thanks |
|
The stack trace still seems to be truncated (and mixed with possibly other concerning errors - although TF seems to ignore them). However, I did just notice that you proved an example serving signature called 'serving_raw', but you didn't set the signature name in your config (the default is to call the model directly if no signature is provided). Try updating your config to the following: If this doesn't work, what I would do is load the saved model directly and try to call inference on it and see if you get an error. For example: |
|
Hi @mdreves , Thanks for the suggestion. Just to give a bit more background, the model i have has 2 signatures. 'serving_raw' is from following the TFX example here (i've included a snippet below) >> https://www.tensorflow.org/tfx/tutorials/tfx/components_keras#trainer However, the client that will be interacting with the model does not have the capability to encode the incoming request into a binary single input as expected by 'serving_default', which is why i've created another signature called 'serving_raw' to accept raw input features for the model to predict. And the signature for the model is: I've checked by loading the saved model back in via And checking it with a test data (The serving component includes the transform layer to transform raw features that's expected by the model hence checking with a 'transformed' example features to test directly with the model) This successfully returns the prediction (probability distribution) for the 2 outputs when i call I've tried to increase the verbosity of the output and have ran the TFMA component again, below is what is printed from the TFMA error. Hopefully this gives a bit more info? Also, i'm thinking it should be fine to leave Eval_Config (with precision and recall taken out for now) >> TFMA trace >> |
|
Hi @mdreves , Thought i'll share with you the updates on my end on the issue above. What was happening before when the error above occurred, is that in the Evaluator component of the tfx pipeline i have, i was passing the 'transformed' outputs from the transform stage for evaluation of the 2 classification outputs i have - reason being in the transform layer, i'm encoding the 2 labels from raw string to their int representation. What worked, was instead of passing in the labels as raw string, i've encoded them separately so at the very early stage of ingesting using ExampleGen, the labels that come with the raw data are already encoded. And instead of then using the 'transformed' examples
I thought by passing in the 'transformed' artifacts, the evaluator component should then run the comparison between model prediction against the transformed labels (which is what i want since it wouldn't work if i pass in the raw string labels to compare). But it looks like this may not be supported in tfma yet..? Ideally, i would like to be able to include the label transformation as part of the pipeline (instead of pre-processing it separately beforehand), and still be able to go through the evaluator component so it can evaluate based on the 'transformed' labels. Are you aware if there's a way we can include the transform graph into the Evaluator component if i can't pass in the transformed dataset; e.g.: So that i can get the transform graph to operate on the raw example_gen.outputs['examples'] data, where it transform the labels, and tfma then use this to evaluate the model?
For Reason this may be applicable is to assess the evaluation of the splits individually - .e.g: check the Precision on Test set vs Evaluation set.
For the eval_config, I understand the need for Anything else, it'll error out. Just wondering, what role does label_key play if the output is already specified in output_names, and in my case of a multi-output model, logically speaking i would be inclined to configure The Would appreciate your input on the above. Thanks! |
|
To use TFT with TFMA you need to add a workaround described in [1]. Passing TFT outputs directly should work if your model takes TFT inputs and the names used for the model input names match the TFT output names. Given the labels were causing the issue I suspect the issue is that the outputs from TFT were not in the form that you expected for the labels (i.e. maybe the TFT output was dense while the metrics you used expected sparse). The only way I know to do this is to add the splits to your examples as features and then slice on them. The label_key does not need to match the output_name. The label_key identifies the name of the input feature to use for the label. If you have multiple outputs and a single label_key then the assumption is that you want to use the same label for all outputs. I think what you want is to specify a different label for each output. This is done using the label_keys field (note the 's' on the end). See [2]. [2] |
|
Hi @mdreves , Thanks for your input. Much appreciated.
and LabelKeysEntry expects a key value pair as mentioned here I'm pretty sure this is just an issue on my config end, but have tried to look around and couldn't find an example that has I've set it to what i believe is required as per the documentation; But am getting an error below when i run the evaluator component Most probably because the label_keys has not been set up correctly - is what i have above the right way to set it? FYI - _CATEGORICAL_LABEL_KEYS is just a list i have for the model output names, and _transformed_name is just a function that will add '_xf' to the suffix of the output name Thanks! |
|
The label_keys are really just a dict mapping output names to label names. Proto is not always obvious and friendly to use, but you can use a dict in this case. Try the following: |
|
Hi @mdreves , Gotcha! I've tried the above and it works. Thanks |
|
Very helpful thread, thank you. I also have a two-headed output, with one target read from the input data and the other calculated based on the provided input target. I have noticed something peculiar: if my input target is called What can be the reason for this? *) This refers to the output features dictionary in the preprocessing function and the outputs dictionary to the Keras model (and by consequence, in the |
|
I'm a bit confused by the use of the word output here, but let's say the the model takes as raw inputs The raw label names are determined by your feature inputs (e.g. tf.Example), the transformed label names are determined by your the names output by your preprocessing function (e.g. TFT, etc), and the output names are determined by your model. |
|
Thank you for your response @mdreves sorry for being a bit unclear. Yes, I totally follow your logic. Here is a more structured explanation of my issue, based on the Chicago taxi example. I define my label names in In In To do evaluation in This works, everything is fine. I can see that the output signature has the corresponding names: What baffles me, is if I try to change the label (and output) names to something else in I get the following error: I feel like I might be missing something very obvious here. Any input would be greatly appreciated. |
|
I don't see the signature definitions, but just to clarify it seems you are using the same names for the both the inputs to the model (i.e. the output of TFT) and the outputs to the model. I don't see an issue, but just want to confirm to avoid confusion. I would be useful to get the signature for the tft_layer or get an example of its outputs by passing an input example to model.tft_layer(...). |
|
Yes, you are absolutely right on your first point. I figured out what caused the error. I had changed the name of my target variables during development of the pipeline, so when the (FYI: I wanted to specify output names of the model, so it will be more legible for downstream components, instead of |
|
Hello @axeltidemann @wlee192 @mdreves , I am having a similar issue but with images. My output names from my model are below. I also check that they are the same as above in the Evaluator by using The signature for my tft_layer is defined below Following the directions in this thread I have created the evaluation configuration below: Then I use the evaluation configuration with the following call in my TFX pipeline: Resulting in this error: So it looks like I am getting a prediction, but the labels are not able to map to them. It was my understanding that Things I have tried:
My feeling is that there is something wrong with my signature creation but I am not sure how to check the output of |
|
I'm not sure about this part of your code: This would remove all the features except the IMAGE_KEY, are you sure it's not the opposite you want? |
|
@mdreves It was my impression that in you remove all features/labels that are not input into the model. In my case my only model input is an image aka In testing removing 'IMAGE_KEY` causes Which makes sense because it expects an image to be passed to it. I also tried keeping all three, but it resulted in the same error as my previous post. |
|
If you remove the labels from the output then TFMA will not have access to them. If you add back just the labels in addition to the model inputs then you should be fine. Typically the way you want to set this up is to have two different signatures (one for serving) and one for the transformed outputs that you want to use with TFMA (e.g. transformed labels and any transformed features for slicing). There is an example here: You can then setup you EvalConfig to specify the signature to use for inference (signature_name) and a separate one for getting access to the transformed labels/features (preprocessing_function_names). |
|
@mdreves Thank you so much! For anyone looking at this later these are my updated signatures: And this is my updated evaluation configuration |
|
Thanks @gcarr1020. Your example clarified multi-output case. |
System information
provided in TensorFlow Model Analysis): NO
Describe the problem
Hi, i've been following the TFX Chicago Taxi Example (https://www.tensorflow.org/tfx/tutorials/tfx/components_keras#evaluator) to factor my TensorFlow code into the TFX framework.
However, for my use case, it's a multi-output keras model, where the model consumes a given input, and produces 2 outputs (both being multi-class).
If i ran the evaluator component with just 1 output (e.g.: disable the other output in my model) , it works fine and i can run
tfma.run_model_analysiswithout an issue.However, reverting to my multi-output model, running the evaluator component throws up an error.
Model - output_0 has 5 classes, and output_1 has 8 classes to predict >>
Eval_Config >>
Running tfma.run_model_analysis using the above eval_config,
I get an error message of the below >>
I've tried to find code examples of multi-output eval_config but haven't come across one yet.
Following the documentation, i've arrived at what i think the
eval_configshould be for a multi-output model - however is it set up correctly given the error message?The text was updated successfully, but these errors were encountered: