[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KeyError: TupleType for load_eval_result from GCS with TFX 0.13.0 #46

Closed
ianhellstrom opened this issue May 28, 2019 · 4 comments
Closed

KeyError: TupleType for load_eval_result from GCS with TFX 0.13.0 #46

ianhellstrom opened this issue May 28, 2019 · 4 comments
Assignees
@mdreves
Labels
stat:awaiting tensorflower type:support

Comments

@ianhellstrom
Copy link
ianhellstrom commented May 28, 2019 •
edited
Loading

TFMA's load_eval_result('gs://bucket/path/to/evaluator/output') on TFX 0.13 (Kubeflow 0.4.1 on GCP with KFP 0.1.19 and Python 3.5) gives the following error:

KeyError                                  Traceback (most recent call last)
<ipython-input-7-4f9962d5bddb> in <module>
----> 1 result = tfma.load_eval_result('gs://bucket/path/to/evaluator/output')

/usr/lib/python3.5/site-packages/tensorflow_model_analysis/api/model_eval_lib.py in load_eval_result(output_path)
    194            for key, plot_data in plots_proto_list]
    195 
--> 196   eval_config = load_eval_config(output_path)
    197   return EvalResult(
    198       slicing_metrics=slicing_metrics, plots=plots, config=eval_config)

/usr/lib/python3.5/site-packages/tensorflow_model_analysis/api/model_eval_lib.py in load_eval_config(output_path)
    115       tf.python_io.tf_record_iterator(
    116           os.path.join(output_path, _EVAL_CONFIG_FILE)))
--> 117   final_dict = pickle.loads(serialized_record)
    118   _check_version(final_dict, output_path)
    119   return final_dict[_EVAL_CONFIG_KEY]

/usr/lib/python3.5/site-packages/dill/_dill.py in _load_type(name)
    575 
    576 def _load_type(name):
--> 577     return _reverse_typemap[name]
    578 
    579 def _create_type(typeobj, *args):

KeyError: 'TupleType'

CMLE's Python version is also 3.5 (as it was provided as a training input).

For TFX 0.12 on the same Kubeflow cluster it works as expected. I updated TFMA's version according to the instructions.

TFDV works without any issues (in the same notebook). And it also works fine when using Airflow for local development.

Could it be related to this?
@ianhellstrom ianhellstrom mentioned this issue May 28, 2019
Closed
@gowthamkpr gowthamkpr self-assigned this May 28, 2019
@chuanyu
Copy link
Contributor
chuanyu commented May 28, 2019

Seems like it could be related to the Stackoverflow question you linked to (thanks for the pointer!): could you try the workaround suggested there and see if it fixes things?

Do something like:

dill._dill._reverse_typemap['TupleType'] = type

before calling load_eval_result and see if that makes the error go away.

@ianhellstrom
Copy link
Author
ianhellstrom commented May 29, 2019 •
edited
Loading

I did, but that leads to KeyError: 'ObjectType'.

If I then add ObjectType in the same way with the reverse type map I get TypeError: type.__new__(SingleSliceSpec): SingleSliceSpec is not a subtype of type.

@gowthamkpr gowthamkpr assigned chuanyu and unassigned gowthamkpr May 29, 2019
@chuanyu chuanyu assigned mdreves and unassigned chuanyu May 31, 2019
@ianhellstrom
Copy link
Author

Any ideas what the problem might be?

@arghyaganguly
Copy link

Closing this as there has been no activity for some time.Please feel free to reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mdreves mdreves

Labels
stat:awaiting tensorflower type:support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mdreves @chuanyu @gowthamkpr @ianhellstrom @arghyaganguly