TablesClient: predict does not work with Python list or dictionary #9887

helinwang · 2019-11-23T00:08:33Z

The problematic line is at:

google-cloud-python/automl/google/cloud/automl_v1beta1/tables/tables_client.py

Lines 402 to 405 in bd5318a

    
           elif type_code == data_types_pb2.ARRAY: 
        
               return {"list_value": value} 
        
           elif type_code == data_types_pb2.STRUCT: 
        
               return {"struct_value": value}

The expected type of the array/struct is google.protobuf.ListValue/google.protobuf.StructValue, not Python list/dictionary.

The failure message is:

*** TypeError: Parameter to MergeFrom() must be instance of same class: expect ed google.protobuf.ListValue got list.

Preferably we should only support Python list/dictionary. Now for backwards compatibility, we should support list/dictionary/google.protobuf.ListValue/google.protobuf.StructValue

The text was updated successfully, but these errors were encountered:

helinwang · 2019-12-03T01:17:35Z

Before this bug being resolved, the user has to convert Python list/struct to google.protobuf.Value. E.g.:

predict({"array_feature": to_list_value([1,2,3]), "struct_feature": to_struct_value({"field": 1})})

Here is the helper function:

from google.protobuf import struct_pb2
import six

def to_list_value(value):
  value, _ = to_proto_value(value)
  return value.list_value

def to_struct_value(value):
  value, _ = to_proto_value(value)
  return value.struct_value

def to_proto_value(value):
  """translates a Python value to a google.protobuf.Value.

  Args:
    value: The Python value to be translated.

  Returns:
    The translated google.protobuf.Value.
  """
  # possible Python types (this is a Python3 module):
  # https://simplejson.readthedocs.io/en/latest/#encoders-and-decoders
  #   JSON              Python 2        Python 3
  # object              dict            dict
  # array               list            list
  # string              unicode         str
  # number (int)        int, long       int
  # number (real)       float           float
  # true                True            True
  # false               False           False
  # null                None            None
  if value is None:
    # translate null to an empty value.
    return struct_pb2.Value(), None
  elif isinstance(value, bool):
    # This check needs to happen before isinstance(value, int),
    # isinstance(value, int) returns True when value is bool.
    return struct_pb2.Value(bool_value=value), None
  if isinstance(value, six.integer_types) or isinstance(value, float):
    return struct_pb2.Value(number_value=value), None
  elif isinstance(value, six.string_types) or isinstance(value, six.text_type):
    return struct_pb2.Value(string_value=value), None
  elif isinstance(value, dict):
    struct_value = struct_pb2.Struct()
    for key, v in value.items():
      field_value, err = to_proto_value(v)
      if err is not None:
        return None, err

      struct_value.fields[key].CopyFrom(field_value)
    return struct_pb2.Value(struct_value=struct_value), None
  elif isinstance(value, list):
    list_value = []
    for v in value:
      proto_value, err = to_proto_value(v)
      if err is not None:
        return None, err
      list_value.append(proto_value)
    return struct_pb2.Value(list_value=struct_pb2.ListValue(
        values=list_value)), None
  else:
    return None, "unsupport data type: {}".format(type(value))

The Predict request payload is proto. Previously Python dict is automatically converted to proto. However, the conversion failed for google.protobuf.ListValue and google.protobuf.Struct. Changing the structure of the Python dict might fix the problem. However, this PR fixes the problem by generating the proto message directly. So there is no auto conversion step. FIXES googleapis#9887

* fix(automl): fix TablesClient.predict for list and struct The Predict request payload is proto. Previously Python dict is automatically converted to proto. However, the conversion failed for google.protobuf.ListValue and google.protobuf.Struct. Changing the structure of the Python dict might fix the problem. However, this PR fixes the problem by generating the proto message directly. So there is no auto conversion step. FIXES #9887 * Address comment: remove protobuf dependency requirement. protobuf is already a transitive dependency (see google-api-core's setup.py)

helinwang self-assigned this Nov 23, 2019

helinwang changed the title ~~TablesClient: predict does not work with array or struct~~ TablesClient: predict does not work with Python list or dictionary Nov 23, 2019

yoshi-automation added the triage me I really want to be triaged. label Nov 23, 2019

emar-kar added api: automl Issues related to the AutoML API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed triage me I really want to be triaged. labels Nov 24, 2019

helinwang mentioned this issue Dec 17, 2019

fix(automl): fix TablesClient.predict for array and struct #9991

Merged

helinwang closed this as completed in #9991 Jan 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TablesClient: predict does not work with Python list or dictionary #9887

TablesClient: predict does not work with Python list or dictionary #9887

TablesClient: predict does not work with Python list or dictionary #9887

TablesClient: predict does not work with Python list or dictionary #9887

Comments