- Added analyze_raw_data(), an API for evaluating TFMA metrics on Pandas DataFrames.
- Previously metrics would only be computed for combinations of keys that
produced different metric values (e.g.
ExampleCountwill be the same for all models, outputs, classes, etc, so only one metric key was used). Now a metric key will be returned for each combination associated with theMetricSpecdefinition even if the values will be the same. Support for model independent metrics has also been removed. This means by default multiple ExampleCount metrics will be created when multiple models are used (one per model). - Fixed issue with label_key and prediction_key settings not working with TF based metrics.
- Fairness Indicators UI
- Thresholds are now sorted in ascending order.
- Barchart can now be sorted by either slice or eval.
- Added support for slicing on any value extracted from the inputs (e.g. raw labels).
- Added support for filtering extracts based on sub-keys.
- Added beam counters to track the feature slices being used for evaluation.
- Adding KeyError when analyze_raw_data is run without a valid label_key or prediction_key within the provided Pandas DataFrame.
- Added documentation for
tfma.analyze_raw_data,tfma.view.SlicedMetrics, andtfma.view.SlicedPlots. - Unchecked Metric thresholds now block the model validation.
- Added support for per slice threshold settings.
- Added support for sharding metrics and plots outputs.
- Updated load_eval_result to support filtering plots by model name. Added support for loading multiple models at same output path using load_eval_results.
- Fix typo in jupyter widgets breaking TimeSeriesView and PlotViewer.
- Add
tfma.slicer.stringify_slice_key(). - Deprecated external use of tfma.slicer.SingleSliceSpec (tfma.SlicingSpec should be used instead).
- Updated tfma.default_eval_shared_model and tfma.default_extractors to better support custom model types.
- Depends on 'tensorflow-metadata>=0.22.2,<0.23'
- Changed to treat CLASSIFY_OUTPUT_SCORES involving 2 values as a multi-class classification prediction instead of converting to binary classification.
- Refactored confidence interval methodology field. The old path under
Options.confidence_interval_methodologyis now atOptions.confidence_intervals.methodology. - Removed model_load_time_callback from ModelLoader construct_fn (timing is now handled by load). Removed access to shared_handle from ModelLoader.