Has there been any progress on this on the TF side? This is an important feature for TF in production.

I second that request. This issue needs discussion and maybe special addressing on the coverage.py side. In particular for TF2, all tf.function decorated functions are not observed by coverage.py (as a work-around, in some cases they can be dynamically un-decorated for unittesting)

Does anyone want to take on writing a coverage.py plugin? I can help with the coverage.py side of things.

The mentioned issue ( Sujit-O/pykg2vec#123 ) hinted me at the function tf.config.experimental_run_functions_eagerly(run_eagerly), which looks like the ideal solution. However it does not seem to work completely as advertised, cf. #38170 , as certain functions cease working if switched between eager/non-eager modes.

With tensorflow 2.5, even if I run tf.config.experimental_run_functions_eagerly(True) directly before my tests, the coverage is not correctly reported. I want to write tests for a custom loss function around CRFs for a production model, where having adequate test coverage is key. Currently, I am stuck on having to exclude this file from coverage.py.

I'm the coverage.py maintainer. I'd be glad to work closely with someone from the Tensorflow side to find a solution to this problem.

Hi @nedbat, Found some articles of unit testing and code coverage of Tensorflow codes , It indicates use of tf.test api .Attaching below for reference .
https://theaisummer.com/unit-test-deep-learning/
https://www.tensorflow.org/api_docs/python/tf/test/Benchmark
https://github.com/tensorflow/docs/blob/master/site/en/community/contribute/tests.md . Are they helpful to proceed?

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

Closing as stale. Please reopen if you'd like to work on this further.

Bump; this is still an issue.

@gsakkis:

I'm the coverage.py maintainer. I'd be glad to work closely with someone from the Tensorflow side to find a solution to this problem.

Ok! Reopening as requested.

@fchollet Any further progress or thoughts on this please? Coverage reports not being accurate is a production concern. Essentially any decorated function is showing up as a miss. Could we please look into prioritising this work with @gsakkis

While this does work for some use cases tf.config.experimental_run_functions_eagerly(True) it may be more helpful to have the transformation mapping work done

I think easiest fix is to just have a global way to fully disable autograph. tf.config.experimental_run_functions_eagerly doesn't actually disable all autograph usages. A layer called in v1 mode ignores that setting entirely. While v1 layers may not support eager, they can still support not using autograph for coverage statistics. tf.data also looks to ignore that setting. Some way for tests under coverage to just fully opt out of autograph, tf.config.disable_autograph would be nice.

edit: One hacky way (depends on internals) that does look to work,

@pytest.fixture(autouse=True)
def disable_autograph_in_coverage() -> None:
    config.CONVERSION_RULES = (DoNotConvert("your_module"),) + config.CONVERSION_RULES

Scanning the tf tests this way looks to be used here. A public api for adjusting conversion rules would work too. Although I think for test case simple global flag to fully turn off tf_convert would be enough.