[Autograph] Inconsistent behaviour with lambda variable in loop #56089

bhack · 2022-05-12T22:17:32Z

Click to expand!

Issue Type

Bug

Source

source

Tensorflow Version

master

Custom Code

No

OS Platform and Distribution

No response

Mobile device

No response

Python version

No response

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

We have an inconsistent behavior with lambda variables in a loop in pure python and graph mode:
https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result

Standalone code to reproduce the issue

import tensorflow as tf

def test_a():
  fns = []
  for i in range(3):
    fns.append(lambda: print(i))
  for f in fns:
    f()

@tf.function
def test_b():
  fns = []
  for i in range(3):
    fns.append(lambda: print(i))
  for f in fns:
    f()

def test_c():
  fns = []
  for i in range(3):
    fns.append(lambda i=i: print(i))
  for f in fns:
    f()

@tf.function 
def test_d():
  fns = []
  for i in range(3):
    fns.append(lambda i=i: print(i))
  for f in fns:
    f()

test_a() 
print("=="*10)
tf.config.run_functions_eagerly(False)
test_b()
print("=="*10)
tf.config.run_functions_eagerly(True)
test_b()
print("=="*10)
test_c() 
print("=="*10)
tf.config.run_functions_eagerly(False)
test_d()
print("=="*10)
tf.config.run_functions_eagerly(True)
test_d()

2
2
2
====================
0
1
2
====================
2
2
2
====================
0
1
2
====================
0
1
2
====================
0
1
2

Relevant log output

test_b is wrongly working "as expected" in graph mode:

# coding=utf-8
def tf__test():
    with ag__.FunctionScope('test', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:
        fns = []

        def get_state():
            return ()

        def set_state(block_vars):
            pass

        def loop_body(itr):
            i = itr
            ag__.converted_call(ag__.ld(fns).append, (ag__.autograph_artifact((lambda : ag__.ld(print)(ag__.ld(i)))),), None, fscope)
        i = ag__.Undefined('i')
        ag__.for_stmt(ag__.converted_call(ag__.ld(range), (3,), None, fscope), None, loop_body, get_state, set_state, (), {'iterate_names': 'i'})

        def get_state_1():
            return ()

        def set_state_1(block_vars):
            pass

        def loop_body_1(itr_1):
            f = itr_1
            ag__.converted_call(ag__.ld(f), (), None, fscope)
        f = ag__.Undefined('f')
        ag__.for_stmt(ag__.ld(fns), None, loop_body_1, get_state_1, set_state_1, (), {'iterate_names': 'f'})

The text was updated successfully, but these errors were encountered:

bhack · 2022-05-12T22:20:00Z

See more at keras-team/keras-cv#432

/cc @mdanatg

bhack · 2022-05-12T23:31:19Z

As referenced in the official Python FAQ:

Note that this behaviour is not peculiar to lambdas, but applies to regular functions too.

mdanatg · 2022-05-13T02:16:20Z

I think it's because we pass the loop variable through a function argument, which is enough to avoid the closure aliasing:

        def loop_body(itr):
            i = itr
            ag__.converted_call(ag__.ld(fns).append, (ag__.autograph_artifact((lambda : ag__.ld(i))),), None, fscope)

In the code above, the lambda would close over the local i which has a copy of the value.

This only happens for the for_stmt operator, which passes the iterate as an argument. If we rewrite the test as a while loop, so that i is closed-ver by the function body, things are once again quirky as intended:

import tensorflow as tf

def test():
  fns = []
  i = 0
  while i < 3:
    fns.append(lambda: print(i))
    i += 1
  for f in fns:
    f()

test()


tf.autograph.set_verbosity(0, True)

@tf.function(autograph=True)
def test():
  fns = []
  i = 0
  while i < 3:
    fns.append(lambda: print(i))
    i += 1
  for f in fns:
    f()

test()

This means that the fix is also to avoid passing the iterate as argument to the loop body, and instead rely on the get_state/set_state functions, as is the case of the while loop.

bhack · 2022-05-13T11:43:12Z

Yes is what I have suspected:

from tensorflow.python.autograph.impl import api
ag__ = api._TRANSPILER.get_extra_locals()['ag__']  # pylint:disable=protected-access
def tf__test_for():
    with ag__.FunctionScope('test', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:
        fns = []

        def get_state():
            return ()

        def set_state(block_vars):
            pass

        def loop_body(itr):
            i = itr
            ag__.converted_call(ag__.ld(fns).append, (ag__.autograph_artifact((lambda : ag__.ld(print)((ag__.converted_call(ag__.ld(str), (ag__.ld(i),), None, fscope))))),), None, fscope)
        i = ag__.Undefined('i')
        ag__.for_stmt(ag__.converted_call(ag__.ld(range), (3,), None, fscope), None, loop_body, get_state, set_state, (), {'iterate_names': 'i'})

        def get_state_1():
            return ()

        def set_state_1(block_vars):
            pass

        def loop_body_1(itr_1):
            f = itr_1
            ag__.converted_call(ag__.ld(f), (), None, fscope)
        f = ag__.Undefined('f')
        ag__.for_stmt(ag__.ld(fns), None, loop_body_1, get_state_1, set_state_1, (), {'iterate_names': 'f'})

def tf__test_while():
    with ag__.FunctionScope('test_while', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:
        fns = []
        i = 0

        def get_state():
            return (i,)

        def set_state(vars_):
            nonlocal i
            (i,) = vars_

        def loop_body():
            nonlocal i
            ag__.converted_call(ag__.ld(fns).append, (ag__.autograph_artifact((lambda : ag__.ld(print)(ag__.ld(i)))),), None, fscope)
            i = ag__.ld(i)
            i += 1

        def loop_test():
            return (ag__.ld(i) < 3)
        ag__.while_stmt(loop_test, loop_body, get_state, set_state, ('i',), {})

        def get_state_1():
            return ()

        def set_state_1(block_vars):
            pass

        def loop_body_1(itr):
            f = itr
            ag__.converted_call(ag__.ld(f), (), None, fscope)
        f = ag__.Undefined('f')
        ag__.for_stmt(ag__.ld(fns), None, loop_body_1, get_state_1, set_state_1, (), {'iterate_names': 'f'})
tf__test_for()
print("====")
tf__test_while()

0
1
2
====
3
3
3

And modify your while example to correctly get the output we "expect":

import tensorflow as tf

def test():
  fns = []
  i = 0
  while i < 3:
    fns.append(lambda i=i: print(i))
    i += 1
  for f in fns:
    f()

test()


tf.autograph.set_verbosity(0, True)

@tf.function(autograph=True)
def test():
  fns = []
  i = 0
  while i < 3:
    fns.append(lambda i=i: print(i))
    i += 1
  for f in fns:
    f()

test()

bhack · 2022-05-13T14:04:25Z

@mdanatg I've manually rewritten the transformation output. Do we need to produce something like this for the lambda: case?

from tensorflow.python.autograph.impl import api
ag__ = api._TRANSPILER.get_extra_locals()['ag__']  # pylint:disable=protected-access
def tf__test_for():
    with ag__.FunctionScope('test', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:
        fns = []
        i = 0
        def get_state():
            return (i,)

        def set_state(block_vars):
            nonlocal i
            (i,) = block_vars

        def loop_body(itr):
            nonlocal i
            ag__.converted_call(ag__.ld(fns).append, (ag__.autograph_artifact((lambda : ag__.ld(print)((ag__.converted_call(ag__.ld(str), (ag__.ld(i),), None, fscope))))),), None, fscope)
            i = ag__.ld(i)
            i += 1
        ag__.for_stmt(ag__.converted_call(ag__.ld(range), (3,), None, fscope), None, loop_body, get_state, set_state, (), {'iterate_names': 'i'})

        def get_state_1():
            return ()

        def set_state_1(block_vars):
            pass

        def loop_body_1(itr_1):
            f = itr_1
            ag__.converted_call(ag__.ld(f), (), None, fscope)
        f = ag__.Undefined('f')
        ag__.for_stmt(ag__.ld(fns), None, loop_body_1, get_state_1, set_state_1, (), {'iterate_names': 'f'})

mdanatg · 2022-05-13T14:55:17Z

Yes, something like that. And then the loop_body function would be a regular thunk: def loop_body():. We may also need to initialize i with Undefined, rather than 0, and might also need to replace (), {'iterate_names': 'i'} with just ('i',), but not sure.

bhack · 2022-05-13T19:36:00Z

Do we want to handle iter as nolocal? As in the helper function above it is only in the undefined bucket:

tensorflow/tensorflow/python/autograph/converters/control_flow.py

Lines 170 to 199 in 44e84ce

    
           def _get_block_vars(self, node, modified): 
        
             """Determines the variables affected inside a control flow statement.""" 
        
             defined_in = anno.getanno(node, anno.Static.DEFINED_VARS_IN) 
        
             live_in = anno.getanno(node, anno.Static.LIVE_VARS_IN) 
        
             live_out = anno.getanno(node, anno.Static.LIVE_VARS_OUT) 
        
             fn_scope = self.state[_Function].scope 
        
             basic_scope_vars = self._get_block_basic_vars( 
        
                 modified, 
        
                 live_in, 
        
                 live_out) 
        
             composite_scope_vars = self._get_block_composite_vars(modified, live_in) 
        
             scope_vars = tuple(basic_scope_vars | composite_scope_vars) 
        
             # Variables that are modified inside the scope, but not defined 
        
             # before entering it. Only simple variables must be defined. The 
        
             # composite ones will be implicitly checked at runtime. 
        
             possibly_undefined = ( 
        
                 modified - defined_in - fn_scope.globals - fn_scope.nonlocals) 
        
             undefined = tuple(v for v in possibly_undefined if not v.is_composite()) 
        
             # Variables that are modified inside the scope, and depend on values outside 
        
             # it. 
        
             input_only = basic_scope_vars & live_in - live_out 
        
             # Place the outputs first, then sort lexicographically. 
        
             scope_vars = sorted(scope_vars, key=lambda v: (v in input_only, v)) 
        
             nouts = len(scope_vars) - len(input_only) 
        
             return scope_vars, undefined, nouts

mdanatg · 2022-05-13T19:46:49Z

Yes, I think we do.

bhack · 2022-05-13T20:34:40Z

I don't know if it makes sense:

I've changed:

      if s in live_in or s in live_out or s in nonlocals or \
        (s not in live_in and s not in live_out):

import tensorflow as tf
tf.autograph.set_verbosity(0, True)
@tf.function
def test_b():
  fns = []
  for i in range(3):
    fns.append(lambda: print(i))
  for f in fns:
    f()
test_b()

2
2
2

And

import tensorflow as tf
tf.autograph.set_verbosity(0, True)
@tf.function
def test_b():
  fns = []
  for i in range(3):
    fns.append(lambda i=i: print(i))
  for f in fns:
    f()
test_b()

0
1
2

So the output is correct but the transformation, surely, it is still quite ugly:

# coding=utf-8
def tf__test_b():
    with ag__.FunctionScope('test_b', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:
        fns = []

        def get_state():
            return (i,)

        def set_state(vars_):
            nonlocal i
            (i,) = vars_

        def loop_body(itr):
            nonlocal i
            i = itr
            ag__.converted_call(ag__.ld(fns).append, (ag__.autograph_artifact((lambda : ag__.ld(print)(ag__.ld(i)))),), None, fscope)
        i = ag__.Undefined('i')
        ag__.for_stmt(ag__.converted_call(ag__.ld(range), (3,), None, fscope), None, loop_body, get_state, set_state, ('i',), {'iterate_names': 'i'})

        def get_state_1():
            return (f,)

        def set_state_1(vars_):
            nonlocal f
            (f,) = vars_

        def loop_body_1(itr_1):
            nonlocal f
            f = itr_1
            ag__.converted_call(ag__.ld(f), (), None, fscope)
        f = ag__.Undefined('f')
        ag__.for_stmt(ag__.ld(fns), None, loop_body_1, get_state_1, set_state_1, ('f',), {'iterate_names': 'f'})

bhack · 2022-05-13T22:54:13Z

P.s. as a side note, in the BUILD the loop scoping integration test is not registered. How the CI is running this currently?:

tensorflow/tensorflow/python/autograph/tests/BUILD

Lines 26 to 27 in f2edfb6

	# # Scoping and modularity

If it is activated some loop_scoping tests are failing:

# # Scoping and modularity
reference_test(name = "loop_scoping_test")

mdanatg · 2022-05-13T23:24:16Z

Ah, that's not intended. Can add it to the build file and mark the failing ones with self.skipTest, then file an issue to get them to pass?

For the transformation, I'm not sure, the rules are quite finnicky, and I'm not sure I'd change them without extensive testing. Likely still safer to manually add the iterate to the list of loop_vars.

mdanatg · 2022-05-23T20:13:00Z

Side note - I just realized the limitations section of autograph does seem to document this case: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/autograph/g3doc/reference/limitations.md#variables-closed-over-by-lambda-functions

google-ml-butler · 2022-08-02T17:32:35Z

Are you satisfied with the resolution of your issue?
Yes
No

clime · 2023-02-04T16:35:36Z

Hello, I have just discovered this in my app logs:

WARNING:tensorflow:From /home/clime/.virtualenvs/keras/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.
Instructions for updating:
Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089

Is there anything I should look for in my code or is this a generic warning that displays to anyone? To be more precise, does the fact that I am seeing this warning mean that I am actually using something which is now deprecated?

bhack · 2023-02-04T17:01:13Z

@clime We hope to align to the python behavior after the deprecation:

https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result

clime · 2023-02-04T18:49:02Z

@clime We hope to align to the python behavior after the deprecation:

https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result

My question rather is, if seeing that warning on my screen necessarily means that I am triggering that deprecated behavior (i.e. I am using lambda somewhere in @tf.function or similar).

bhack · 2023-02-04T19:18:39Z

Do you have a small code gist to reproduce this?

clime · 2023-02-04T19:26:03Z

Do you have a small code gist to reproduce this?

Sorry, I don't understand what I should be reproducing. I just have a pretty big code base and I just want to know how much relevant the warning is for me :).

clime · 2023-02-04T19:29:35Z

I guess the answer is here: 6197fa3

bhack · 2023-02-04T20:50:10Z

If it was internal someone introduced a new deprecation case internally ignoring the warning.

sokrypton · 2023-02-14T06:10:37Z

Are there any instructions on how to hide this warning? :P

python3.7/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.
Instructions for updating:
Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089

bhack · 2023-02-14T09:09:52Z

It is suppressed in nightly and in the next release

MaximeLee · 2023-10-09T07:52:41Z

Hi,
How can one disable this warning so that it is not printed? I am using TF 2.11.1 and I cant really change my TF version.

It is suppressed in nightly and in the next release

google-ml-butler bot added the type:bug Bug label May 12, 2022

google-ml-butler bot assigned tilakrayal May 12, 2022

bhack mentioned this issue May 12, 2022

Fix the comment about the lambda variable scope in a loop keras-team/keras-cv#432

Closed

bhack changed the title ~~[Autograph] Inconsistent behaviour between lambda variable in loop~~ [Autograph] Inconsistent behaviour with lambda variable in loop May 12, 2022

tilakrayal added comp:autograph Autograph related issues stat:awaiting response Status - Awaiting response from author labels May 13, 2022

tilakrayal added stat:awaiting tensorflower Status - Awaiting response from tensorflower and removed stat:awaiting response Status - Awaiting response from author labels May 13, 2022

This was referenced May 13, 2022

[Autograph] Enable loop_scoping_test #56106

Merged

Add lamba var loop test #56119

Merged

Unexpected output when lambda layers are used in a for loop #56169

Closed

copybara-service bot closed this as completed in #56119 Aug 2, 2022

bhack reopened this Sep 23, 2022

leandrolcampos mentioned this issue Dec 5, 2022

Ranking models with MultiWorkerMirroredStrategy strategy tensorflow/models#10849

Closed

3 tasks

yuanpengpeng mentioned this issue Dec 12, 2022

train error zhangzhehao95/CNN-MoCo#2

Closed

fiona-lxd mentioned this issue Dec 29, 2022

Would you like to give scripts or hyper-parameter table for other settings? yongchaoz/FRePo#3

Closed

DivertisseurGames mentioned this issue Jan 6, 2023

Evaluation of SimpleRLPlayer not working hsahovic/poke-env#349

Open

SuryanarayanaY mentioned this issue Feb 3, 2023

Conditional padding issue with jit_compile=True #59474

Open

Mohamedellebody mentioned this issue Feb 6, 2023

Fine tuning stops when reaching this"Starting training loop at step 330751" google-research/scenic#679

Open

snerligit mentioned this issue Feb 16, 2023

Unable to execute simple finetune examples due to jnp.linalg.norm error phbradley/alphafold_finetune#4

Open

rohitupreti mentioned this issue Feb 17, 2023

issue while running evaluate.py SlapBot/drone-detection#29

Open

AndreZeug mentioned this issue Mar 18, 2023

overflow encountered in double_scalars IuliiaBaglaeva/TransientAnalyzerUI#1

Open

lbw124765283 mentioned this issue Mar 25, 2023

Error when running akita_train.py calico/basenji#144

Open

ariG23498 mentioned this issue Mar 27, 2023

feat: Random Flips with same seed instead of using tf.cond TrystAI/restorers#59

Open

Amg9794 mentioned this issue Apr 29, 2023

getting error SamanaBS/Keyword-Spotting-System#1

Open

von-elfen mentioned this issue May 3, 2023

RuntimeError: jaxlib/gpu/solver_kernels.cc:45: operation gpusolverDnCreate(&handle) failed: cuSolver internal error google-deepmind/alphafold#758

Closed

jprokos26 mentioned this issue Aug 17, 2023

[BUG]: export_batches errors when using model with tensor output instead of numpy twosixlabs/armory#1982

Closed

Alexey234432 mentioned this issue Dec 14, 2023

coco_eval_script script with yolov7_w6_coco.h5 leads to low AP leondgarse/keras_cv_attention_models#147

Closed

easonyang1996 mentioned this issue Jan 26, 2024

Error when loading large-3m allenai/unified-io-2#11

Closed

Aya121298 mentioned this issue Mar 12, 2024

Bug: import autokeras as ak --> ModuleNotFoundError: No module named 'tensorflow.keras.layers.experimental' keras-team/autokeras#1906

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Autograph] Inconsistent behaviour with lambda variable in loop #56089

[Autograph] Inconsistent behaviour with lambda variable in loop #56089

Issue Type

Source

Tensorflow Version

Custom Code

OS Platform and Distribution

Mobile device

Python version

Bazel version

GCC/Compiler version

CUDA/cuDNN version

GPU model and memory

Current Behaviour?

Standalone code to reproduce the issue

Relevant log output

[Autograph] Inconsistent behaviour with lambda variable in loop #56089

[Autograph] Inconsistent behaviour with lambda variable in loop #56089

Comments

Issue Type

Source

Tensorflow Version

Custom Code

OS Platform and Distribution

Mobile device

Python version

Bazel version

GCC/Compiler version

CUDA/cuDNN version

GPU model and memory

Current Behaviour?

Standalone code to reproduce the issue

Relevant log output