[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aborted (core dumped) in tf.raw_ops.BatchFunction #69701

Open
x0w3n opened this issue Jun 13, 2024 · 2 comments
Open

Aborted (core dumped) in tf.raw_ops.BatchFunction #69701

x0w3n opened this issue Jun 13, 2024 · 2 comments
Assignees
Labels
comp:ops OPs related issues stale This label marks the issue/pr stale - to be closed automatically if no activity stat:awaiting response Status - Awaiting response from author TF 2.16 type:bug Bug

Comments

@x0w3n
Copy link
x0w3n commented Jun 13, 2024

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

Yes

Source

source

TensorFlow version

tf 2.16

Custom code

Yes

OS platform and distribution

Linux Ubuntu 22.04.3 LTS (x86_64)

Mobile device

No response

Python version

3.9.13

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

When num_batch_threads is too large, tf.raw_ops.BatchFunction triggers crash.
image

Standalone code to reproduce the issue

import tensorflow as tf

@tf.function
def simple_fn(x, y):
    return x + y

input_tensor = tf.constant([1.0, 2.0, 3.0], dtype=tf.float32)
captured_tensor = tf.constant([4.0], dtype=tf.float32)

def wrapped_fn(in_tensors, captured_tensors):
    return simple_fn(in_tensors[0], captured_tensors[0])

defun_func = tf.function(wrapped_fn).get_concrete_function([input_tensor], [captured_tensor])


result = tf.raw_ops.BatchFunction(
    in_tensors=[input_tensor],
    captured_tensors=[captured_tensor],
    f=defun_func,
    num_batch_threads=tf.constant(93389718, dtype=tf.int32),
    max_batch_size=10,
    batch_timeout_micros=1000,
    Tout=[tf.float32],
    max_enqueued_batches=10,
    allowed_batch_sizes=[5, 10],
    container='',
    shared_name='',
    batching_queue='',
    low_priority_max_batch_size=0,
    low_priority_batch_timeout_micros=0,
    low_priority_allowed_batch_sizes=[],
    low_priority_max_enqueued_batches=0,
    enable_large_batch_splitting=False,
    name=None
)

print(result)

Relevant log output

2024-06-13 14:42:13.314670: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-13 14:42:19.076124: F external/local_tsl/tsl/platform/default/env.cc:74] Check failed: ret == 0 (11 vs. 0)Thread batch_threads_ creation via pthread_create() failed.
Aborted (core dumped)
@sushreebarsa
Copy link
Contributor

@x0w3n I was able to replicate the issue reported here. One workaround could be to use tf.raw_ops.BatchFunction that offers low-level control, a simpler approach for batching a function might be using tf.data.Dataset.map with vectorization?
Could you please let us know if it helps?
Thank you!

@sushreebarsa sushreebarsa added the stat:awaiting response Status - Awaiting response from author label Jun 19, 2024
Copy link

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

@github-actions github-actions bot added the stale This label marks the issue/pr stale - to be closed automatically if no activity label Jun 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:ops OPs related issues stale This label marks the issue/pr stale - to be closed automatically if no activity stat:awaiting response Status - Awaiting response from author TF 2.16 type:bug Bug
Projects
None yet
Development

No branches or pull requests

2 participants