For fellow TF users here I illustrate - using the sample code provided above - the two currently available methods to minimize and limit GPU memory usage (which can be combined together):

• allow_growth and
• per_process_gpu_memory_fraction:

# Initialize the variables (i.e. assign their default value)
init = tf.global_variables_initializer()

# Limit GPU memory use
cfg = tf.ConfigProto()
# allow dynamic GPU memory allocation (increased as needed,
# but not released once allocated - requiring python kernel restart)
cfg.gpu_options.allow_growth=True
# define the hard limit on GPU memory allocation
# (caution: can cause OOM errors)
cfg.gpu_options.per_process_gpu_memory_fraction = 0.20

# Start training (notice we used custom config)
with tf.Session(config=cfg) as sess:
    sess.run(init)
    [..]

Note: to induce python's GPU memory usage to amounts visible in nvidia-smi (larger than the default minimum of 200 MiB used by most frameworks) we need to increase the number of samples in the example above to at least 20k:

n_samples=20000

I'm not sure whether to categorize this as a duplicate of #15880 or an independent feature request. My best understanding is that you simply want a lightweight solution to shrinking the GPU memory allocation without completely killing the TF process. Towards that end, I think @zheng-xq 's suggestion about extending Session.reset() sounds most promising. I don't understand how catboost provides a model solution. I'm not familiar with that program.

It is a duplicate with a twist. I'm not calling for lower utilization of GPU - it is perfectly fine to claim all of it, if there is an option to limit the usage to a certain percentage (per_process_gpu_memory_fraction). What's useful in catboost's approach is the idea to deallocate GPU memory after returning results from the GPU back to CPU. BTW you have already something in common with catboost: you are the only two GPU-enabled frameworks (among the 12 or so I've tested for our containers at work) that give such control over GPU memory usage to the final user - something that even docker is not allowing us to do.

I noticed that it is customary among deep learning frameworks to create such GPU memory leaks (which it is) - none of them release GPU memory, requiring users to restart python kernel to release memory. Incidentally, none apart from Tensorflow claims nearly all of it, but that's what's so special about you:). In contrast to DNNs, GBDT frameworks tend to release what they no longer need - that applies not only to catboost but also to lightgbm.

IMO it would be good to contact catboost's GPU guy, Vasily (Noxomo) from Yandex team, because catboost manages to not only allocate 95% of GPU memory but also to use 100% of GPU compute power as well (despite sequential nature of boosting itself), which I've seen only in one DNN - the sadly neglected Theano...

Nagging Assignee @poxvoculi: It has been 18 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

As pointed out in the #15880 thread, it's not plausible to immediately and automatically release GPU memory after GPU ops complete because TF has persistent state objects (Variables) that live in GPU memory and are not restricted to any special sub-region.

Nagging Assignee @poxvoculi: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @poxvoculi: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

But there is certainly excessive greediness in Tensorflow’s memory allocation: it claims all RAM on all available GPU’s, but only uses by default one GPU – the one with the lowest ID.

https://www.tensorflow.org/guide/using_gpu

Allocating video memory on all GPUs but using only 1 GPU's cores looks like a bug that could be addressed (by setting up less greedy defaults) - it is particularly important because multi-tenant environments such as OpenShift do not allow us to limit video RAM usage or GPU cores usage (unlike standard CPU case), so the current approach leads to blocking all resources by just one user operating on default settings.