-
Notifications
You must be signed in to change notification settings - Fork 132
Open
Description
Hi! I was working with relational-gcn with my dataset (X.shape = [17069, 300], num relations = 11) and I got some problems with GPU's memmory. I have two GeForce GTX 1080 Ti
InternalError: 2 root error(s) found.
(0) Internal: Dst tensor is not initialized.
[[{{node _arg_input_11_0_6}}]]
[[loss_2/add/_147]]
(1) Internal: Dst tensor is not initialized.
[[{{node _arg_input_11_0_6}}]]
0 successful operations.
0 derived errors ignored.
Stats:
Limit: 10977044071
InUse: 8214125056
MaxInUse: 8214411008
NumAllocs: 130
MaxAllocSize: 1220657664
The code for setup GPUs usage
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
try:
# Restrict TensorFlow to only use the fourth GPU
tf.config.experimental.set_visible_devices(gpus[0], 'GPU')
tf.config.experimental.per_process_gpu_memory_fraction=0.4
# Currently, memory growth needs to be the same across GPUs
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
logical_gpus = tf.config.experimental.list_logical_devices('GPU')
print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
except RuntimeError as e:
# Memory growth must be set before GPUs have been initialized
print(e)
Is it possible to vary the amount of memory when training the model?
Please, let me know if I'm doing something wrong
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels