OOM isssues2

[ckcraig01]

~/MetaOptNet$ python train.py --gpu 0 --save-path "./experiments/miniImageNet_MetaOptNet_SVM" --train-shot 15 --head SVM --network ResNet --dataset miniImageNet --eps 0.1 --episodes-per-batch 1
Loading mini ImageNet dataset - phase train
Loading mini ImageNet dataset - phase val
('using gpu:', '0')
{'episodes_per_batch': 1, 'head': 'SVM', 'val_query': 15, 'test_way': 5, 'train_way': 5, 'eps': 0.1, 'save_epoch': 10, 'val_episode': 2000, 'num_epoch': 60, 'train_query': 6, 'save_path': './experiments/miniImageNet_MetaOptNet_SVM', 'train_shot': 15, 'val_shot': 5, 'gpu': '0', 'dataset': 'miniImageNet', 'network': 'ResNet'}
Train Epoch: 1 Learning Rate: 0.1000
10%|███████████▍ | 99/1000 [01:04<09:34, 1.57it/s]Train Epoch: 1 Batch: [100/1000] Loss: 1.4024 Accuracy: 46.83 % (56.67 %)
20%|██████████████████████▉ | 199/1000 [02:09<08:43, 1.53it/s]Train Epoch: 1 Batch: [200/1000] Loss: 1.4666 Accuracy: 46.93 % (46.67 %)
30%|██████████████████████████████████▍ | 299/1000 [03:13<07:46, 1.50it/s]Train Epoch: 1 Batch: [300/1000] Loss: 1.4943 Accuracy: 47.28 % (40.00 %)
40%|█████████████████████████████████████████████▉ | 399/1000 [04:22<06:36, 1.52it/s]Train Epoch: 1 Batch: [400/1000] Loss: 1.4684 Accuracy: 47.37 % (43.33 %)
50%|█████████████████████████████████████████████████████████▍ | 499/1000 [05:31<05:45, 1.45it/s]Train Epoch: 1 Batch: [500/1000] Loss: 1.5299 Accuracy: 47.83 % (33.33 %)
60%|████████████████████████████████████████████████████████████████████▉ | 599/1000 [06:39<04:32, 1.47it/s]Train Epoch: 1 Batch: [600/1000] Loss: 1.4971 Accuracy: 48.47 % (33.33 %)
70%|████████████████████████████████████████████████████████████████████████████████▍ | 699/1000 [07:48<03:26, 1.45it/s]Train Epoch: 1 Batch: [700/1000] Loss: 1.3542 Accuracy: 48.91 % (43.33 %)
80%|███████████████████████████████████████████████████████████████████████████████████████████▉ | 799/1000 [08:57<02:16, 1.47it/s]Train Epoch: 1 Batch: [800/1000] Loss: 1.2025 Accuracy: 49.25 % (60.00 %)
90%|███████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 899/1000 [10:06<01:09, 1.45it/s]Train Epoch: 1 Batch: [900/1000] Loss: 1.4882 Accuracy: 49.54 % (33.33 %)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 999/1000 [11:15<00:00, 1.44it/s]Train Epoch: 1 Batch: [1000/1000] Loss: 1.1104 Accuracy: 49.76 % (66.67 %)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [11:16<00:00, 1.45it/s]
0%| | 1/2000 [00:00<16:29, 2.02it/s]THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524577523076/work/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Exception KeyError: KeyError(<weakref at 0x7fa14aa80e10; to 'tqdm' at 0x7fa126bc1d10>,) in <bound method tqdm.del of 0%| | 1/2000 [00:01<16:29, 2.02it/s]> ignored
Traceback (most recent call last):
File "train.py", line 245, in
emb_query = embedding_net(data_query.reshape([-1] + list(data_query.shape[-3:])))
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/xxx/MetaOptNet/models/ResNet12_embedding.py", line 114, in forward
x = self.layer2(x)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/xxx//MetaOptNet/models/ResNet12_embedding.py", line 56, in forward
out = self.relu(out)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/activation.py", line 447, in forward
return F.leaky_relu(input, self.negative_slope, self.inplace)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/functional.py", line 731, in leaky_relu
return torch._C._nn.leaky_relu(input, negative_slope)
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1524577523076/work/aten/src/THC/generic/THCStorage.cu:58