oom issue 3

[ckcraig01]

(metaopnet) x~/MetaOptNet$ python train.py --gpu 0 --save-path "./experiments/miniImageNet_MetaOptNet_SVM" --train-shot 5 --head SVM --network ResNet --dataset miniImageNet --eps 0.1 --episodes-per-batch 1
Loading mini ImageNet dataset - phase train
Loading mini ImageNet dataset - phase val
('using gpu:', '0')
{'episodes_per_batch': 1, 'head': 'SVM', 'val_query': 15, 'test_way': 5, 'train_way': 5, 'eps': 0.1, 'save_epoch': 10, 'val_episode': 2000, 'num_epoch': 60, 'train_query': 6, 'save_path': './experiments/miniImageNet_MetaOptNet_SVM', 'train_shot': 5, 'val_shot': 5, 'gpu': '0', 'dataset': 'miniImageNet', 'network': 'ResNet'}
Train Epoch: 1 Learning Rate: 0.1000
10%|███████████▍ | 99/1000 [00:28<04:20, 3.46it/s]Train Epoch: 1 Batch: [100/1000] Loss: 1.5324 Accuracy: 39.87 % (36.67 %)
20%|██████████████████████▉ | 199/1000 [00:57<03:59, 3.34it/s]Train Epoch: 1 Batch: [200/1000] Loss: 1.4797 Accuracy: 39.05 % (43.33 %)
30%|██████████████████████████████████▍ | 299/1000 [01:26<03:25, 3.41it/s]Train Epoch: 1 Batch: [300/1000] Loss: 1.3376 Accuracy: 39.41 % (53.33 %)
40%|█████████████████████████████████████████████▉ | 399/1000 [01:55<02:52, 3.48it/s]Train Epoch: 1 Batch: [400/1000] Loss: 1.3189 Accuracy: 39.65 % (53.33 %)
50%|█████████████████████████████████████████████████████████▍ | 499/1000 [02:24<02:25, 3.43it/s]Train Epoch: 1 Batch: [500/1000] Loss: 1.5739 Accuracy: 39.84 % (43.33 %)
60%|████████████████████████████████████████████████████████████████████▉ | 599/1000 [02:54<01:57, 3.41it/s]Train Epoch: 1 Batch: [600/1000] Loss: 1.2768 Accuracy: 40.34 % (53.33 %)
70%|████████████████████████████████████████████████████████████████████████████████▍ | 699/1000 [03:24<01:31, 3.29it/s]Train Epoch: 1 Batch: [700/1000] Loss: 1.6253 Accuracy: 40.80 % (30.00 %)
80%|███████████████████████████████████████████████████████████████████████████████████████████▉ | 799/1000 [03:55<01:01, 3.29it/s]Train Epoch: 1 Batch: [800/1000] Loss: 1.3110 Accuracy: 41.17 % (53.33 %)
90%|███████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 899/1000 [04:26<00:31, 3.26it/s]Train Epoch: 1 Batch: [900/1000] Loss: 1.3345 Accuracy: 41.54 % (56.67 %)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 999/1000 [04:57<00:00, 3.16it/s]Train Epoch: 1 Batch: [1000/1000] Loss: 1.2660 Accuracy: 41.81 % (56.67 %)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [04:57<00:00, 3.20it/s]
0%| | 1/2000 [00:00<09:57, 3.35it/s]THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524577523076/work/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Exception KeyError: KeyError(<weakref at 0x7f60e517ae10; to 'tqdm' at 0x7f60c12bbd10>,) in <bound method tqdm.del of 0%| | 1/2000 [00:00<09:57, 3.35it/s]> ignored
Traceback (most recent call last):
File "train.py", line 245, in
emb_query = embedding_net(data_query.reshape([-1] + list(data_query.shape[-3:])))
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/xxx/MetaOptNet/models/ResNet12_embedding.py", line 114, in forward
x = self.layer2(x)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/xxx/MetaOptNet/models/ResNet12_embedding.py", line 54, in forward
residual = self.downsample(x)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/xxx/anaconda3/envs/metaopnet/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1524577523076/work/aten/src/THC/generic/THCStorage.cu:58