Context switching causes multi GPU idling

[CanyonWind]

Single-Path-One-Shot-NAS-MXNet/oneshot_nas_blocks.py

Line 166 in df1c1ff

x = F.broadcast_mul(x, block_channel_mask.as_in_context(x.context))

Single-Path-One-Shot-NAS-MXNet/oneshot_nas_blocks.py

Lines 467 to 470 in df1c1ff

	running_mean = F.add(F.multiply(self.running_mean.data(), self.momentum.as_in_context(x.context)),
	F.multiply(mean, self.momentum_rest.as_in_context(x.context)))
	running_var = F.add(F.multiply(self.running_var.data(), self.momentum.as_in_context(x.context)),
	F.multiply(var, self.momentum_rest.as_in_context(x.context)))

[CanyonWind]

Fix ChannelSelector training stage multi GPU idling: b88a486 69d3f72

[cavalleria]

@CanyonWind hi，when i run train_oneshot-s+ scripts, first epoch is normal, but when second ecoch is done ,the val test hangs and gpu utils is 0 , the program is stuck.