[Backport main] Remove skip building graph check for quantization use case

[Backport main] Remove skip building graph check for quantization use case #5223