mmdeploy converted model not working properly in deepstream

[imoyakin]

Hi, i'm trying use deepstream pipeline infer to execute the mmdet model which convert from mmdeploy. but it seems to be not working properly.
There is no problem when I use mmdeploy to infer.

for example,the source image:

deepsteam infer result:

mmdepoly infer result:

To understand why it happened, I printed the log:

deepstream

deepstream[27659]: dets: xc 152.875000, yc 81.750000, w 236.250000, h 302.750000, score 0.784484, channelSize 5 
deepstream[27659]: dets: xc 231.000000, yc 59.625000, w 288.000000, h 296.000000, score 0.764362, channelSize 5 
deepstream[27659]: dets: xc 92.250000, yc 82.812500, w 164.125000, h 278.500000, score 0.719975, channelSize 5 
deepstream[27659]: dets: xc 14.250000, yc 57.437500, w 114.000000, h 283.250000, score 0.685319, channelSize 5 
# hide score less than 0.5

mmdeploy cpp example()

label_id: 0, score: 0.819605, bbox: 1484.8, 262.625, 1844.8, 1425.34
label_id: 0, score: 0.803483, bbox: 977.6, 401.1, 1512, 1453.99
label_id: 0, score: 0.693058, bbox: 67.6, 270.683, 708.8, 1371.62
label_id: 0, score: 0.665411, bbox: 588, 397.519, 1115.2, 1344.16
# hide score less than 0.5

Question:

here is my pgie config file,which property is missing?

[property]
gpu-id=0

net-scale-factor=0.017453292519943295769236907684886127134428718885417
onnx-file=end2end.onnx
model-engine-file=end2end.engine
#force-implicit-batch-dim=1
parse-bbox-func-name=NvDsInferParseMMDetRet
custom-lib-path=tensorrt/libmmdeploy_tensorrt_ops.so
batch-size=1
network-mode=1
num-detected-classes=80
interval=0
gie-unique-id=1
#scaling-filter=0
#scaling-compute-hw=0
cluster-mode=2

[class-attrs-all]
pre-cluster-threshold=0.2
topk=20
nms-iou-threshold=0.5

Or is something wrong with my function?

extern "C" bool
NvDsInferParseMMDetRet(std::vector<NvDsInferLayerInfo> const &outputLayersInfo, NvDsInferNetworkInfo const &networkInfo,
                       NvDsInferParseDetectionParams const &detectionParams, std::vector<NvDsInferInstanceMaskInfo> &objectList)
{
  if (outputLayersInfo.empty())
  {
    return false;
  }

  const NvDsInferLayerInfo &output = outputLayersInfo.at(0);
  const NvDsInferLayerInfo &labels = outputLayersInfo.at(1);

  std::vector<NvDsInferInstanceMaskInfo> objects;
  std::vector<NvDsInferInstanceMaskInfo> binfo;

  const float *outputbuffer = (float *)output.buffer;
  const int32_t *labelsbuffer = (int32_t *)labels.buffer;
  const uint outputSize = output.inferDims.d[0];
  const uint labelsSize = labels.inferDims.d[0];
  const uint channelsSize = output.inferDims.d[1];

  auto test = output.inferDims.d[2];

  if (outputSize != labelsSize)
  {
    return false;
  }

  for (uint b = 0; b < outputSize; b++)
  {
    int label = labelsbuffer[b];
    NvDsInferInstanceMaskInfo bbi;
    bbi.classId = labelsbuffer[b];
    bbi.left = outputbuffer[b * channelsSize + 0];
    bbi.top = outputbuffer[b * channelsSize + 1];
    bbi.width = outputbuffer[b * channelsSize + 2];
    bbi.height = outputbuffer[b * channelsSize + 3];
    bbi.detectionConfidence = outputbuffer[b * channelsSize + 4];

    binfo.push_back(bbi);
  }

  objectList.clear();
  objectList = binfo;

  return true;
}

[ajlorenzo1315]

did you manage to solve it?

[imoyakin]

yes, i solve it.
NvDsInferInstanceMaskInfo bbi; use the wrong way to calculate. width and height fix to
bbi.width = outputbuffer[b * channelsSize + 2] - outputbuffer[b * channelsSize + 0];
bbi.height = outputbuffer[b * channelsSize + 3] - outputbuffer[b * channelsSize + 1];

[Jeck-Liu-Create]

Hello, may I ask if the model converted with mmdeploy needs additional plugins to be used in DeepStream? I want to use the mmocr model in DeepStream.

[imoyakin]

if you can transformer to tensor_rt, that can be running in DeepStream.
For some reason, mmdetect need some input from pose, that's a problem made model reduced accuracy.
we not using mmocr, so we not sure you maybe facing the same problem.