Fix for some sync issues b/w qemu and dpdk-vdpa

[asaini-xilinx]

vdpa/sfc: resolve race between libvhost and sfc_vdpa_pmd

For vhost-user backend, qemu sets callfd twice for each VQ.
Once when it opens vdpa-socket and thenn again when it starts
VQ(s)

To start VQ(s) qemu issues the following messages for all VQ(s):
VHOST_USER_SET_VRING_NUM
VHOST_USER_SET_VRING_BASE
VHOST_USER_SET_VRING_ADDR
VHOST_USER_SET_VRING_KICK
VHOST_USER_SET_VRING_CALL (... this overwrites the old callfd)

After processing each message dpdk-libvhost checks vq_is_ready(),
and if vq_is_ready() returns true it calls vdpa_dev->ops->dev_conf()

static bool vq_is_ready()
{
...
    ring_ok = vq->desc && vq->avail && vq->used;
...
    return ring_ok &&
            vq->kickfd != VIRTIO_UNINITIALIZED_EVENTFD &&
            vq->callfd != VIRTIO_UNINITIALIZED_EVENTFD &&
            vq->enabled;
}

And after processing VHOST_USER_SET_VRING_KICK message for the
last VQ vq_is_ready() return true b/c vq->callfd contains the old
value. So dev_conf is called before VHOST_USER_SET_VRING_CALL can be
processed for the last VQ, and VHOST_USER_SET_VRING_CALL message for
the last VQ is processed after dev_conf completes.

sfc_vdpa_pmd's dev_conf spawns a thread to set rte_vhost_host_notifier_ctrl()
before returning control to libvhost. This parallel thread in turn invokes
get_notify_area(). To get the notify_area, sfc_vdpa_pmd needs to
query the HW and for this query it needs an enabled VQ.

But at the same time libvhost is trying reset the callfd for the last VQ
to the new value and to do this it momentarily disables that VQ causing
a race b/w the processing VHOST_USER_SET_VRING_CALL and setting
rte_vhost_host_notifier_ctrl()

To resolve this race condition, query the HW and cache notify_area
inside dev_conf() instead of the parallel thread.

vdpa/sfc: handle sync issue between qemu and vhost-user

When DPDK app is running in the VF, it sometimes rings the doorbell
before sfc_vdpa_dev_config has had a chance to complete and hence it
misses the event. As workaround, ring the doorbell when vDPA reports
the notify_area to QEMU.

Signed-off-by: Vijay Kumar Srivastava <[email protected]>
Signed-off-by: Abhimanyu Saini <[email protected]>

[ol-andrewr]

Shouldn't we use already calculated *offset here?

[ol-andrewr]

One more point. I think it is a fix which should be backported to stable branches. So, add "Fixes: " tag with reference to earlier changeset to be fixed and add:
Cc: [email protected]

[ol-andrewr]

It looks like there are two spaces before &doorbeel.

[ol-andrewr]

LGTM, but please, avoid sfc_vdpa_pmd in the description. Just name it "driver"

b/w .. -> between the libvhost and the driver.