ImageSharp(3.1.2) Not Resizing Images loaded from another server(virtual directory) using Umbraco 13.2.0

[umaad]

For images inside project's wwwroot folder ,ImageSharp works fine while changing query parameters.But for images loaded using virtual directory ,from another server does not get processed by ImageSharp.
For example:
https://localhost:44305/media/Min-04.jpg/?quality=50&width=500&format=jpeg
Any work around for this?

I'm rendering virtual images like this,
var virDirImages = new FileServerOptions
{
FileProvider = new PhysicalFileProvider(@"\server1\projectname\wwwroot\media"),
RequestPath = new PathString("/media"),
EnableDirectoryBrowsing = false
};
services.AddImageSharp()
.SetRequestParser()
.Configure(options => options.ProviderRootPath = null)
.AddProvider()
.AddProcessor()
.AddProcessor()
.AddProcessor()
.AddProcessor()
.AddProcessor()
.AddProcessor();

app.UseFileServer(virDirImages);
app.UseImageSharp();
app.UseStaticFiles();