[Hubs][Power BI] CostSummary -> Finops hub connection fails. ftk_LoadData expects DataLake Storage not Blob

[DennisR73]

🐛 Problem

Replacing the StorageUrl parameter in the CostSummary report (v.0.1.1) with the value from the Finops Hub deployment results in error:

An error occurred in the ‘ftk_LoadData’ query. DataSource.Error: AzureDataLakeStorage failed to get contents from 'https://finopshubhqluyac2ikl7s.blob.core.windows.net/ingestion'. Status code: 404, description: 'The specified container does not exist.'.
Details:
    DataSourceKind=AzureDataLakeStorage
    DataSourcePath=https://finopshubhqluyac2ikl7s.blob.core.windows.net/ingestion
    Url=https://finopshubhqluyac2ikl7s.blob.core.windows.net/ingestion

It seems to expect DataLake storage not Blob. Did I miss something in the documentation or should this be the way to change the data source?

👣 Repro steps

1. Open CostSummary.pbix
2. Transform Data
3. Update Storage Url parameter under Finops hubs[4]
4. Try to refresh CostDetails data set

[flanakin]

We definitely expect Azure DataLake Storage Gen2 in Power BI. That said, this can be worked around by switching to the storage connector.

@MSBrett, does Data Factory require ADLS or would blob storage be okay as is?

Did you customize the template to use blob storage? The template should be locked down to only use ADLS today, unless we have a gap.

Out of curiosity, is there a reason you want to use blob storage rather than ADLS? I believe ADLS has perf improvements that will be needed for larger datasets.

[DennisR73]

@flanakin, I dont have a preference for Blob. I just followed the deployment of the FInops hub which creates Blob storage.

[MSBrett]

Blob will work too if you update the connection in the PBI. ADLS was selected because it has some parquet enhancements with respect to server-side filtering of data. Regards, Brett/

…

________________________________ From: Michael Flanakin ***@***.***> Sent: Friday, December 1, 2023 1:01 AM To: microsoft/finops-toolkit ***@***.***> Cc: Mention ***@***.***> Subject: Re: [microsoft/finops-toolkit] [Hubs][Power BI] CostSummary -> Finops hub connection fails. ftk_LoadData expects DataLake Storage not Blob (Issue #472) We definitely expect Azure DataLake Storage Gen2 in Power BI. That said, this can be worked around by switching to the storage connector. @MSBrett<https://github.com/MSBrett>, does Data Factory require ADLS or would blob storage be okay as is? Did you customize the template to use blob storage? The template should be locked down to only use ADLS today, unless we have a gap. Out of curiosity, is there a reason you want to use blob storage rather than ADLS? I believe ADLS has perf improvements that will be needed for larger datasets. — Reply to this email directly, view it on GitHub<#472 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFZLL6FCQBKK5GRTCQDSE4TYHGMFNBFKMF2HI4TJMJ2XIZLTSSBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVI2TAMBSGI2DINZUGCSG4YLNMWUWQYLTL5WGCYTFNSWHG5LCNJSWG5C7OR4XAZNMJFZXG5LFINXW23LFNZ2KM5DPOBUWG44TQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLKJVHA3TSNRVHA3DFAVEOR4XAZNFNFZXG5LFUV3GC3DVMWVDEMBRHA4TCMJTGU4IFJDUPFYGLJLMMFRGK3FFOZQWY5LFVI2TAMBSGI2DINZUGCTXI4TJM5TWK4VGMNZGKYLUMU>. You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[flanakin]

@DennisR73 the template should create an ADLS Gen2 storage account. Can you share a screenshot of the parameters used in the template deployment?

You can convert the storage account to Data Lake Gen2 from the storage account by selecting the Data Lake Gen2 upgrade menu item for the storage account.

[DennisR73]

@flanakin, I re-deployed the test environment and the deployment did create a data lake storage account. I might have specified the blob endpoint instead of the dfs in the data source settings because now all seem to be working fine.

[flanakin]

Let us know if you run into this again. Happy to tweak things if there's a way to avoid issues. I'm also happy to add blob support if there's value in that.

Thanks for circling back! Let us know if you run into any other issues or have any feedback. Would love to know how things are going and how we can improve your experience.