#180 fix for destinationTable not returned

[prabhav007]

closes #180

[yaronneuman]

why this is needed?

[kromiii]

datasetId must be unique although tableId does not need to. That's why we needed.

reference:
https://cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Project#Google__Cloud__Bigquery__Project_create_dataset_instance_
https://cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Dataset#Google__Cloud__Bigquery__Dataset_create_table_instance_

[goccy]

I think the jobID is already unique for each request. So if you just want to make datasetID unique, jobID seems fine.

[kromiii]

It seems fine, but actually, after deleting this ds_ I got an error below when using bigquery ruby client.
Do you know what causes this erorr? @goccy

2023-09-21 21:49:45 bq-emulator-test-bq-1  | 2023-09-21T12:49:45.555Z   ERROR   server/handler.go:2355   internalError   {"error": "internalError: failed to query SELECT * FROM `job_JSsjGMduSIBf5L_poPWkkL2vZdkd`: no such table: test_job_JSsjGMduSIBf5L_poPWkkL2vZdkd"}

[goccy]

@kromiii Sorry, I don't known by this information. I'd like to have reproducible test code. Go is preferred, but other languages are also acceptable. Also, please attach the logs obtained at that time.

[kromiii]

I created a repo to test this. https://github.com/kromiii/bq-emulator-test
This issue depends on the ruby client, so I attached a simple ruby script for testing bigquery-emulator.

[kromiii]

@prabhav007 Let us know the reason of ds_ if you are available.

[prabhav007]

In case of jobs with anonymous tables, when the dataset name and the table name are the same, an error response is received. When they are different, it works fine. PFB the error response I got locally.

http://localhost:9050/bigquery/v2/projects/demo-0d168064/datasets/0ffd9b42-7541-4317-ad16-fad5ec3beb55/tables/0ffd9b42-7541-4317-ad16-fad5ec3beb55/data?maxResults=10&prettyPrint=false

{"error":{"errors":[{"reason":"internalError","location":"","debugInfo":"","message":"failed to query SELECT * FROM 0ffd9b42-7541-4317-ad16-fad5ec3beb55: no such table: demo-0d168064_0ffd9b42-7541-4317-ad16-fad5ec3beb55"}],"code":500,"message":"failed to query SELECT * FROM 0ffd9b42-7541-4317-ad16-fad5ec3beb55: no such table: demo-0d168064_0ffd9b42-7541-4317-ad16-fad5ec3beb55"}}

[kromiii]

@prabhav007 Thank you. I could reproduce the error. However, the quetion is that it works well in the python clinent even if the dataset name and the table name is the same. What makes this difference?

[prabhav007]

I am not sure, I was using the java client for spring.

[pyama86]

I have the same problem. How about I take over if there is no progress?

[kromiii]

I applied these changes to my branch, and it resolved the error.
These changes would be necessary for using ruby-client for the bigquery-emulator.

[anderseriksson]

I applied these changes to my branch, and it resolved the error. These changes would be necessary for using ruby-client for the bigquery-emulator.

Yes your changes and your image solved a similar problem when I issue a query with a specific JobId. I hope we can have this fix merged in to goccy project and image.

[benvdh-incentro]

@yaronneuman it seems your question has been answered 3 weeks ago by someone else. Also others mention the code works well... is there anything that's still keeping you from getting this merged? Getting this fixed can also help to resolve an issue being the discussed in the DBT community:

dbt-labs/dbt-bigquery#358

@goccy can you help perhaps?

[goccy]

@benvdh-incentro
Thank you, I've commented https://github.com/goccy/bigquery-emulator/pull/187/files#r1332797665

[benvdh-incentro]

@goccy I see @prabhav007 and @kromiii have delivered some test code and further rationale behind the current PR, anything else that is still needed to get this merged?

[goccy]

The reason I cannot merge this PR is that there is no test code in this repository that corresponds to the modification. If there is no test code, I need to determine the validity of the modification from the diffs, which is currently difficult.

[simi]

I can confirm this breaks Ruby client. 🤔 Any idea how to move this forward? Even with this patch only query like SELECT * FROM finance_v2_local.statements_2 works, but query like SELECT * FROM finance_v2_local.statements_2 WHERE Artist = 'ABC' fails since missing destination dataset table. It seems to be related to fact if query returns any value. For queries with 0 returned rows it fails per

bigquery-emulator/server/handler.go

Line 1427 in 316038b

} else if response.TotalRows > 0 {

.

I have checked BQ response and it returns destination table even for empty result.

[simi]

@kromiii I can try to help with writing test, but seems destination table is not tested currently at all.

[System-Glitch]

Any update on this? The Go client doesn't work because the job status destination table is not populated.

[turb]

FYI this also seems to break Scala client on this line.

[turb]

If this cannot be merged soon, any chance you can rebase your branch and release a fresh version @kromiii?

[kromiii]

If this cannot be merged soon, any chance you can rebase your branch and release a fresh version @kromiii?

Ok, I rebuild the package applying this patch. @turb
https://github.com/kromiii/bigquery-emulator/pkgs/container/bigquery-emulator

I hope it will help.