Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document BigQuery RPC settings #24638

Merged
merged 3 commits into from
Jan 13, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 55 additions & 2 deletions docs/src/main/sphinx/connector/bigquery.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ a few caveats:
- The project ID Google Cloud Project to bill for the export.
- Taken from the service account
* - `bigquery.views-enabled`
- Enables the connector to read from views and not only tables. Please read
- Enables the connector to read from views and not only tables. Read
[this section](bigquery-reading-from-views) before enabling this feature.
- `false`
* - `bigquery.view-expire-duration`
Expand Down Expand Up @@ -175,8 +175,61 @@ a few caveats:
- `false`
* - `bigquery.arrow-serialization.enabled`
- Enable using Apache Arrow serialization when reading data from BigQuery.
Please read this [section](bigquery-arrow-serialization-support) before using this feature.
Read this [section](bigquery-arrow-serialization-support) before using this feature.
- `true`
* - `bigquery.channel-pool.initial-size`
- The initial size of the connection pool, also known as a channel pool,
used for gRPC communication.
mosabua marked this conversation as resolved.
Show resolved Hide resolved
- `1`
* - `bigquery.channel-pool.min-size`
- The minimum number of connections in the connection pool, also known as a
channel pool, used for gRPC communication.
- `1`
* - `bigquery.channel-pool.max-size`
- The maximum number of connections in the connection pool, also known as a
channel pool, used for gRPC communication.
- `1`
* - `bigquery.channel-pool.min-rpc-per-channel`
- Threshold to start scaling down the channel pool.
When the average of outstanding RPCs in a single minute drop below this
threshold, channels are removed from the pool.
- `0`
* - `bigquery.channel-pool.max-rpc-per-channel`
- Threshold to start scaling up the channel pool.
When the average of outstanding RPCs in a single minute surpass this
threshold, channels are added to the pool.
- `2147483647`
* - `bigquery.rpc-retries`
- The maximum number of retry attempts to perform for the RPC calls.
If this value is set to `0`, the value from
`bigquery.rpc-timeout` is used.
Retry is deactivated when both `bigquery.rpc-retries` and
`bigquery.rpc-timeout` are `0`.
If this value is positive, and the number of attempts exceeds
`bigquery.rpc-retries` limit, retries stop even if
the total retry time is still lower than `bigquery.rpc-timeout`.
- `0`
* - `bigquery.rpc-timeout`
- Timeout [duration](prop-type-duration) on when the retries for the
RPC call should be given up completely. The higher the timeout, the
more retries can be attempted. If this value is `0s`, then
findinpath marked this conversation as resolved.
Show resolved Hide resolved
`bigquery.rpc-retries` is used to determine retries.
Retry is deactivated when `bigquery.rpc-retries` and
`bigquery.rpc-timeout` are both `0`.
If this value is positive, and the retry duration has reached the timeout
value, retries stop even if the number of attempts is lower than
the `bigquery.rpc-retries` value.
- `0s`
* - `bigquery.rpc-retry-delay`
- The delay [duration](prop-type-duration) before the first retry attempt
for RPC calls.
- `0s`
* - `bigquery.rpc-retry-delay-multiplier`
- Controls the change in delay before the next retry.
The retry delay of the previous call is multiplied by the
`bigquery.rpc-retry-delay-multiplier` to calculate the retry delay
for the next RPC call.
- `1.0`
* - `bigquery.rpc-proxy.enabled`
- Use a proxy for communication with BigQuery.
- `false`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
package io.trino.plugin.bigquery;

import io.airlift.configuration.Config;
import io.airlift.configuration.ConfigHidden;
import io.airlift.units.Duration;
import io.airlift.units.MaxDuration;
import io.airlift.units.MinDuration;
Expand Down Expand Up @@ -44,7 +43,6 @@ public int getRpcInitialChannelCount()
return rpcInitialChannelCount;
}

@ConfigHidden
@Config("bigquery.channel-pool.initial-size")
public BigQueryRpcConfig setRpcInitialChannelCount(int rpcInitialChannelCount)
{
Expand All @@ -59,7 +57,6 @@ public int getRpcMinChannelCount()
return rpcMinChannelCount;
}

@ConfigHidden
@Config("bigquery.channel-pool.min-size")
public BigQueryRpcConfig setRpcMinChannelCount(int rpcMinChannelCount)
{
Expand All @@ -74,7 +71,6 @@ public int getRpcMaxChannelCount()
return rpcMaxChannelCount;
}

@ConfigHidden
@Config("bigquery.channel-pool.max-size")
public BigQueryRpcConfig setRpcMaxChannelCount(int rpcMaxChannelCount)
{
Expand All @@ -88,7 +84,6 @@ public int getMinRpcPerChannel()
return minRpcPerChannel;
}

@ConfigHidden
@Config("bigquery.channel-pool.min-rpc-per-channel")
public BigQueryRpcConfig setMinRpcPerChannel(int minRpcPerChannel)
{
Expand All @@ -102,7 +97,6 @@ public int getMaxRpcPerChannel()
return maxRpcPerChannel;
}

@ConfigHidden
@Config("bigquery.channel-pool.max-rpc-per-channel")
public BigQueryRpcConfig setMaxRpcPerChannel(int maxRpcPerChannel)
{
Expand All @@ -117,7 +111,6 @@ public int getRetries()
return retries;
}

@ConfigHidden
@Config("bigquery.rpc-retries")
public BigQueryRpcConfig setRetries(int maxRetries)
{
Expand All @@ -132,7 +125,6 @@ public Duration getTimeout()
return timeout;
}

@ConfigHidden
@Config("bigquery.rpc-timeout")
public BigQueryRpcConfig setTimeout(Duration timeout)
{
Expand All @@ -147,15 +139,13 @@ public Duration getRetryDelay()
return retryDelay;
}

@ConfigHidden
@Config("bigquery.rpc-retry-delay")
public BigQueryRpcConfig setRetryDelay(Duration retryDelay)
{
this.retryDelay = retryDelay;
return this;
}

@ConfigHidden
@Config("bigquery.rpc-retry-delay-multiplier")
public BigQueryRpcConfig setRetryMultiplier(double retryMultiplier)
{
Expand Down
Loading