Skip to content

Commit

Permalink
Doc: add clarification for cross-region PL (#31018)
Browse files Browse the repository at this point in the history
<!--
Describe the contents of the PR briefly but completely.

If you write detailed commit messages, it is acceptable to copy/paste
them
here, or write "see commit messages for details." If there is only one
commit
in the PR, GitHub will have already added its commit message above.
-->

### Motivation

Depends on: MaterializeInc/cloud#10617 once it
is merged we can update the documentation as well.

The AWS official documentation for releasing cross-region PrivateLink
connections:

>
https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-cross-region-connectivity-for-aws-privatelink/

---------

Co-authored-by: morsapaes <[email protected]>
Co-authored-by: Justin Bradfield <[email protected]>
  • Loading branch information
3 people authored Jan 24, 2025
1 parent f9d3074 commit 8017522
Show file tree
Hide file tree
Showing 14 changed files with 507 additions and 106 deletions.
26 changes: 16 additions & 10 deletions doc/user/content/ingest-data/kafka/amazon-msk.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,24 +32,30 @@ Before you begin, you must have:

## Creating a connection

There are various ways to configure your Kafka network to allow Materialize to connect:
There are various ways to configure your Kafka network to allow Materialize to
connect:

- **Use AWS PrivateLink**: If your Kafka cluster is running on AWS, you can use AWS PrivateLink to connect Materialize to the cluster.
- **Allow Materialize IPs:** If your Kafka cluster is publicly accessible, you
can configure your firewall to allow connections from a set of static
Materialize IP addresses.

- **Use an SSH tunnel**: If your Kafka cluster is running in a private network, you can use an SSH tunnel to connect Materialize to the cluster.
- **Use AWS PrivateLink**: If your Kafka cluster is running in a private network, you
can use [AWS PrivateLink](/ingest-data/network-security/privatelink/) to
connect Materialize to the cluster. For details, see [AWS PrivateLink](/ingest-data/network-security/privatelink/).

- **Allow Materialize IPs**: If your Kafka cluster is publicly accessible, you can configure your firewall to allow connections from a set of static Materialize IP addresses.

Select the option that works best for you.
- **Use an SSH tunnel:** If your Kafka cluster is running in a private network, you
can use an SSH tunnel to connect Materialize to the cluster.

{{< tabs tabID="1" >}}

{{< tab "Privatelink">}}
{{< tab "PrivateLink">}}

{{< note >}}
Materialize provides Terraform modules for both [Amazon MSK clusters](https://github.com/MaterializeInc/terraform-aws-msk-privatelink) and [self-managed Kafka clusters](https://github.com/MaterializeInc/terraform-aws-kafka-privatelink) which can be used to create the target groups for each Kafka broker (step 1), the network load balancer (step 2),
the TCP listeners (step 3) and the VPC endpoint service (step 5).
{{< /note >}}
Materialize provides a Terraform module that automates the creation and
configuration of AWS resources for a PrivateLink connection. For more details,
see the Terraform module repositories for [Amazon MSK](https://github.com/MaterializeInc/terraform-aws-msk-privatelink)
and [self-managed Kafka clusters](https://github.com/MaterializeInc/terraform-aws-kafka-privatelink).
{{</ note >}}

{{% network-security/privatelink-kafka %}}

Expand Down
17 changes: 9 additions & 8 deletions doc/user/content/ingest-data/mysql/amazon-aurora.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,15 +64,12 @@ to connect:
configure your database's security group to allow connections from a set of
static Materialize IP addresses.

- **Use an SSH tunnel:** If your database is running in a private network, you
can use an SSH tunnel to connect Materialize to the database.

- **Use AWS PrivateLink**: If your database is running in a private network, you
can use [AWS PrivateLink](/ingest-data/network-security/privatelink/) to
connect Materialize to the database. For details, see [AWS
PrivateLink](/ingest-data/network-security/privatelink/).
connect Materialize to the database. For details, see [AWS PrivateLink](/ingest-data/network-security/privatelink/).

Select the option that works best for you.
- **Use an SSH tunnel:** If your database is running in a private network, you
can use an SSH tunnel to connect Materialize to the database.

{{< tabs >}}

Expand Down Expand Up @@ -131,7 +128,7 @@ see the [Terraform module repository](https://github.com/MaterializeInc/terrafor
- Choose the **IP addresses** type.
- Set the protocol and port to **TCP** and **5432**.
- Set the protocol and port to **TCP** and **3306**.
- Choose the same VPC as your RDS instance.
Expand All @@ -154,7 +151,7 @@ see the [Terraform module repository](https://github.com/MaterializeInc/terrafor
in.
- For **Listeners and routing**, set the protocol and port to **TCP**
and **5432** and select the target group you created in the previous
and **3306** and select the target group you created in the previous
step.
1. In the security group of your Aurora instance, [allow traffic from the the
Expand Down Expand Up @@ -276,6 +273,10 @@ networking configuration, so start by selecting the relevant option.
{{% mysql-direct/ingesting-data/allow-materialize-ips %}}
{{< /tab >}}
{{< tab "Use AWS PrivateLink">}}
{{% mysql-direct/ingesting-data/use-aws-privatelink %}}
{{< /tab >}}
{{< tab "Use an SSH tunnel">}}
{{% mysql-direct/ingesting-data/use-ssh-tunnel %}}
{{< /tab >}}
Expand Down
17 changes: 10 additions & 7 deletions doc/user/content/ingest-data/mysql/amazon-rds.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,13 +118,12 @@ to connect:
configure your database's security group to allow connections from a set of
static Materialize IP addresses.

- **Use an SSH tunnel:** If your database is running in a private network, you
can use an SSH tunnel to connect Materialize to the database.

- **Use AWS PrivateLink**: If your database is running in a private network, you
can use [AWS PrivateLink](/ingest-data/network-security/privatelink/) to
connect Materialize to the database. For details, see [AWS
PrivateLink](/ingest-data/network-security/privatelink/).
connect Materialize to the database. For details, see [AWS PrivateLink](/ingest-data/network-security/privatelink/).

- **Use an SSH tunnel:** If your database is running in a private network, you
can use an SSH tunnel to connect Materialize to the database.

{{< tabs >}}

Expand Down Expand Up @@ -184,7 +183,7 @@ see the [Terraform module repository](https://github.com/MaterializeInc/terrafor
- Choose the **IP addresses** type.
- Set the protocol and port to **TCP** and **5432**.
- Set the protocol and port to **TCP** and **3306**.
- Choose the same VPC as your RDS instance.
Expand All @@ -207,7 +206,7 @@ see the [Terraform module repository](https://github.com/MaterializeInc/terrafor
in.
- For **Listeners and routing**, set the protocol and port to **TCP**
and **5432** and select the target group you created in the previous
and **3306** and select the target group you created in the previous
step.
1. In the security group of your RDS instance, [allow traffic from the network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-register-targets.html).
Expand Down Expand Up @@ -327,6 +326,10 @@ start by selecting the relevant option.
{{% mysql-direct/ingesting-data/allow-materialize-ips %}}
{{< /tab >}}
{{< tab "Use AWS PrivateLink">}}
{{% mysql-direct/ingesting-data/use-aws-privatelink %}}
{{< /tab >}}
{{< tab "Use an SSH tunnel">}}
{{% mysql-direct/ingesting-data/use-ssh-tunnel %}}
{{< /tab >}}
Expand Down
37 changes: 28 additions & 9 deletions doc/user/content/ingest-data/network-security/privatelink.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
title: "AWS PrivateLink connections"
description: "How to connect Materialize to a Kafka broker, or a PostgreSQL database using an AWS PrivateLink connection"
description: "How to connect Materialize to a Kafka broker, a Confluent Schema Registry server, a
PostgreSQL database, or a MySQL database through an AWS PrivateLink service."
aliases:
- /ops/network-security/privatelink/
- /connect-sources/privatelink/
Expand All @@ -10,35 +11,53 @@ menu:
name: "AWS PrivateLink connections"
---

[//]: # "TODO(morsapaes) Add shortcode with instructions for AWS RDS MySQL"

Materialize can connect to a Kafka broker, a Confluent Schema Registry server, a
PostgreSQL database, or a MySQL database through an [AWS PrivateLink](https://aws.amazon.com/privatelink/)
service.

In this guide, we'll cover how to create `AWS PRIVATELINK` connections
and retrieve the AWS principal needed to configure the AWS PrivateLink service.
In this guide, we'll cover how to create `AWS PRIVATELINK` connections and
retrieve the AWS principal needed to configure the AWS PrivateLink service.

## Create an AWS PrivateLink connection

{{< tabs tabID="1" >}}
{{< tab "Kafka on AWS">}}

{{< note >}}
Materialize provides Terraform modules for both [MSK cluster](https://github.com/MaterializeInc/terraform-aws-msk-privatelink) and [self-managed Kafka clusters](https://github.com/MaterializeInc/terraform-aws-kafka-privatelink) which can be used to create the target groups for each Kafka broker (step 1), the network load balancer (step 2),
the TCP listeners (step 3) and the VPC endpoint service (step 5).
{{< /note >}}
Materialize provides a Terraform module that automates the creation and
configuration of AWS resources for a PrivateLink connection. For more details,
see the Terraform module repositories for [Amazon MSK](https://github.com/MaterializeInc/terraform-aws-msk-privatelink)
and [self-managed Kafka clusters](https://github.com/MaterializeInc/terraform-aws-kafka-privatelink).
{{</ note >}}

{{% network-security/privatelink-kafka %}}

{{< /tab >}}

{{< tab "AWS RDS">}}
{{< tab "PostgreSQL on AWS">}}

{{< note >}}
Materialize provides a Terraform module that automates the creation and
configuration of AWS resources for a PrivateLink connection. For more details,
see the [Terraform module repository](https://github.com/MaterializeInc/terraform-aws-rds-privatelink).
{{</ note >}}

{{% network-security/privatelink-postgres %}}

{{< /tab >}}

{{< tab "MySQL on AWS">}}

{{< note >}}
Materialize provides a Terraform module that automates the creation and
configuration of AWS resources for a PrivateLink connection. For more details,
see the [Terraform module repository](https://github.com/MaterializeInc/terraform-aws-rds-privatelink).
{{</ note >}}

{{% network-security/privatelink-mysql %}}

{{< /tab >}}

{{< /tabs >}}

## Related pages
Expand Down
19 changes: 10 additions & 9 deletions doc/user/content/ingest-data/postgres/amazon-aurora.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ menu:
---

This page shows you how to stream data from [Amazon Aurora for PostgreSQL](https://aws.amazon.com/rds/aurora/)
to Materialize using the[PostgreSQL source](/sql/create-source/postgres/).
to Materialize using the [PostgreSQL source](/sql/create-source/postgres/).

{{< tip >}}
{{< guided-tour-blurb-for-ingest-data >}}
Expand Down Expand Up @@ -56,17 +56,18 @@ to connect:
configure your database's security group to allow connections from a set of
static Materialize IP addresses.

- **Use AWS PrivateLink** or **Use an SSH tunnel:** If your database is running
in a private network, you can use either [AWS PrivateLink](https://aws.amazon.com/privatelink/)
or an SSH tunnel to connect Materialize to the database.
- **Use AWS PrivateLink**: If your database is running in a private network, you
can use [AWS PrivateLink](/ingest-data/network-security/privatelink/) to
connect Materialize to the database. For details, see [AWS PrivateLink](/ingest-data/network-security/privatelink/).

Select the option that works best for you.
- **Use an SSH tunnel:** If your database is running in a private network, you
can use an SSH tunnel to connect Materialize to the database.

{{< tabs >}}

{{< tab "Allow Materialize IPs">}}

1. In the [Materialize console's SQL Shell](https://console.materialize.com/) or your preferred SQL
1. In the [SQL Shell](https://console.materialize.com/) or your preferred SQL
client connected to Materialize, find the static egress IP addresses for the
Materialize region you are running in:

Expand Down Expand Up @@ -323,15 +324,15 @@ start by selecting the relevant option.
{{< tab "Use AWS PrivateLink">}}
1. In the [Materialize console's SQL Shell](https://console.materialize.com/),
1. In the [SQL Shell](https://console.materialize.com/),
or your preferred SQL client connected to Materialize, use the [`CREATE
CONNECTION`](/sql/create-connection/#aws-privatelink) command to create an
AWS PrivateLink connection:
```mzsql
CREATE CONNECTION privatelink_svc TO AWS PRIVATELINK (
SERVICE NAME 'com.amazonaws.vpce.us-east-1.vpce-svc-0356210a8a432d9e9',
AVAILABILITY ZONES ('use1-az1', 'use1-az2', 'use1-az3')
AVAILABILITY ZONES ('use1-az1', 'use1-az2', 'use1-az4')
);
```
Expand Down Expand Up @@ -428,7 +429,7 @@ details for Materialize to use:
{{< tab "Use an SSH tunnel">}}
1. In the [Materialize console's SQL Shell](https://console.materialize.com/),
1. In the [SQL Shell](https://console.materialize.com/),
or your preferred SQL client connected to Materialize, use the [`CREATE
CONNECTION`](/sql/create-connection/#ssh-tunnel) command to create an SSH
tunnel connection:
Expand Down
65 changes: 46 additions & 19 deletions doc/user/content/ingest-data/postgres/amazon-rds.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,19 +113,20 @@ to connect:
configure your database's security group to allow connections from a set of
static Materialize IP addresses.
- **Use AWS PrivateLink** or **Use an SSH tunnel:** If your database is running
in a private network, you can use either [AWS PrivateLink](https://aws.amazon.com/privatelink/)
or an SSH tunnel to connect Materialize to the database.
- **Use AWS PrivateLink**: If your database is running in a private network, you
can use [AWS PrivateLink](/ingest-data/network-security/privatelink/) to
connect Materialize to the database. For details, see [AWS PrivateLink](/ingest-data/network-security/privatelink/).
Select the option that works best for you.
- **Use an SSH tunnel:** If your database is running in a private network, you
can use an SSH tunnel to connect Materialize to the database.
{{< tabs >}}
{{< tab "Allow Materialize IPs">}}
1. In the [Materialize console's SQL Shell](https://console.materialize.com/),
or your preferred SQL client connected to Materialize, find the static egress
IP addresses for the Materialize region you are running in:
1. In the [SQL Shell](https://console.materialize.com/), or your preferred SQL
client connected to Materialize, find the static egress IP addresses for the
Materialize region you are running in:
```mzsql
SELECT * FROM mz_egress_ips;
Expand Down Expand Up @@ -316,7 +317,7 @@ start by selecting the relevant option.
{{< tab "Allow Materialize IPs">}}
1. In [Materialize console's SQL Shell](https://console.materialize.com/), or
1. In the [SQL Shell](https://console.materialize.com/), or
your preferred SQL client connected to Materialize, use the [`CREATE
SECRET`](/sql/create-secret/) command to securely store the password for the
`materialize` PostgreSQL user you created
Expand Down Expand Up @@ -373,28 +374,54 @@ start by selecting the relevant option.
{{< tab "Use AWS PrivateLink">}}
1. In the [Materialize console's SQL Shell](https://console.materialize.com/),
or your preferred SQL client connected to Materialize, use the [`CREATE
CONNECTION`](/sql/create-connection/#aws-privatelink) command to create an
AWS PrivateLink connection:
1. In the [SQL Shell](https://console.materialize.com/), or your preferred SQL
client connected to Materialize, use the [`CREATE CONNECTION`](/sql/create-connection/#aws-privatelink)
command to create an **in-region** or **cross-region** AWS PrivateLink
connection.
```mzsql
CREATE CONNECTION privatelink_svc TO AWS PRIVATELINK (
SERVICE NAME 'com.amazonaws.vpce.us-east-1.vpce-svc-0356210a8a432d9e9',
AVAILABILITY ZONES ('use1-az1', 'use1-az2', 'use1-az3')
);
```
**In-region connections**
To connect to an AWS PrivateLink endpoint service in the **same region** as your
Materialize environment:
```mzsql
CREATE CONNECTION privatelink_svc TO AWS PRIVATELINK (
SERVICE NAME 'com.amazonaws.vpce.<region_id>.vpce-svc-<endpoint_service_id>',
AVAILABILITY ZONES ('use1-az1', 'use1-az2', 'use1-az4')
);
```
- Replace the `SERVICE NAME` value with the service name you noted [earlier](#b-optional-configure-network-security).
- Replace the `AVAILABILITY ZONES` list with the IDs of the availability
zones in your AWS account.
zones in your AWS account. For in-region connections the availability
zones of the NLB and the consumer VPC **must match**.
To find your availability zone IDs, select your database in the RDS
Console and click the subnets under **Connectivity & security**. For each
subnet, look for **Availability Zone ID** (e.g., `use1-az6`),
not **Availability Zone** (e.g., `us-east-1d`).
**Cross-region connections**
To connect to an AWS PrivateLink endpoint service in a **different region** to
the one where your Materialize environment is deployed:
```mzsql
CREATE CONNECTION privatelink_svc TO AWS PRIVATELINK (
SERVICE NAME 'com.amazonaws.vpce.us-west-1.vpce-svc-<endpoint_service_id>',
-- For now, the AVAILABILITY ZONES clause **is** required, but will be
-- made optional in a future release.
AVAILABILITY ZONES ()
);
```
- Replace the `SERVICE NAME` value with the service name you noted [earlier](#b-optional-configure-network-security).
- The service name region refers to where the endpoint service was created.
You **do not need** to specify `AVAILABILITY ZONES` manually — these will
be optimally auto-assigned when none are provided.
1. Retrieve the AWS principal for the AWS PrivateLink connection you just
created:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
- Replace `<host>` with your MySQL endpoint.

1. Use the [`CREATE SOURCE`](/sql/create-source/) command to connect Materialize
to your Azure instance and start ingesting data:
to your MySQL instance and start ingesting data:

```mzsql
CREATE SOURCE mz_source
Expand Down
Loading

0 comments on commit 8017522

Please sign in to comment.