[Bug]: SDK 2.59.0 Java SpannerIO ChangeStream Can't Use the same metadata database for multiple pipelines

[bangau1]

What happened?

With the latest apache beam sdk 2.59.0, the SpannerIO changestream's metadata table is adding the indexes: https://github.com/apache/beam/blame/160dffd88e5a60077a48e3e2f8fff331aecced08/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataAdminDao.java#L82-L86

Since the indexes' name are hardcoded to WatermarkIndex and CreatedAtStartTimestampIndex, this poses problem when there are multiple pipelines that are using the same metadata database but using different metadata tablename, since the index name must be unique within the same database.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[bangau1]

After upgraded to dataflow SDK 2.62.0, the behavior is now changed. It assigns UUID to the index. However it's still using the database name as base name. #32689

Question is: why don't we use the metadata table name as the base name instead? So we can let user choose their own metadata table name as the unit of job. @thiagotnunes