Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quarkus azure eventhubs Extension #310

Merged
merged 40 commits into from
Dec 25, 2024
Merged
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
77c891c
add extension: azure-eventhubs
backwind1233 Dec 4, 2024
6b848c1
Add Azure Eventhubs config and improve EventhubsClientProducer format…
backwind1233 Dec 4, 2024
3563447
Fix inconsistent indentation in EventhubsClientProducer.java file.
backwind1233 Dec 4, 2024
36bd9b5
Add Azure EventHubs integration tests and update documentation.
backwind1233 Dec 10, 2024
73e2b29
Add event consumption feature and update tests in EventhubsResource.
backwind1233 Dec 10, 2024
7386b6b
Add Azure Blob Checkpoint Store dependency to pom.xml.
backwind1233 Dec 10, 2024
29a1dbb
Add runtime initialization for TcpClientSecure in EventhubsProcessor.
backwind1233 Dec 10, 2024
e017dd8
Add .gitattributes to enforce LF line endings for .adoc files.
backwind1233 Dec 10, 2024
947be7c
Add integration tests for Event Hubs async operations and logging.
backwind1233 Dec 12, 2024
8ba91cf
Rename fullyQualifiedNamespace to namespace and update related code.
backwind1233 Dec 12, 2024
d16144d
Add Azure Event Hubs creation and deletion commands to scripts.
backwind1233 Dec 13, 2024
1906926
Refactor event publishing to use SendOptions for improved reliability.
backwind1233 Dec 13, 2024
3ee1a31
Add partition count to Azure Event Hub resource creation script.
backwind1233 Dec 13, 2024
7347093
Refactor EventHubs to use simple namespace in docs and code.
backwind1233 Dec 16, 2024
78d26f9
Correct variable names for Event Hub configuration consistency.
backwind1233 Dec 16, 2024
87d5314
Update README to reflect Azure Eventhubs integration instead of CosmosDB
backwind1233 Dec 16, 2024
0918294
Update README to reflect Azure Eventhub setup
backwind1233 Dec 16, 2024
f5a98a3
Merge branch 'main' into quarkus-azure-eventhub
backwind1233 Dec 17, 2024
3ea050e
Merge branch 'main' into quarkus-azure-eventhub
backwind1233 Dec 17, 2024
6280529
Remove unused import and reorder remaining imports for clarity.
backwind1233 Dec 17, 2024
aa4a431
Reorder import statements for consistency across files.
backwind1233 Dec 17, 2024
6bf3af5
Update assignee-object-id with correct object ID in Azure role assign…
backwind1233 Dec 17, 2024
f67cbae
Add config check before creating EventHub clients.
backwind1233 Dec 17, 2024
4406ffc
Update namespace property and role assignment scope for Event Hubs
backwind1233 Dec 17, 2024
0fe2fe0
Rename workflow to include integration tests in build process
backwind1233 Dec 17, 2024
5f799f0
Add Quarkus Azure Event Hubs extension documentation and script adjus…
backwind1233 Dec 17, 2024
0d5a0b0
Increase sleep duration for event sending and remove producer closure.
backwind1233 Dec 18, 2024
15cf5b4
Update documentation to include example resource for Event Hubs.
backwind1233 Dec 18, 2024
7c50955
Update Azure Event Hubs documentation to include async client support.
backwind1233 Dec 18, 2024
e3fdbf8
Update build.yml to include hyphens in resource naming conventions.
backwind1233 Dec 18, 2024
9f99e05
Update storage account name format in build workflow configuration.
backwind1233 Dec 18, 2024
488735b
Add newline at the end of the create-azure-resources.sh file.
backwind1233 Dec 20, 2024
c628c7b
Fix missing newline at end of file in pom.xml for Azure Event Hubs.
backwind1233 Dec 20, 2024
098ae92
Update API endpoints from publish/consume to send/receive for clarity.
backwind1233 Dec 20, 2024
5d87f2f
Corrects "Eventhubs" to "Event Hubs" across multiple documentation fi…
backwind1233 Dec 20, 2024
a74837b
Replace SLF4J Logger with JBoss logging in Eventhubs resources.
backwind1233 Dec 23, 2024
2051cf1
Update Azure Event Hubs documentation and configuration properties.
backwind1233 Dec 24, 2024
1c959e9
Rename event hub variables for consistency across documentation and s…
backwind1233 Dec 24, 2024
023c4b3
Add Quarkus Azure Event Hubs Extension information to the README.
backwind1233 Dec 24, 2024
780cdb4
Update event hub naming conventions for consistency across files.
backwind1233 Dec 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add Azure EventHubs integration tests and update documentation.
backwind1233 committed Dec 10, 2024
commit 36bd9b57bed97f736ad398531d131d9dd60e46f9
10 changes: 10 additions & 0 deletions bom/pom.xml
Original file line number Diff line number Diff line change
@@ -135,6 +135,16 @@
<artifactId>quarkus-azure-cosmos-deployment</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>io.quarkiverse.azureservices</groupId>
<artifactId>quarkus-azure-eventhubs</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>io.quarkiverse.azureservices</groupId>
<artifactId>quarkus-azure-eventhubs-deployment</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>io.quarkiverse.azureservices</groupId>
<artifactId>quarkus-azure-keyvault</artifactId>
17 changes: 0 additions & 17 deletions docs/modules/ROOT/pages/includes/quarkus-azure-eventhubs.adoc
Original file line number Diff line number Diff line change
@@ -24,23 +24,6 @@ endif::add-copy-button-to-env-var[]
|boolean
|`true`

a| [[quarkus-azure-eventhubs_quarkus-azure-eventhubs-endpoint]] [.property-path]##link:#quarkus-azure-eventhubs_quarkus-azure-eventhubs-endpoint[`quarkus.azure.eventhubs.endpoint`]##

[.description]
--
The endpoint of Azure Eventhubs. Required if quarkus.azure.eventhubs.enabled is set to true


ifdef::add-copy-button-to-env-var[]
Environment variable: env_var_with_copy_button:+++QUARKUS_AZURE_EVENTHUBS_ENDPOINT+++[]
endif::add-copy-button-to-env-var[]
ifndef::add-copy-button-to-env-var[]
Environment variable: `+++QUARKUS_AZURE_EVENTHUBS_ENDPOINT+++`
endif::add-copy-button-to-env-var[]
--
|string
|

a| [[quarkus-azure-eventhubs_quarkus-azure-eventhubs-fully-qualified-namespace]] [.property-path]##link:#quarkus-azure-eventhubs_quarkus-azure-eventhubs-fully-qualified-namespace[`quarkus.azure.eventhubs.fully-qualified-namespace`]##

[.description]
Original file line number Diff line number Diff line change
@@ -24,23 +24,6 @@ endif::add-copy-button-to-env-var[]
|boolean
|`true`

a| [[quarkus-azure-eventhubs_quarkus-azure-eventhubs-endpoint]] [.property-path]##link:#quarkus-azure-eventhubs_quarkus-azure-eventhubs-endpoint[`quarkus.azure.eventhubs.endpoint`]##

[.description]
--
The endpoint of Azure Eventhubs. Required if quarkus.azure.eventhubs.enabled is set to true


ifdef::add-copy-button-to-env-var[]
Environment variable: env_var_with_copy_button:+++QUARKUS_AZURE_EVENTHUBS_ENDPOINT+++[]
endif::add-copy-button-to-env-var[]
ifndef::add-copy-button-to-env-var[]
Environment variable: `+++QUARKUS_AZURE_EVENTHUBS_ENDPOINT+++`
endif::add-copy-button-to-env-var[]
--
|string
|

a| [[quarkus-azure-eventhubs_quarkus-azure-eventhubs-fully-qualified-namespace]] [.property-path]##link:#quarkus-azure-eventhubs_quarkus-azure-eventhubs-fully-qualified-namespace[`quarkus.azure.eventhubs.fully-qualified-namespace`]##

[.description]
Original file line number Diff line number Diff line change
@@ -35,7 +35,7 @@ public EventHubConsumerClient createEventHubConsumerClient() {

return null == builder ? null
: builder.consumerGroup(EventHubClientBuilder.DEFAULT_CONSUMER_GROUP_NAME)
.buildConsumerClient();
.buildConsumerClient();
}

@Produces
@@ -44,15 +44,16 @@ public EventHubConsumerAsyncClient createEventhubClient() {

return null == builder ? null
: builder.consumerGroup(EventHubClientBuilder.DEFAULT_CONSUMER_GROUP_NAME)
.buildAsyncConsumerClient();
.buildAsyncConsumerClient();
}

private EventHubClientBuilder getBuilder() {
if (!eventhubsConfiguration.enabled()) {
return null;
}

assert eventhubsConfiguration.endpoint().isPresent() : "The endpoint of Azure Eventhubs must be set";
// @TOOD

EventHubClientBuilder builder = new EventHubClientBuilder()
.credential(eventhubsConfiguration.fullyQualifiedNamespace(),
eventhubsConfiguration.eventHubName(),
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
package io.quarkiverse.azure.eventhubs.runtime;

import java.util.Optional;

import io.quarkus.runtime.annotations.ConfigPhase;
import io.quarkus.runtime.annotations.ConfigRoot;
import io.smallrye.config.ConfigMapping;
@@ -17,11 +15,6 @@ public interface EventhubsConfig {
@WithDefault("true")
boolean enabled();

/**
* The endpoint of Azure Eventhubs. Required if quarkus.azure.eventhubs.enabled is set to true
*/
Optional<String> endpoint();

/**
* The fully qualified namespace of Azure Eventhubs. Required if quarkus.azure.eventhubs.enabled is set to true
*/
224 changes: 224 additions & 0 deletions integration-tests/azure-eventhubs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,224 @@
# Azure Eventhubs sample

This is a sample about implementing REST endpoints using the Quarkus extension to read/write data stored in Azure Cosmos DB. Though the sample uses a relational style usage pattern, the full functionality of Cosmos DB is enabled by the Quarkus extension.

## Prerequisites

To successfully run this sample, you need:

* JDK 17+ installed with JAVA_HOME configured appropriately
* Apache Maven 3.8.6+
* Azure CLI and Azure subscription
* Docker if you want to build the app as a native executable

You also need to clone the repository and switch to the directory of the sample.

```
git clone https://github.com/quarkiverse/quarkus-azure-services.git
cd quarkus-azure-services/integration-tests/azure-cosmos
```

### Use development iteration version

By default, the sample is kept in sync with the development iteration
version, which is `999-SNAPSHOT`. To install the development iteration
version, build it locally.

```
mvn clean install -DskipTests --file ../../pom.xml
```

### Use release version

If you want to use the release version, you need to update the version of dependencies in the `pom.xml` file.

First, you need to find out the latest release version of the Quarkus Azure services extensions
from [releases](https://github.com/quarkiverse/quarkus-azure-services/releases), for example, `1.0.7`.

Then, update the version of dependencies in the `pom.xml` file, for example:

```xml
<parent>
<groupId>io.quarkiverse.azureservices</groupId>
<artifactId>quarkus-azure-services-parent</artifactId>
<version>1.0.7</version>
<relativePath></relativePath>
</parent>
```

## Preparing the Azure services

You need to create an Azure Cosmos DB account before running the sample application.

### Logging into Azure

Log into Azure and create a resource group for hosting the Azure Cosmos DB account to be created.

```
az login

RESOURCE_GROUP_NAME=<resource-group-name>
az group create \
--name ${RESOURCE_GROUP_NAME} \
--location westus
```

### Creating Azure Azure Cosmos DB account

Run the following commands to create an Azure Cosmos DB account, and export its endpoint as an environment variable.

```
COSMOSDB_ACCOUNT_NAME=<unique-cosmosdb-account-name>
az cosmosdb create \
-n ${COSMOSDB_ACCOUNT_NAME} \
-g ${RESOURCE_GROUP_NAME} \
--default-consistency-level Session \
--locations regionName='West US' failoverPriority=0 isZoneRedundant=False

export QUARKUS_AZURE_COSMOS_ENDPOINT=$(az cosmosdb show \
-n ${COSMOSDB_ACCOUNT_NAME} \
-g ${RESOURCE_GROUP_NAME} \
--query documentEndpoint -o tsv)
echo "The value of 'quarkus.azure.cosmos.endpoint' is: ${QUARKUS_AZURE_COSMOS_ENDPOINT}"
```

The value of environment variable `QUARKUS_AZURE_COSMOS_ENDPOINT` will be read by Quarkus as the value of config
property `quarkus.azure.cosmos.endpoint` of `azure-cosmos` extension in order to set up the
connection to the Azure Cosmos DB.

Assign the `Cosmos DB Built-in Data Contributor` role to the signed-in user as a Microsoft Entra identity, so that the sample application can do data plane CRUD operations.

```
az ad signed-in-user show --query id -o tsv \
| az cosmosdb sql role assignment create \
--account-name ${COSMOSDB_ACCOUNT_NAME} \
--resource-group ${RESOURCE_GROUP_NAME} \
--scope "/" \
--principal-id @- \
--role-definition-id 00000000-0000-0000-0000-000000000002
```

You cannot use any Azure Cosmos DB data plane SDK to authenticate management operations with a Microsoft Entra identity, so you need to create database and container manually.
The following commands create a database `demodb` and a container `democontainer` using Azure CLI.

```
az cosmosdb sql database create \
-a ${COSMOSDB_ACCOUNT_NAME} \
-g ${RESOURCE_GROUP_NAME} \
-n demodb
az cosmosdb sql container create \
-a ${COSMOSDB_ACCOUNT_NAME} \
-g ${RESOURCE_GROUP_NAME} \
-d demodb \
-n democontainer \
-p "/id"
```

## Running the sample

You have different choices to run the sample. Make sure you have followed [Preparing the Azure services](#preparing-the-azure-services) to create the required Azure services. Select an option and proceed to [Testing the sample](#testing-the-sample). For any choice, make sure the environment variable `QUARKUS_AZURE_COSMOS_ENDPOINT` is defined correctly in the environment before starting Quarkus.

### Running the sample in development mode

First, you can launch the sample in `dev` mode.

```
mvn quarkus:dev
```

### Running and test the sample in JVM mode

You can also run the sample in JVM mode.

```
# Build the package.
mvn package

# Run the generated jar file.
java -jar ./target/quarkus-app/quarkus-run.jar
```

### Running and test the sample as a native executable

You can even run the sample as a native executable. Make sure you have installed Docker.

```
# Build the native executable using the Docker.
mvn package -Dnative -Dquarkus.native.container-build

# Run the native executable.
version=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
./target/quarkus-azure-integration-test-cosmos-${version}-runner
```

## Testing the sample

Open a new terminal and run the following commands to test the sample:

```
# Create an item {"id": "1", "name": "dog"} in Azure Cosmos DB database demodb and container democontainer.
curl http://localhost:8080/quarkus-azure-cosmos/demodb/democontainer -X POST -d '{"id": "1", "name": "dog"}' -H "Content-Type: application/json"

# Read the item from Azure Cosmos DB database demodb and container democontainer. You should see {"id":"1","name":"dog"} in the response.
curl http://localhost:8080/quarkus-azure-cosmos/demodb/democontainer/1 -X GET

# List items from Azure Cosmos DB database demodb and container democontainer. You should see [{"id":"1","name":"dog"}] in the response.
curl http://localhost:8080/quarkus-azure-cosmos/demodb/democontainer -X GET

# Update the item {"id": "1", "name": "dog"} to {"id": "1", "name": "cat"} in Azure Cosmos DB database demodb and container democontainer.
curl http://localhost:8080/quarkus-azure-cosmos/demodb/democontainer -X POST -d '{"id": "1", "name": "cat"}' -H "Content-Type: application/json"

# Read the updated item from Azure Cosmos DB database demodb and container democontainer. You should see {"id":"1","name":"cat"} in the response.
curl http://localhost:8080/quarkus-azure-cosmos/demodb/democontainer/1 -X GET

# List items again from Azure Cosmos DB database demodb and container democontainer. You should see [{"id":"1","name":"cat"}] in the response.
curl http://localhost:8080/quarkus-azure-cosmos/demodb/democontainer -X GET

# Delete the item from Azure Cosmos DB database demodb and container democontainer.
curl http://localhost:8080/quarkus-azure-cosmos/demodb/democontainer/1 -X DELETE

# Read the deleted item from Azure Cosmos DB database demodb and container democontainer. You should see HTTP status code 500 in the response.
curl http://localhost:8080/quarkus-azure-cosmos/demodb/democontainer/1 -X DELETE -w "%{http_code}" -s -o /dev/null

# List items again from Azure Cosmos DB database demodb and container democontainer. You should see [] in the response.
curl http://localhost:8080/quarkus-azure-cosmos/demodb/democontainer -X GET

# Do the same operations, but with the async API. Create an item {"id": "1", "name": "dog"} in Azure Cosmos DB database demodb and container democontainer using the async API.
curl http://localhost:8080/quarkus-azure-cosmos-async/demodb/democontainer -X POST -d '{"id": "1", "name": "dog"}' -H "Content-Type: application/json"

# Read the item from Azure Cosmos DB database demodb and container democontainer using the async API. You should see {"id":"1","name":"dog"} in the response.
curl http://localhost:8080/quarkus-azure-cosmos-async/demodb/democontainer/1 -X GET

# List items from Azure Cosmos DB database demodb and container democontainer using the async API. You should see [{"id":"1","name":"dog"}] in the response.
curl http://localhost:8080/quarkus-azure-cosmos-async/demodb/democontainer -X GET

# Update the item {"id": "1", "name": "dog"} to {"id": "1", "name": "cat"} in Azure Cosmos DB database demodb and container democontainer using the async API.
curl http://localhost:8080/quarkus-azure-cosmos-async/demodb/democontainer -X POST -d '{"id": "1", "name": "cat"}' -H "Content-Type: application/json"

# Read the updated item from Azure Cosmos DB database demodb and container democontainer using the async API. You should see {"id":"1","name":"cat"} in the response.
curl http://localhost:8080/quarkus-azure-cosmos-async/demodb/democontainer/1 -X GET

# List items again from Azure Cosmos DB database demodb and container democontainer using the async API. You should see [{"id":"1","name":"cat"}] in the response.
curl http://localhost:8080/quarkus-azure-cosmos-async/demodb/democontainer -X GET

# Delete the item from Azure Cosmos DB database demodb and container democontainer using the async API.
curl http://localhost:8080/quarkus-azure-cosmos-async/demodb/democontainer/1 -X DELETE

# Read the deleted item from Azure Cosmos DB database demodb and container democontainer using the async API. You should see HTTP status code 500 in the response.
curl http://localhost:8080/quarkus-azure-cosmos-async/demodb/democontainer/1 -X DELETE -w "%{http_code}" -s -o /dev/null

# List items again from Azure Cosmos DB database demodb and container democontainer using the async API. You should see [] in the response.
curl http://localhost:8080/quarkus-azure-cosmos-async/demodb/democontainer -X GET
```

Press `Ctrl + C` to stop the sample once you complete the try and test.

## Cleaning up Azure resources

Run the following command to clean up the Azure resources if you created before:

```
az group delete \
--name ${RESOURCE_GROUP_NAME} \
--yes --no-wait
```
Loading