Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quarkus azure eventhubs Extension #310

Merged
merged 40 commits into from
Dec 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
77c891c
add extension: azure-eventhubs
backwind1233 Dec 4, 2024
6b848c1
Add Azure Eventhubs config and improve EventhubsClientProducer format…
backwind1233 Dec 4, 2024
3563447
Fix inconsistent indentation in EventhubsClientProducer.java file.
backwind1233 Dec 4, 2024
36bd9b5
Add Azure EventHubs integration tests and update documentation.
backwind1233 Dec 10, 2024
73e2b29
Add event consumption feature and update tests in EventhubsResource.
backwind1233 Dec 10, 2024
7386b6b
Add Azure Blob Checkpoint Store dependency to pom.xml.
backwind1233 Dec 10, 2024
29a1dbb
Add runtime initialization for TcpClientSecure in EventhubsProcessor.
backwind1233 Dec 10, 2024
e017dd8
Add .gitattributes to enforce LF line endings for .adoc files.
backwind1233 Dec 10, 2024
947be7c
Add integration tests for Event Hubs async operations and logging.
backwind1233 Dec 12, 2024
8ba91cf
Rename fullyQualifiedNamespace to namespace and update related code.
backwind1233 Dec 12, 2024
d16144d
Add Azure Event Hubs creation and deletion commands to scripts.
backwind1233 Dec 13, 2024
1906926
Refactor event publishing to use SendOptions for improved reliability.
backwind1233 Dec 13, 2024
3ee1a31
Add partition count to Azure Event Hub resource creation script.
backwind1233 Dec 13, 2024
7347093
Refactor EventHubs to use simple namespace in docs and code.
backwind1233 Dec 16, 2024
78d26f9
Correct variable names for Event Hub configuration consistency.
backwind1233 Dec 16, 2024
87d5314
Update README to reflect Azure Eventhubs integration instead of CosmosDB
backwind1233 Dec 16, 2024
0918294
Update README to reflect Azure Eventhub setup
backwind1233 Dec 16, 2024
f5a98a3
Merge branch 'main' into quarkus-azure-eventhub
backwind1233 Dec 17, 2024
3ea050e
Merge branch 'main' into quarkus-azure-eventhub
backwind1233 Dec 17, 2024
6280529
Remove unused import and reorder remaining imports for clarity.
backwind1233 Dec 17, 2024
aa4a431
Reorder import statements for consistency across files.
backwind1233 Dec 17, 2024
6bf3af5
Update assignee-object-id with correct object ID in Azure role assign…
backwind1233 Dec 17, 2024
f67cbae
Add config check before creating EventHub clients.
backwind1233 Dec 17, 2024
4406ffc
Update namespace property and role assignment scope for Event Hubs
backwind1233 Dec 17, 2024
0fe2fe0
Rename workflow to include integration tests in build process
backwind1233 Dec 17, 2024
5f799f0
Add Quarkus Azure Event Hubs extension documentation and script adjus…
backwind1233 Dec 17, 2024
0d5a0b0
Increase sleep duration for event sending and remove producer closure.
backwind1233 Dec 18, 2024
15cf5b4
Update documentation to include example resource for Event Hubs.
backwind1233 Dec 18, 2024
7c50955
Update Azure Event Hubs documentation to include async client support.
backwind1233 Dec 18, 2024
e3fdbf8
Update build.yml to include hyphens in resource naming conventions.
backwind1233 Dec 18, 2024
9f99e05
Update storage account name format in build workflow configuration.
backwind1233 Dec 18, 2024
488735b
Add newline at the end of the create-azure-resources.sh file.
backwind1233 Dec 20, 2024
c628c7b
Fix missing newline at end of file in pom.xml for Azure Event Hubs.
backwind1233 Dec 20, 2024
098ae92
Update API endpoints from publish/consume to send/receive for clarity.
backwind1233 Dec 20, 2024
5d87f2f
Corrects "Eventhubs" to "Event Hubs" across multiple documentation fi…
backwind1233 Dec 20, 2024
a74837b
Replace SLF4J Logger with JBoss logging in Eventhubs resources.
backwind1233 Dec 23, 2024
2051cf1
Update Azure Event Hubs documentation and configuration properties.
backwind1233 Dec 24, 2024
1c959e9
Rename event hub variables for consistency across documentation and s…
backwind1233 Dec 24, 2024
023c4b3
Add Quarkus Azure Event Hubs Extension information to the README.
backwind1233 Dec 24, 2024
780cdb4
Update event hub naming conventions for consistency across files.
backwind1233 Dec 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*.adoc text eol=lf
13 changes: 13 additions & 0 deletions .github/create-azure-resources.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ set -Euo pipefail
# - APP_CONFIG_NAME
# - KEY_VAULT_NAME
# - COSMOSDB_ACCOUNT_NAME
# - EVENTHUBS_NAMESPACE
# - EVENTHUBS_EVENTHUB_NAME

# Create a resource group
az group create \
Expand Down Expand Up @@ -69,3 +71,14 @@ az cosmosdb create \
-g ${RESOURCE_GROUP_NAME} \
--default-consistency-level Session \
--locations regionName='West US' failoverPriority=0 isZoneRedundant=False

# Azure Event Hubs Extension
az eventhubs namespace create \
--name ${EVENTHUBS_NAMESPACE} \
--resource-group ${RESOURCE_GROUP_NAME}

az eventhubs eventhub create \
--name ${EVENTHUBS_EVENTHUB_NAME} \
--namespace-name ${EVENTHUBS_NAMESPACE} \
--resource-group ${RESOURCE_GROUP_NAME} \
--partition-count 2
majguo marked this conversation as resolved.
Show resolved Hide resolved
1 change: 1 addition & 0 deletions .github/delete-azure-resources.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ az appconfig delete --name "${APP_CONFIG_NAME}" --resource-group "${RESOURCE_GRO
az appconfig purge --name "${APP_CONFIG_NAME}" --yes
az keyvault delete --name "${KEY_VAULT_NAME}" --resource-group "${RESOURCE_GROUP_NAME}"
az keyvault purge --name "${KEY_VAULT_NAME}"
az eventhubs namespace delete --resource-group "${RESOURCE_GROUP_NAME}" --name "${EVENTHUBS_NAMESPACE}"
az group delete --name "${RESOURCE_GROUP_NAME}" --yes --no-wait
10 changes: 10 additions & 0 deletions .github/run-integration-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,16 @@ else
--role-definition-id 00000000-0000-0000-0000-000000000002
fi

# Azure Event Hubs Extension
OBJECT_ID=$(az ad sp list --filter "appId eq '$AZURE_CLIENT_ID'" --query '[0].id' -o tsv)
az role assignment create \
--role "Azure Event Hubs Data Owner" \
--assignee-object-id ${OBJECT_ID} \
--scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP_NAME}/providers/Microsoft.EventHub/namespaces/${EVENTHUBS_NAMESPACE}"

export QUARKUS_AZURE_EVENTHUBS_NAMESPACE=${EVENTHUBS_NAMESPACE}
export QUARKUS_AZURE_EVENTHUBS_EVENTHUB_NAME=${EVENTHUBS_EVENTHUB_NAME}

majguo marked this conversation as resolved.
Show resolved Hide resolved
# Run integration test with existing native executables against Azure services
mvn -B test-compile failsafe:integration-test -Dnative -Dazure.test=true

Expand Down
12 changes: 7 additions & 5 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Build
name: Build and Integration Test

on:
push:
Expand Down Expand Up @@ -39,11 +39,13 @@ jobs:
AZURE_CLIENT_ID: ${{ vars.AZURE_CLIENT_ID }}
AZURE_TENANT_ID: ${{ vars.AZURE_TENANT_ID }}
AZURE_SUBSCRIPTION_ID: ${{ vars.AZURE_SUBSCRIPTION_ID }}
RESOURCE_GROUP_NAME: ${{ github.repository_owner }}-quarkus${{ github.run_id }}${{ github.run_number }}
RESOURCE_GROUP_NAME: ${{ github.repository_owner }}-quarkus${{ github.run_id }}-${{ github.run_number }}
STORAGE_ACCOUNT_NAME: storage${{ github.run_id }}${{ github.run_number }}
APP_CONFIG_NAME: appconfig${{ github.run_id }}${{ github.run_number }}
KEY_VAULT_NAME: kv${{ github.run_id }}${{ github.run_number }}
COSMOSDB_ACCOUNT_NAME: cosmos${{ github.run_id }}${{ github.run_number }}
APP_CONFIG_NAME: appconfig${{ github.run_id }}-${{ github.run_number }}
KEY_VAULT_NAME: kv${{ github.run_id }}-${{ github.run_number }}
COSMOSDB_ACCOUNT_NAME: cosmos${{ github.run_id }}-${{ github.run_number }}
EVENTHUBS_NAMESPACE: eventhubs-ns-${{ github.run_id }}-${{ github.run_number }}
EVENTHUBS_EVENTHUB_NAME: eventhubs-ehn-${{ github.run_id }}-${{ github.run_number }}
name: Build on ${{ matrix.os }}
strategy:
fail-fast: false
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ Here's the current selection of Quarkus Azure Services:
- [Quarkus Azure Blob Storage Extension](https://docs.quarkiverse.io/quarkus-azure-services/dev/quarkus-azure-storage-blob.html): [Azure Blob Storage](https://azure.microsoft.com/products/storage/blobs/)
is a massively scalable and secure object storage for cloud-native workloads, archives, data lakes, high-performance
computing, and machine learning.
- [Quarkus Azure Event Hubs Extension](https://docs.quarkiverse.io/quarkus-azure-services/dev/quarkus-azure-eventhubs.html): [Azure Event Hubs](https://azure.microsoft.com/products/event-hubs)
is a big data streaming platform and event ingestion service. It can receive and process millions of events per second.

## Azure Services outside this repository
- [Quarkus Opentelemetry Exporter for Microsoft Azure](https://docs.quarkiverse.io/quarkus-opentelemetry-exporter/dev/quarkus-opentelemetry-exporter-azure.html) enables [Azure Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview) telemetry for Quarkus native applications
Expand All @@ -37,6 +39,7 @@ Example applications can be found inside the [integration-tests](integration-tes
- Load a secret from Azure Key Vault as property using `ConfigProperty`.
- [Azure Blob Storage sample](integration-tests/azure-storage-blob): REST endpoint using the Quarkus extension to
upload and download files to/from Azure Blob Storage.
- [Azure Event Hubs sample](integration-tests/azure-eventhubs): REST endpoint using the Quarkus extension to send/receive data to/from Azure Event Hubs.

## How to release a new version

Expand Down
10 changes: 10 additions & 0 deletions bom/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,16 @@
<artifactId>quarkus-azure-cosmos-deployment</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>io.quarkiverse.azureservices</groupId>
<artifactId>quarkus-azure-eventhubs</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>io.quarkiverse.azureservices</groupId>
<artifactId>quarkus-azure-eventhubs-deployment</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>io.quarkiverse.azureservices</groupId>
<artifactId>quarkus-azure-keyvault</artifactId>
Expand Down
79 changes: 79 additions & 0 deletions docs/modules/ROOT/pages/includes/quarkus-azure-eventhubs.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
[.configuration-legend]
icon:lock[title=Fixed at build time] Configuration property fixed at build time - All other configuration properties are overridable at runtime
[.configuration-reference.searchable, cols="80,.^10,.^10"]
|===

h|[.header-title]##Configuration property##
h|Type
h|Default

a| [[quarkus-azure-eventhubs_quarkus-azure-eventhubs-enabled]] [.property-path]##link:#quarkus-azure-eventhubs_quarkus-azure-eventhubs-enabled[`quarkus.azure.eventhubs.enabled`]##

[.description]
--
The flag to enable the eventhubs. If set to false, the eventhubs will be disabled


ifdef::add-copy-button-to-env-var[]
Environment variable: env_var_with_copy_button:+++QUARKUS_AZURE_EVENTHUBS_ENABLED+++[]
endif::add-copy-button-to-env-var[]
ifndef::add-copy-button-to-env-var[]
Environment variable: `+++QUARKUS_AZURE_EVENTHUBS_ENABLED+++`
endif::add-copy-button-to-env-var[]
--
|boolean
|`true`

a| [[quarkus-azure-eventhubs_quarkus-azure-eventhubs-namespace]] [.property-path]##link:#quarkus-azure-eventhubs_quarkus-azure-eventhubs-namespace[`quarkus.azure.eventhubs.namespace`]##

[.description]
--
The namespace of the event hub. Required if quarkus.azure.eventhubs.enabled is set to true


ifdef::add-copy-button-to-env-var[]
Environment variable: env_var_with_copy_button:+++QUARKUS_AZURE_EVENTHUBS_NAMESPACE+++[]
endif::add-copy-button-to-env-var[]
ifndef::add-copy-button-to-env-var[]
Environment variable: `+++QUARKUS_AZURE_EVENTHUBS_NAMESPACE+++`
endif::add-copy-button-to-env-var[]
--
|string
|required icon:exclamation-circle[title=Configuration property is required]

a| [[quarkus-azure-eventhubs_quarkus-azure-eventhubs-domain-name]] [.property-path]##link:#quarkus-azure-eventhubs_quarkus-azure-eventhubs-domain-name[`quarkus.azure.eventhubs.domain-name`]##

[.description]
--
The domain name of the event hub. Required if quarkus.azure.eventhubs.enabled is set to true


ifdef::add-copy-button-to-env-var[]
Environment variable: env_var_with_copy_button:+++QUARKUS_AZURE_EVENTHUBS_DOMAIN_NAME+++[]
endif::add-copy-button-to-env-var[]
ifndef::add-copy-button-to-env-var[]
Environment variable: `+++QUARKUS_AZURE_EVENTHUBS_DOMAIN_NAME+++`
endif::add-copy-button-to-env-var[]
--
|string
|`servicebus.windows.net`

a| [[quarkus-azure-eventhubs_quarkus-azure-eventhubs-eventhub-name]] [.property-path]##link:#quarkus-azure-eventhubs_quarkus-azure-eventhubs-eventhub-name[`quarkus.azure.eventhubs.eventhub-name`]##

[.description]
--
The name of the event hub. Required if quarkus.azure.eventhubs.enabled is set to true


ifdef::add-copy-button-to-env-var[]
Environment variable: env_var_with_copy_button:+++QUARKUS_AZURE_EVENTHUBS_EVENTHUB_NAME+++[]
endif::add-copy-button-to-env-var[]
ifndef::add-copy-button-to-env-var[]
Environment variable: `+++QUARKUS_AZURE_EVENTHUBS_EVENTHUB_NAME+++`
endif::add-copy-button-to-env-var[]
--
|string
|required icon:exclamation-circle[title=Configuration property is required]

|===

Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
[.configuration-legend]
icon:lock[title=Fixed at build time] Configuration property fixed at build time - All other configuration properties are overridable at runtime
[.configuration-reference.searchable, cols="80,.^10,.^10"]
|===

h|[.header-title]##Configuration property##
h|Type
h|Default

a| [[quarkus-azure-eventhubs_quarkus-azure-eventhubs-enabled]] [.property-path]##link:#quarkus-azure-eventhubs_quarkus-azure-eventhubs-enabled[`quarkus.azure.eventhubs.enabled`]##

[.description]
--
The flag to enable the eventhubs. If set to false, the eventhubs will be disabled


ifdef::add-copy-button-to-env-var[]
Environment variable: env_var_with_copy_button:+++QUARKUS_AZURE_EVENTHUBS_ENABLED+++[]
endif::add-copy-button-to-env-var[]
ifndef::add-copy-button-to-env-var[]
Environment variable: `+++QUARKUS_AZURE_EVENTHUBS_ENABLED+++`
endif::add-copy-button-to-env-var[]
--
|boolean
|`true`

a| [[quarkus-azure-eventhubs_quarkus-azure-eventhubs-namespace]] [.property-path]##link:#quarkus-azure-eventhubs_quarkus-azure-eventhubs-namespace[`quarkus.azure.eventhubs.namespace`]##

[.description]
--
The namespace of the event hub. Required if quarkus.azure.eventhubs.enabled is set to true


ifdef::add-copy-button-to-env-var[]
Environment variable: env_var_with_copy_button:+++QUARKUS_AZURE_EVENTHUBS_NAMESPACE+++[]
endif::add-copy-button-to-env-var[]
ifndef::add-copy-button-to-env-var[]
Environment variable: `+++QUARKUS_AZURE_EVENTHUBS_NAMESPACE+++`
endif::add-copy-button-to-env-var[]
--
|string
|required icon:exclamation-circle[title=Configuration property is required]

a| [[quarkus-azure-eventhubs_quarkus-azure-eventhubs-domain-name]] [.property-path]##link:#quarkus-azure-eventhubs_quarkus-azure-eventhubs-domain-name[`quarkus.azure.eventhubs.domain-name`]##

[.description]
--
The domain name of the event hub. Required if quarkus.azure.eventhubs.enabled is set to true


ifdef::add-copy-button-to-env-var[]
Environment variable: env_var_with_copy_button:+++QUARKUS_AZURE_EVENTHUBS_DOMAIN_NAME+++[]
endif::add-copy-button-to-env-var[]
ifndef::add-copy-button-to-env-var[]
Environment variable: `+++QUARKUS_AZURE_EVENTHUBS_DOMAIN_NAME+++`
endif::add-copy-button-to-env-var[]
--
|string
|`servicebus.windows.net`

a| [[quarkus-azure-eventhubs_quarkus-azure-eventhubs-eventhub-name]] [.property-path]##link:#quarkus-azure-eventhubs_quarkus-azure-eventhubs-eventhub-name[`quarkus.azure.eventhubs.eventhub-name`]##

[.description]
--
The name of the event hub. Required if quarkus.azure.eventhubs.enabled is set to true


ifdef::add-copy-button-to-env-var[]
Environment variable: env_var_with_copy_button:+++QUARKUS_AZURE_EVENTHUBS_EVENTHUB_NAME+++[]
endif::add-copy-button-to-env-var[]
ifndef::add-copy-button-to-env-var[]
Environment variable: `+++QUARKUS_AZURE_EVENTHUBS_EVENTHUB_NAME+++`
endif::add-copy-button-to-env-var[]
--
|string
|required icon:exclamation-circle[title=Configuration property is required]

|===

2 changes: 2 additions & 0 deletions docs/modules/ROOT/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,5 @@ include::quarkus-azure-cosmos.adoc[leveloffset=+1, opts=optional]
include::quarkus-azure-key-vault.adoc[leveloffset=+1, opts=optional]

include::quarkus-azure-storage-blob.adoc[leveloffset=+1, opts=optional]

include::quarkus-azure-eventhubs.adoc[leveloffset=+1, opts=optional]
Loading
Loading