HADOOP-19232: [ABFS][FNSOverBlob] Implementing Ingress Support with various Fallback Handling #7272

anmolanmol1234 · 2025-01-06T07:05:23Z

Description of PR :

This Pr is in correlation to the series of work done under Parent Jira: [HADOOP-19179]

(https://issues.apache.org/jira/browse/HADOOP-19179)
Jira for this Patch: https://issues.apache.org/jira/browse/HADOOP-19232

Scope of this task is to refactor the AbfsOutputStream class to handle the ingress for DFS and Blob endpoint effectively.

Production code changes :

The AbfsOutputStream class is crucial for handling the data being written to Azure Storage. Its primary responsibilities include:

Buffering: Temporarily holding data in memory before it is uploaded.
Streaming: Efficiently streaming data to Azure Storage.
Committing: Ensuring that buffered data is correctly uploaded and committed to Azure Storage.

New Additions

The new additions introduce a more modular and flexible approach to managing data ingress (data being written to storage), catering to both Azure Data Lake Storage (ADLS) and Azure Blob Storage.

AzureIngressHandler

The AzureIngressHandler is a new parent class designed to encapsulate common logic for data ingress operations. It simplifies the process of writing data to Azure Storage by providing a unified interface. This class has two specialized child classes:

AzureDfsIngressHandler:
- Manages data ingress specifically for Azure Data Lake Storage (DFS).
- Handles operations like creating, appending, and flushing data blocks for DFS.
AzureBlobIngressHandler:
- Manages data ingress specifically for Azure Blob Storage (BLOB).
- Handles operations like creating, appending, and flushing data blocks for Blob Storage, while ensuring that each block has a unique blockId.

AbfsBlock and AbfsBlobBlock

Data is managed in discrete blocks to improve efficiency and manageability.

AbfsBlock:
- A basic structure for buffering data.
- Used as a common block type for both DFS and Blob Storage.
AbfsBlobBlock:
- A subclass of AbfsBlock tailored for Blob Storage.
- Requires a unique blockId for each block, which is necessary for the Blob Storage API.

Block Managers

To manage these data blocks, new manager classes have been introduced. These classes handle the lifecycle of blocks, including creation, appending, and flushing.

AzureBlockManager:
- A parent class for managing the lifecycle of data blocks.
- Provides common functionality for block management.
AzureDFSBlockManager:
- Manages the lifecycle of AbfsBlock instances for DFS.
- Handles the specifics of appending and flushing blocks in DFS.
AzureBlobBlockManager:
- Manages the lifecycle of AbfsBlobBlock instances for Blob Storage.
- Ensures each block has a unique blockId.
- Handles the specifics of appending and flushing blocks in Blob Storage.

Integration with AbfsOutputStream

The AbfsOutputStream class has been updated to incorporate the new ingress flow logic, enhancing its ability to handle data writes to both DFS and Blob Storage. Here’s how it integrates:

Configuration Selection:
- The AbfsOutputStream reads the configuration parameter fs.azure.ingress.service.type to determine whether the user has configured the system to use BLOB or DFS for data ingress.
Handler Initialization:
- Based on the configuration, AbfsOutputStream initializes the appropriate handler (AzureBlobIngressHandler or AzureDfsIngressHandler).
Buffering Data:
- As data is written to AbfsOutputStream, it is buffered into blocks (AbfsBlock for DFS or AbfsBlobBlock for Blob Storage).
Managing Blocks:
- The corresponding block manager (AzureDFSBlockManager or AzureBlobBlockManager) manages the lifecycle of these blocks, ensuring that data is correctly created, appended, and flushed.
Block Id Management (Blob Specific):
- For Blob Storage, AzureBlobBlockManager ensures that each block has a unique blockId, adhering to the requirements of the Blob Storage API.

Detailed Flow

Creating Data Blocks:
- When data is written to AbfsOutputStream, it is divided into blocks (AbfsBlock for DFS or AbfsBlobBlock for Blob Storage).
Appending Data:
- These blocks are appended to the Azure storage system via the appropriate handler (AzureBlobIngressHandler or AzureDfsIngressHandler).
Flushing Data:
- Once all data has been buffered, the blocks are flushed to ensure all buffered data is committed to the storage system.
Lifecycle Management:
- The block managers (AzureDFSBlockManager and AzureBlobBlockManager) oversee the lifecycle of blocks, handling retries, errors, and ensuring data integrity.

Test Code Changes:

Existing tests are modified to work with new abstracted design.
Test Suite was run on DFS Endpoint to make sure the original driver works seamlessly and undisturbed.
Adding some new tests around the new code

anujmodi2021

Added some thoughts around production code.
Will do another round of review for test code,.

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java

...op-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java

...ls/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java

.../java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/InvalidIngressServiceException.java

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlock.java

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlockStatus.java

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsErrors.java

anujmodi2021 · 2025-01-06T10:10:10Z

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java

+
+  /**
+   * Creates or retrieves an existing Azure ingress handler based on the service type and provided parameters.
+   * <p>


seems like xml tag is not closed

which tag is not closed ?

<p> tag in comments

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java

anujmodi2021

Some Suggestions for test code improvement.

hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml

.../java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/InvalidIngressServiceException.java

...ools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsCustomEncryption.java

...ls/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java

...ools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsCustomEncryption.java

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAppend.java

anujmodi2021

Pending Test Code Review

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/AzcopyExecutionException.java

...oop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsPaginatedDelete.java

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsOutputStream.java

...src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsHttpClientRequestExecutor.java

...ls/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFilesystemAcl.java

...ls/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java

bhattmanish98 · 2025-01-09T07:12:52Z

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java

@@ -1140,7 +1387,7 @@ public boolean checkIsDir(AbfsHttpOperation result) {
  public boolean checkUserError(int responseStatusCode) {
    return (responseStatusCode >= HttpURLConnection.HTTP_BAD_REQUEST
        && responseStatusCode < HttpURLConnection.HTTP_INTERNAL_ERROR
-        && responseStatusCode != HttpURLConnection.HTTP_CONFLICT);
+        && responseStatusCode != HTTP_CONFLICT);


We can do same changes for above two codes (HTTP_INTERNAL_ERROR, HTTP_BAD_REQUEST) as well.

reverted this change

bhattmanish98 · 2025-01-09T07:15:52Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlock.java

+    this.outputStream = outputStream;
+    this.offset = offset;
+    DataBlocks.BlockFactory blockFactory = outputStream.getBlockManager().getBlockFactory();
+    long blockCount = outputStream.getBlockManager().getBlockCount();


Since we are using these variables (blockCount, blockSize) only at one place, it would be better to call it inplace.

better to store them as variables for better readability

bhattmanish98 · 2025-01-09T07:25:00Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsErrors.java

+      + "exists and its resource type is invalid for this operation.";
+  public static final String BLOB_OPERATION_NOT_SUPPORTED = "Blob operation is not supported.";
+  public static final String INVALID_APPEND_OPERATION = "The resource was created or modified by the Azure Blob Service API "
+      + "and cannot be appended to by the Azure Data Lake Storage Service API";


Typo error: and cannot be appended by the Azure Data Lake Storage Service API.

This is how the error message comes from the backend

bhattmanish98 · 2025-01-09T07:35:55Z

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java

+  boolean hasActiveBlockDataToUpload() {
+    AzureBlockManager blockManager = getBlockManager();
+    AbfsBlock activeBlock = blockManager.getActiveBlock();
+    return blockManager.hasActiveBlock() && activeBlock.hasData();


Can blockManager and activeBlock be null in any of the case?

hasActiveBlock checks for null for activeBlock and blockManager can not be null

bhattmanish98 · 2025-01-09T07:38:15Z

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java

-      if (hasActiveBlock()) {
-        clearActiveBlock();
-      }
+      getBlockManager().clearActiveBlock();


Should we not have check on hasActiveBlock before clearActiveBlock?

bhattmanish98 · 2025-01-09T07:42:21Z

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java

  @VisibleForTesting
-  AbfsClient getClient() {
+  synchronized AbfsClient getClient() {


What is the need of this change?

was reported as a spot bug hence I added it but dont see the need for it hence reverting

bhattmanish98 · 2025-01-09T07:48:54Z

...hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AzureBlobBlockManager.java

+    AbfsBlobClient blobClient = abfsOutputStream.getClientHandler().getBlobClient();
+    final AbfsRestOperation op = blobClient
+        .getBlockList(abfsOutputStream.getPath(), tracingContext);
+    committedBlockIdList = op.getResult().getBlockIdList();


A check on op and op.getResult before calling getBlockIdList is needed.

bhattmanish98 · 2025-01-09T07:55:09Z

...doop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AzureBlobIngressHandler.java

+    BlobAppendRequestParameters blobParams = new BlobAppendRequestParameters(blockToUpload.getBlockId(), getETag());
+    reqParams.setBlobParams(blobParams);
+    AbfsRestOperation op;
+    long threadId = Thread.currentThread().getId();


Can we get Thread.currentThread().getId() inplace as we are only using it once?

bhattmanish98 · 2025-01-09T09:14:30Z

...adoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AzureDFSIngressHandler.java

+      TracingContext tracingContext) throws IOException {
+    TracingContext tracingContextAppend = new TracingContext(tracingContext);
+    long threadId = Thread.currentThread().getId();
+    String threadIdStr = String.valueOf(threadId);


Same as above, if threadIf is getting used at only one place, can we keep it inplace?

bhattmanish98 · 2025-01-09T09:17:11Z

...adoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AzureDFSIngressHandler.java

+      IOUtils.closeStreams(uploadData, activeBlock);
+    }
+  }
+


Please remove 1 extra empty line.

bhattmanish98 · 2025-01-09T09:18:05Z

...c/main/java/org/apache/hadoop/fs/azurebfs/services/AzureDfsToBlobIngressFallbackHandler.java

+    AbfsRestOperation op;
+    TracingContext tracingContextAppend = new TracingContext(tracingContext);
+    long threadId = Thread.currentThread().getId();
+    String threadIdStr = String.valueOf(threadId);


Same as above.

bhattmanish98 · 2025-01-09T09:20:12Z

...adoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AzureDFSIngressHandler.java

+      throws IOException {
+    TracingContext tracingContextFlush = new TracingContext(tracingContext);
+    if (tracingContextFlush.getIngressHandler().equals(EMPTY_STRING)) {
+      tracingContextFlush.setIngressHandler("DFlush");


Can we create constant for DFlush and use it whereever required?

bhattmanish98 · 2025-01-09T09:21:31Z

...adoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AzureDFSIngressHandler.java

+    // Perform the upload within a performance tracking context.
+    try (AbfsPerfInfo perfInfo = new AbfsPerfInfo(
+        dfsClient.getAbfsPerfTracker(),
+        "writeCurrentBufferToService", "append")) {


can we create constant for append?

already present, added it here

bhattmanish98 · 2025-01-09T09:22:30Z

...c/main/java/org/apache/hadoop/fs/azurebfs/services/AzureDfsToBlobIngressFallbackHandler.java

+    }
+    try {
+      TracingContext tracingContextFlush = new TracingContext(tracingContext);
+      tracingContextFlush.setIngressHandler("FBFlush");


Same as above, constant for FBFlush

bhattmanish98 · 2025-01-09T09:23:09Z

...c/main/java/org/apache/hadoop/fs/azurebfs/services/AzureDfsToBlobIngressFallbackHandler.java

+    // Perform the upload within a performance tracking context.
+    try (AbfsPerfInfo perfInfo = new AbfsPerfInfo(
+        getClient().getAbfsPerfTracker(),
+        "writeCurrentBufferToService", "append")) {


Same as above, constant for append.

hadoop-yetus · 2025-01-13T06:48:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 32 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	22m 25s		trunk passed
+1 💚	compile	0m 23s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 20s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 27s		trunk passed
+1 💚	javadoc	0m 28s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 22s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 43s		trunk passed
+1 💚	shadedclient	19m 51s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	20m 3s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 20s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 19s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 14s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 12 new + 19 unchanged - 1 fixed = 31 total (was 20)
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 17s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 43s		the patch passed
+1 💚	shadedclient	19m 30s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 0s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		77m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7272/10/artifact/out/Dockerfile
GITHUB PR	#7272
JIRA Issue	HADOOP-19232
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux e7c37c718562 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b9f2f7b`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7272/10/testReport/
Max. process+thread count	736 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7272/10/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-01-15T11:36:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 32 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	21m 51s		trunk passed
+1 💚	compile	0m 24s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 23s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 21s		trunk passed
+1 💚	mvnsite	0m 24s		trunk passed
+1 💚	javadoc	0m 27s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 20s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 43s		trunk passed
+1 💚	shadedclient	19m 19s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	19m 31s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 19s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 13s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 12 new + 22 unchanged - 1 fixed = 34 total (was 23)
+1 💚	mvnsite	0m 19s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 16s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 45s		the patch passed
+1 💚	shadedclient	19m 26s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 59s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		69m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7272/11/artifact/out/Dockerfile
GITHUB PR	#7272
JIRA Issue	HADOOP-19232
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 1f388635e774 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0d2c693`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7272/11/testReport/
Max. process+thread count	555 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7272/11/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

anujmodi2021

Reviewed Latest Iteration

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

...ls/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java

.../java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/InvalidIngressServiceException.java

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemCreate.java

.../hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemLease.java

hadoop-yetus · 2025-01-17T10:26:08Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 2s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 32 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 49s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 31s		trunk passed
+1 💚	mvnsite	0m 39s		trunk passed
+1 💚	javadoc	0m 39s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 33s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 8s		trunk passed
+1 💚	shadedclient	35m 45s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	36m 7s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 20s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 21 new + 22 unchanged - 1 fixed = 43 total (was 23)
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 6s		the patch passed
+1 💚	shadedclient	34m 0s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 20s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		119m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7272/13/artifact/out/Dockerfile
GITHUB PR	#7272
JIRA Issue	HADOOP-19232
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux e885af0a36c5 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `51ebb13`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7272/13/testReport/
Max. process+thread count	554 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7272/13/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-01-17T10:26:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 25s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 32 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 28s		trunk passed
+1 💚	compile	0m 38s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 37s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 32s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 40s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 35s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 6s		trunk passed
+1 💚	shadedclient	33m 7s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	33m 28s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 30s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 20s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 19 new + 22 unchanged - 1 fixed = 41 total (was 23)
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 4s		the patch passed
+1 💚	shadedclient	32m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 24s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		115m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7272/14/artifact/out/Dockerfile
GITHUB PR	#7272
JIRA Issue	HADOOP-19232
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux d720330b6793 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `dabcb1e`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7272/14/testReport/
Max. process+thread count	699 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7272/14/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

anmolanmol1234 · 2025-01-20T06:41:09Z

============================================================
HNS-OAuth

[ERROR] testTwoWritersCreateAppendWithInfiniteLeaseEnabled(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease) Time elapsed: 98.189 s <<< ERROR!

[WARNING] Tests run: 160, Failures: 0, Errors: 0, Skipped: 4
[ERROR] Tests run: 717, Failures: 0, Errors: 1, Skipped: 108
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey

[ERROR] testTwoWritersCreateAppendWithInfiniteLeaseEnabled(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease) Time elapsed: 91.939 s <<< ERROR!

[WARNING] Tests run: 160, Failures: 0, Errors: 0, Skipped: 5
[ERROR] Tests run: 717, Failures: 0, Errors: 1, Skipped: 60
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 10

============================================================
NonHNS-SharedKey

[ERROR] testTwoWritersCreateAppendWithInfiniteLeaseEnabled(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease) Time elapsed: 100.864 s <<< ERROR!

[WARNING] Tests run: 160, Failures: 0, Errors: 0, Skipped: 11
[ERROR] Tests run: 701, Failures: 0, Errors: 1, Skipped: 303
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 27
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 11

============================================================
AppendBlob-HNS-OAuth

[WARNING] Tests run: 160, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 717, Failures: 0, Errors: 0, Skipped: 113
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 49
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23

Ingress changes with fallback

a0164df

github-actions bot added trunk TOOLS ABFS labels Jan 6, 2025

anmolanmol1234 changed the title ~~Ingress changes with fallback~~ HADOOP-19232: [ABFS][FNSOverBlob] Implementing Ingress Support with various Fallback Handling Jan 6, 2025

This comment was marked as outdated.

Sign in to view

anmolanmol1234 marked this pull request as draft January 6, 2025 08:45

anujmodi2021 reviewed Jan 6, 2025

View reviewed changes

spotbugs and checkstyle fixes

8effcb0

This comment was marked as outdated.

Sign in to view

anmolanmol1234 added 3 commits January 6, 2025 23:10

spotbugs

03d51bf

revert param changes

38c412f

revert SAS changes

d487e74

This comment was marked as outdated.

Sign in to view

javadoc fix

ffc22e6

This comment was marked as outdated.

Sign in to view

anujmodi2021 reviewed Jan 8, 2025

View reviewed changes

anujmodi2021 requested changes Jan 9, 2025

View reviewed changes

bhattmanish98 reviewed Jan 9, 2025

View reviewed changes

review comments

2d60502

This comment was marked as outdated.

Sign in to view

changes for spotbugs and checkstyle

3b3005d

This comment was marked as outdated.

Sign in to view

inconsistent sync

e0a228a

This comment was marked as outdated.

Sign in to view

anmolanmol1234 marked this pull request as ready for review January 13, 2025 05:07

uncomment code

b9f2f7b

Fixed SAS support for FNS Blob

0d2c693

anujmodi2021 requested changes Jan 16, 2025

View reviewed changes

anujmodi2021 reviewed Jan 16, 2025

View reviewed changes

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemCreate.java Outdated Show resolved Hide resolved

anujmodi2021 reviewed Jan 16, 2025

View reviewed changes

.../hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemLease.java Outdated Show resolved Hide resolved

PR review comments

1df66bb

This comment was marked as outdated.

Sign in to view

anmolanmol1234 added 2 commits January 17, 2025 00:22

Fix tests

51ebb13

unused code

dabcb1e

HADOOP-19232: [ABFS][FNSOverBlob] Implementing Ingress Support with various Fallback Handling #7272

Are you sure you want to change the base?

HADOOP-19232: [ABFS][FNSOverBlob] Implementing Ingress Support with various Fallback Handling #7272

Conversation

anmolanmol1234 commented Jan 6, 2025 • edited Loading

Description of PR :

Production code changes :

New Additions

AzureIngressHandler

AbfsBlock and AbfsBlobBlock

Block Managers

Integration with AbfsOutputStream

Detailed Flow

Test Code Changes:

This comment was marked as outdated.

anujmodi2021 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anujmodi2021 Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

anujmodi2021 left a comment

Choose a reason for hiding this comment

anujmodi2021 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

hadoop-yetus commented Jan 13, 2025

hadoop-yetus commented Jan 15, 2025

anujmodi2021 left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

hadoop-yetus commented Jan 17, 2025

hadoop-yetus commented Jan 17, 2025

anmolanmol1234 commented Jan 20, 2025

============================================================ HNS-OAuth

============================================================ HNS-SharedKey

============================================================ NonHNS-SharedKey

============================================================ AppendBlob-HNS-OAuth

anmolanmol1234 commented Jan 6, 2025 •

edited

Loading

anujmodi2021 Jan 7, 2025 •

edited

Loading

============================================================
HNS-OAuth

============================================================
HNS-SharedKey

============================================================
NonHNS-SharedKey

============================================================
AppendBlob-HNS-OAuth