Merge pull request #163 from jofriedm-msft/master

Java Storage Client Library 5.2.0

BreakingChanges.txt

@@ -1,4 +1,4 @@
-Changes in 5.1.1
+Changes in 5.1.1
 OTHER
   * Reverted the code from 5.1.0 because it contained a regression and an accidental change.
 
@@ -9,9 +9,9 @@ BLOB
  * getQualifiedStorageUri() has been deprecated. Please use getSnapshotQualifiedStorageUri() instead. This new function will return the blob including the snapshot (if present) and no SAS token.
  * Fixed a bug where copying from a blob that included a SAS token and a snapshot did not use the SAS token.
 
- FILE
+FILE
  * Fixed a bug where copying from a blob that included a SAS token and a snapshot did not use the SAS token.
- 
+
 QUEUE
  * For addMessage() the CloudQueueMessage message passed in will be populated with the pop receipt, insertion/expiration time, and message ID.
 
@@ -168,8 +168,8 @@ Changes in 0.5.0
 
  OTHER
  * The packaging structure has changed to
-	1. com.microsoft.windowsazure.storage � RetryPolicies, LocationMode, StorageException, Storage Credentials etc. Basically all public classes that are common across services.
-	2. com.microsoft.windowsazure.storage.core � These classes are reserved for internal use and users never have to include this namespace in their projects. 
+	1. com.microsoft.windowsazure.storage � RetryPolicies, LocationMode, StorageException, Storage Credentials etc. Basically all public classes that are common across services.
+	2. com.microsoft.windowsazure.storage.core � These classes are reserved for internal use and users never have to include this namespace in their projects. 
 	3. com.microsoft.windowsazure.storage.blob
 	4. com.microsoft.windowsazure.storage.queue
 	5. com.microsoft.windowsazure.storage.table

ChangeLog.txt

@@ -1,3 +1,12 @@
+2017.05.23 Version 5.2.0
+ * Fixed Exists() calls on Shares and Directories to now populate metadata. This was already being done for Files.
+ * Changed blob constants to support up to 256 MB on put blob for block blobs. The default value for put blob threshold has also been updated to half of the maximum, or 128 MB currently.
+ * Fixed a bug that prevented setting content MD5 to true when creating a new file.
+ * Fixed a bug where access conditions, options, and operation context were not being passed when calling openWriteExisting() on a page blob or a file.
+ * Fixed a bug where an exception was being thrown on a range get of a blob or file when the options disableContentMD5Validation is set to false and useTransactionalContentMD5 is set to true and there is no overall MD5.
+ * Fixed a bug where retries were happening immediately if a socket exception was thrown.
+ * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. This check has been moved to create() and uploadProperties() in CloudFileShare.
+
 2017.05.14 Version 5.1.1
  * Reverted version 5.1.0 due to a regression which was caught after publishing. Version 5.2.0 will be released that contains the fixes from 5.1.0 without the regression.
 

README.md

@@ -1,10 +1,10 @@
-#Microsoft Azure Storage SDK for Java
+# Microsoft Azure Storage SDK for Java
 
 This project provides a client library in Java that makes it easy to consume Microsoft Azure Storage services. For documentation please see the Microsoft Azure [Java Developer Center](http://azure.microsoft.com/en-us/develop/java/) and the [JavaDocs](http://azure.github.io/azure-storage-java/).
 
 > If you are looking for the Azure Storage Android SDK, please visit [https://github.com/Azure/azure-storage-android](https://github.com/Azure/azure-storage-android).
 
-#Features
+# Features
   * Blob
       * Create/Read/Update/Delete containers
       * Create/Read/Update/Delete blobs
@@ -19,34 +19,34 @@ This project provides a client library in Java that makes it easy to consume Mic
       * Batch operations
       * Advanced Table Operations
 
-#Getting Started
+# Getting Started
 
-##Download
-###Option 1: Via Maven
+## Download
+### Option 1: Via Maven
 
 To get the binaries of this library as distributed by Microsoft, ready for use within your project, you can use Maven.
 
 ```xml
 <dependency>
 	<groupId>com.microsoft.azure</groupId>
 	<artifactId>azure-storage</artifactId>
-	<version>5.1.1</version>
+	<version>5.2.0</version>
 </dependency>
 ```
 
-###Option 2: Source Via Git
+### Option 2: Source Via Git
 
 To get the source code of the SDK via git just type:
 
     git clone git://github.com/Azure/azure-storage-java.git
-    cd ./azure-storage-java/microsoft-azure-storage
+    cd ./azure-storage-java
     mvn compile
 
-###Option 3: Source Zip
+### Option 3: Source Zip
 
 To download a copy of the source code, click "Download ZIP" on the right side of the page or click [here](https://github.com/Azure/azure-storage-java/archive/master.zip). Unzip and navigate to the microsoft-azure-storage folder.
 
-##Minimum Requirements
+## Minimum Requirements
 
 * Java 1.6+
 * [Jackson-Core](https://github.com/FasterXML/jackson-core) is used for JSON parsing. 
@@ -58,13 +58,13 @@ The two dependencies, [Jackson-Core](https://github.com/FasterXML/jackson-core)
 
 SLF4J is only needed if you enable logging through the OperationContext class. If you plan to use logging, please also download an [SLF4J binding](http://repo2.maven.org/maven2/org/slf4j/) which will link the SLF4J API with the logging implementation of your choice. Simple is a good default. See the [SLF4J user manual](http://www.slf4j.org/manual.html) for more information.
 
-##Usage
+## Usage
 
 To use this SDK to call Microsoft Azure storage services, you need to first [create an account](https://account.windowsazure.com/signup).
 
 Samples are provided in the microsoft-azure-storage-samples folder. The unit tests in microsoft-azure-storage-test can also be helpful.
 
-##Code Sample
+## Code Sample
 
 The following is a quick example on how to upload a file to azure blob and download it back. You may also download and view the samples in the microsoft-azure-storage-samples folder. For additional information on using the client libraries to access Azure services see the How To guides for [blobs](http://azure.microsoft.com/en-us/documentation/articles/storage-java-how-to-use-blob-storage/), [queues](http://azure.microsoft.com/en-us/documentation/articles/storage-java-how-to-use-queue-storage/), [tables](http://azure.microsoft.com/en-us/documentation/articles/storage-java-how-to-use-table-storage/) and the [general documentation](http://azure.microsoft.com/en-us/develop/java/).
 
@@ -117,17 +117,17 @@ public class BlobSample {
 }
 ```
 
-#Need Help?
+# Need Help?
 
 Be sure to check out the Microsoft Azure [Developer Forums on MSDN](http://social.msdn.microsoft.com/Forums/windowsazure/en-US/home?forum=windowsazuredata) or the [Developer Forums on Stack Overflow](http://stackoverflow.com/questions/tagged/azure+windows-azure-storage) if you have trouble with the provided code.
 
-#Contribute Code or Provide Feedback
+# Contribute Code or Provide Feedback
 
 If you would like to become an active contributor to this project please follow the instructions provided in [Azure Projects Contribution Guidelines](http://azure.github.io/guidelines/).
 
 If you encounter any bugs with the library please file an issue in the [Issues](https://github.com/Azure/azure-storage-java/issues) section of the project.
 
-#Learn More
+# Learn More
 
 * [Azure Developer Center](http://azure.microsoft.com/en-us/develop/java/)
 * [Azure Storage Service](http://azure.microsoft.com/en-us/documentation/services/storage/)

microsoft-azure-storage-samples/pom.xml

@@ -26,7 +26,7 @@
 		<dependency>
 			<groupId>com.microsoft.azure</groupId>
 			<artifactId>azure-storage</artifactId>
-			<version>5.1.1</version>
+			<version>5.2.0</version>
 		</dependency>
 		<dependency>
 			<groupId>com.microsoft.azure</groupId>

microsoft-azure-storage-test/src/com/microsoft/azure/storage/MaximumExecutionTimeTests.java

@@ -191,6 +191,9 @@ public void testMaximumExecutionTimeBlobWrites() throws URISyntaxException, Stor
         BlobRequestOptions options = new BlobRequestOptions();
         options.setMaximumExecutionTimeInMs(5000);
 
+        // set a lower put blob threshold so that we perform multiple put block requests that timeout
+        options.setSingleBlobPutThresholdInBytes(32 * Constants.MB);
+
         CloudBlobClient blobClient = TestHelper.createCloudBlobClient();
         CloudBlobContainer container = blobClient.getContainerReference(generateRandomName("container"));
 

microsoft-azure-storage-test/src/com/microsoft/azure/storage/blob/CloudBlobClientEncryptionTests.java

@@ -23,6 +23,7 @@
 import java.io.FileOutputStream;
 import java.io.IOException;
 import java.net.URISyntaxException;
+import java.nio.charset.Charset;
 import java.security.InvalidAlgorithmParameterException;
 import java.security.InvalidKeyException;
 import java.security.NoSuchAlgorithmException;
@@ -143,7 +144,41 @@ else if (type == BlobType.APPEND_BLOB) {
         TestHelper.assertStreamsAreEqualAtIndex(stream, new ByteArrayInputStream(outputStream.toByteArray()), 0, 0, 
                 size, 2 * 1024);
     }
-
+
+    @Test
+    public void testDownloadUnencryptedBlobWithEncryptionPolicy() throws StorageException, IOException, URISyntaxException, NoSuchAlgorithmException
+    {
+        String blobName = BlobTestHelper.generateRandomBlobNameWithPrefix("test");
+        CloudBlockBlob blob = container.getBlockBlobReference(blobName);
+        blob.deleteIfExists();
+
+        byte[] msg = "my message".getBytes();
+        // Upload data without encryption
+        blob.uploadFromByteArray(msg, 0, msg.length);
+
+        // Create options with encryption policy
+        BlobRequestOptions options = new BlobRequestOptions();
+        options.setEncryptionPolicy(new BlobEncryptionPolicy(new RsaKey("myKey", 1024), null));
+        options.setRequireEncryption(true);
+
+        try {
+            blob.downloadText(Charset.defaultCharset().name(), null, options, null);
+            fail("Expect exception");
+        }
+        catch (StorageException e) {
+            assertEquals(SR.ENCRYPTION_DATA_NOT_PRESENT_ERROR, e.getMessage());
+        }
+
+        byte[] buffer = new byte[msg.length];
+        try {
+            blob.downloadRangeToByteArray(0, (long) buffer.length, buffer, 0, null, options, null);
+            fail("Expect exception");
+        }
+        catch (StorageException e) {
+            assertEquals(SR.ENCRYPTION_DATA_NOT_PRESENT_ERROR, e.getMessage());
+        }
+    }
+
     @Test
     public void testBlobEncryptionWithFile() throws URISyntaxException, StorageException, IOException,
             InvalidKeyException, NoSuchAlgorithmException, NoSuchPaddingException {

microsoft-azure-storage-test/src/com/microsoft/azure/storage/blob/CloudBlobClientTests.java

@@ -247,11 +247,11 @@ public void testSingleBlobPutThresholdInBytes() throws URISyntaxException, Stora
         try {
             bClient.getDefaultRequestOptions().setSingleBlobPutThresholdInBytes(
                     BlobConstants.MAX_SINGLE_UPLOAD_BLOB_SIZE_IN_BYTES + 1);
-            fail("Cannot set upload blob threshold above 64 MB");
+            fail("Cannot set upload blob threshold above 256 MB");
         }
         catch (IllegalArgumentException e) {
             assertEquals(
-                    "The argument is out of range. Argument name: singleBlobPutThresholdInBytes, Value passed: 67108865.",
+                    "The argument is out of range. Argument name: singleBlobPutThresholdInBytes, Value passed: 268435457.",
                     e.getMessage());
         }
 

microsoft-azure-storage-test/src/com/microsoft/azure/storage/blob/CloudBlockBlobTests.java

@@ -33,8 +33,6 @@
 import com.microsoft.azure.storage.file.SharedAccessFilePermissions;
 import com.microsoft.azure.storage.file.SharedAccessFilePolicy;
 
-import junit.framework.Assert;
-
 import org.junit.After;
 import org.junit.Before;
 import org.junit.Test;
@@ -60,33 +58,11 @@
 import java.util.Random;
 import java.util.TimeZone;
 
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.experimental.categories.Category;
-
-import com.microsoft.azure.storage.AccessCondition;
-import com.microsoft.azure.storage.Constants;
-import com.microsoft.azure.storage.NameValidator;
-import com.microsoft.azure.storage.OperationContext;
-import com.microsoft.azure.storage.RetryNoRetry;
-import com.microsoft.azure.storage.SendingRequestEvent;
-import com.microsoft.azure.storage.StorageCredentialsAnonymous;
-import com.microsoft.azure.storage.StorageCredentialsSharedAccessSignature;
 import com.microsoft.azure.storage.StorageErrorCodeStrings;
-import com.microsoft.azure.storage.StorageEvent;
-import com.microsoft.azure.storage.StorageException;
 import com.microsoft.azure.storage.TestRunners.CloudTests;
 import com.microsoft.azure.storage.TestRunners.DevFabricTests;
 import com.microsoft.azure.storage.TestRunners.DevStoreTests;
 import com.microsoft.azure.storage.TestRunners.SlowTests;
-import com.microsoft.azure.storage.core.Utility;
-import com.microsoft.azure.storage.file.CloudFile;
-import com.microsoft.azure.storage.file.CloudFileShare;
-import com.microsoft.azure.storage.file.FileProperties;
-import com.microsoft.azure.storage.file.FileTestHelper;
-import com.microsoft.azure.storage.file.SharedAccessFilePermissions;
-import com.microsoft.azure.storage.file.SharedAccessFilePolicy;
 
 import static org.junit.Assert.*;
 
@@ -1018,6 +994,44 @@ public void testLargeBlobUploadFromStreamTest() throws URISyntaxException, Stora
         blockBlobRef.download(dstStream);
         BlobTestHelper.assertStreamsAreEqual(srcStream, new ByteArrayInputStream(dstStream.toByteArray()));
     }
+
+    @Test
+    @Category({ DevFabricTests.class, DevStoreTests.class, SlowTests.class })
+    public void testLargeSinglePutBlobTest() throws URISyntaxException, StorageException, IOException {
+        final String blockBlobName = BlobTestHelper.generateRandomBlobNameWithPrefix("testBlockBlob");
+        final String blockBlobName2 = BlobTestHelper.generateRandomBlobNameWithPrefix("testBlockBlob");
+        final String blockBlobName3 = BlobTestHelper.generateRandomBlobNameWithPrefix("testBlockBlob");
+        final CloudBlockBlob blob = this.container.getBlockBlobReference(blockBlobName);
+        final CloudBlockBlob blob2 = this.container.getBlockBlobReference(blockBlobName2);
+        final CloudBlockBlob blob3 = this.container.getBlockBlobReference(blockBlobName3);
+        BlobRequestOptions options = new BlobRequestOptions();
+        options.setStoreBlobContentMD5(false);
+        options.setEncryptionPolicy(null);
+        try
+        {
+            byte[] buffer = BlobTestHelper.getRandomBuffer(256 * Constants.MB);
+
+            OperationContext operationContext = new OperationContext();
+
+            blob.uploadFromByteArray(buffer, 0, 128 * Constants.MB, null, null, operationContext);
+            assertEquals(1, operationContext.getRequestResults().size());
+
+            options.setSingleBlobPutThresholdInBytes(256 * Constants.MB);
+            blob2.uploadFromByteArray(buffer, 0, 256 * Constants.MB, null, options, operationContext);
+            assertEquals(1, operationContext.getRequestResults().size());
+
+            // Reduce threshold and upload data greater than the single put blob upload threshold
+            options.setSingleBlobPutThresholdInBytes(Constants.MB);
+            blob3.uploadFromByteArray(buffer, 0, 3 * Constants.MB, null, options, operationContext);
+            assertTrue(operationContext.getRequestResults().size() > 1);
+        }
+        finally
+        {
+            blob.delete();
+            blob2.delete();
+            blob3.delete();
+        }
+    }
 
     @Test
     @Category({ DevFabricTests.class, DevStoreTests.class })
@@ -1159,6 +1173,29 @@ public void testBlobUploadWithoutMD5Validation() throws URISyntaxException, Stor
         assertEquals("MDAwMDAwMDA=", blockBlobRef2.properties.getContentMD5());
     }
 
+    @Test
+    @Category({ DevFabricTests.class, DevStoreTests.class })
+    public void testVerifyTransactionalMD5ValidationMissingOverallMD5() throws URISyntaxException, StorageException, IOException {
+        final String blockBlobName = BlobTestHelper.generateRandomBlobNameWithPrefix("testBlockBlob");
+        final CloudBlockBlob blockBlobRef = this.container.getBlockBlobReference(blockBlobName);
+
+        final int length = 2 * 1024 * 1024;
+        ByteArrayInputStream srcStream = BlobTestHelper.getRandomDataStream(length);
+        BlobRequestOptions options = new BlobRequestOptions();
+        options.setSingleBlobPutThresholdInBytes(1024*1024);
+        options.setDisableContentMD5Validation(true);
+        options.setStoreBlobContentMD5(false);
+
+        blockBlobRef.upload(srcStream, -1, null, options, null);
+
+        options.setDisableContentMD5Validation(false);
+        options.setStoreBlobContentMD5(true);
+        options.setUseTransactionalContentMD5(true);
+        final CloudBlockBlob blockBlobRef2 = this.container.getBlockBlobReference(blockBlobName);
+        blockBlobRef2.downloadRange(1024, (long)1024, new ByteArrayOutputStream(), null, options, null);
+        assertNull(blockBlobRef2.getProperties().getContentMD5());
+    }
+
     @Test
     @Category({ DevFabricTests.class, DevStoreTests.class })
     public void testBlockBlobUploadContentMD5() throws URISyntaxException, StorageException, IOException {
@@ -1198,7 +1235,7 @@ public void eventOccurred(SendingRequestEvent eventArg) {
             }
         };
 
-        length = 33 * Constants.MB;
+        length = BlobConstants.DEFAULT_SINGLE_BLOB_PUT_THRESHOLD_IN_BYTES + 1;
         srcStream = BlobTestHelper.getRandomDataStream(length);
 
         sendingRequestEventContext.getSendingRequestEventHandler().addListener(event);
@@ -1709,6 +1746,23 @@ public void testBlobConditionalAccess() throws StorageException, IOException, UR
         newETag = blob.getProperties().getEtag();
         assertFalse("ETage should be modified on write metadata", newETag.equals(currentETag));
     }
+
+    @Test
+    public void testBlobExceedMaxRange() throws URISyntaxException, StorageException, IOException
+    {
+        CloudBlockBlob blob = container.getBlockBlobReference("blockblob4");
+        blob.deleteIfExists();
+
+        byte[] msg = "my message".getBytes("UTF-8");
+        blob.uploadFromByteArray(msg, 0, msg.length);
+
+        byte[] buffer = new byte[msg.length + 5];
+
+        blob.downloadRangeToByteArray(0, (long) buffer.length, buffer, 0, null, null, null);
+        String expected = new String (msg, "UTF-8");
+        String actual = new String(buffer, "UTF-8").substring(0, 10);
+        assertEquals(expected, actual);
+    }
 
     @Test
     @Category({ DevFabricTests.class, DevStoreTests.class })