upload-on-s3-with-jclouds
Upload on S3 with the jclouds library
There are several good ways to upload content to an S3 bucket in the
Java world – in this article we’ll look at what
the jclouds library provides for this purpose.
To use jclouds – specifically the APIs discussed in this article, this
simple
Maven dependency should be added to the pom of the project:
<dependency>
<groupId>org.jclouds</groupId>
<artifactId>jclouds-allblobstore</artifactId>
<version>1.5.10</version>
</dependency>1. Uploading to Amazon S3
The first step, in order to access any of these APIs, is to create a
BlobStoreContext:
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(BlobStoreContext.class);This represents the entry-point to a general key-value storage service,
such as Amazon S3 – but not limited to it.
For the more specific S3 only implementation, the context can be created
similarly:
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(S3BlobStoreContext.class);And even more specifically:
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(AWSS3BlobStoreContext.class);When the authenticated context is no longer needed, closing it is
required to release all resources – threads and connections – associated
to it.
2. The four S3 APIs of jclouds
The jclouds library provides four different APIs to upload content to S3
bucket, ranging from simple but inflexible to complex and powerful, all
obtained via the BlobStoreContext. Let’s start with the simplest.
2.1. Upload via the Map API
The easiest way jclouds can be used to interact with an S3 bucket is by
representing that bucket as a Map. The API is obtained from the context:
InputStreamMap bucket = context.createInputStreamMap("bucketName");Then, to upload a simple HTML file:
bucket.putString("index1.html", "<html><body>hello world1</body></html>");The InputStreamMap API exposes several other types of PUT operations –
files, raw bytes – both for single and bulk.
A simple integration test can be used as an example:
@Test
public void whenFileIsUploadedToS3WithMapApi_thenNoExceptions() {
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(AWSS3BlobStoreContext.class);
InputStreamMap bucket = context.createInputStreamMap("bucketName");
bucket.putString("index1.html", "<html><body>hello world1</body></html>");
context.close();
}2.2. Upload via BlobMap
Using the simple Map API is straightforward but ultimately limited – for
example, there is no way to pass in metadata about the content being
uploaded. When more flexibility and customization is necessary, this
simplified approach to uploading data to S3 via a Map is no longer
enough.
The next API we’ll look at is the Blob Map API – this is obtained from
the context:
BlobMap bucket = context.createBlobMap("bucketName");The API allows the client to access more lower level details, such as
Content–Length, Content-Type, Content-Encoding, eTag hash and
others; to upload new content in the bucket:
Blob blob = bucket.blobBuilder().name("index2.html").
payload("<html><body>hello world2</body></html>").
contentType("text/html").calculateMD5().build();The API also allows setting a variety of payloads on the create request.
A simple integration test for uploading a basic HTML file to S3 via the
Blob Map API:
@Test
public void whenFileIsUploadedToS3WithBlobMap_thenNoExceptions() throws IOException {
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(AWSS3BlobStoreContext.class);
BlobMap bucket = context.createBlobMap("bucketName");
Blob blob = bucket.blobBuilder().name("index2.html").
payload("<html><body>hello world2</body></html>").
contentType("text/html").calculateMD5().build();
bucket.put(blob.getMetadata().getName(), blob);
context.close();
}2.3. Upload via BlobStore
The previous APIs had no way to upload content using multipart upload
– this makes them ill suited when working with large files. This
limitation is addressed by the next API we’re going to look at – the
synchronous BlobStore API.
This is obtained from the context:
BlobStore blobStore = context.getBlobStore();To use the multipart support and upload a file to S3:
Blob blob = blobStore.blobBuilder("index3.html").
payload("<html><body>hello world3</body></html>").contentType("text/html").build();
blobStore.putBlob("bucketName", blob, PutOptions.Builder.multipart());The payload builder is the same one that was being used by the BlobMap
API, so the same flexibility in specifying lower level metadata
information about the blob is available here. The difference is the
PutOptions supported by the PUT operation of the API – namely the
multipart support.
The previous integration test now has multipart enabled:
@Test
public void whenFileIsUploadedToS3WithBlobStore_thenNoExceptions() {
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(AWSS3BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
Blob blob = blobStore.blobBuilder("index3.html").
payload("<html><body>hello world3</body></html>").contentType("text/html").build();
blobStore.putBlob("bucketName", blob, PutOptions.Builder.multipart());
context.close();
}2.4. Upload via AsyncBlobStore
While the previous BlobStore API was synchronous, there is also an
asynchronous API for BlobStore – AsyncBlobStore. The API is
similarly obtained from the context:
AsyncBlobStore blobStore = context.getAsyncBlobStore();The only difference between the two is that the async API is returning
ListenableFuture for the PUT asynchronous operation:
Blob blob = blobStore.blobBuilder("index4.html").
.payload("<html><body>hello world4</body></html>").build();
blobStore.putBlob("bucketName", blob)<strong>.get()</strong>;The integration test displaying this operation is similar to the
synchronous one:
@Test
public void whenFileIsUploadedToS3WithBlobStore_thenNoExceptions() {
BlobStoreContext context =
ContextBuilder.newBuilder("aws-s3").credentials(identity, credentials)
.buildView(AWSS3BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
Blob blob = blobStore.blobBuilder("index4.html").
payload("<html><body>hello world4</body></html>").contentType("text/html").build();
Future<String> putOp = blobStore.putBlob("bucketName", blob, PutOptions.Builder.multipart());
putOp.get();
context.close();
}3. Conclusion
In this article, we analyzed the four APIs that the jclouds library
provides to upload content to Amazon S3. These four APIs are generic
and they work with other key-value storage services as well – such as
Microsoft Azure Storage for example.
In the next article we’ll look at the Amazon specific S3 API available
in jclouds – the AWSS3Client. We’ll implement the operation of
uploading a large file, dynamically calculate the optimal number of
parts for any given file, and perform the upload of all parts in
parallel.
