107 lines
19 KiB
HTML
107 lines
19 KiB
HTML
<html><head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
|
|
<title>9. Resource handling</title><link rel="stylesheet" type="text/css" href="css/manual-multipage.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"><link rel="home" href="multi_spring-cloud-aws.html" title="Spring Cloud AWS"><link rel="up" href="multi_spring-cloud-aws.html" title="Spring Cloud AWS"><link rel="prev" href="multi__sending_mails.html" title="8. Sending mails"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">9. Resource handling</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="multi__sending_mails.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> </td></tr></table><hr></div><div class="chapter"><div class="titlepage"><div><div><h1 class="title"><a name="_resource_handling" href="#_resource_handling"></a>9. Resource handling</h1></div></div></div><p>The Spring Framework provides a <code class="literal">org.springframework.core.io.ResourceLoader</code> abstraction to load files from the filesystem,
|
|
servlet context and the classpath. Spring Cloud AWS adds support for the <a class="link" href="http://aws.amazon.com/s3/" target="_top">Amazon S3</a> service
|
|
to load and write resources with the resource loader and the <code class="literal">s3</code> protocol.</p><p>The resource loader is part of the context module, therefore no additional dependencies are necessary to use the resource
|
|
handling support.</p><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_configuring_the_resource_loader" href="#_configuring_the_resource_loader"></a>9.1 Configuring the resource loader</h2></div></div></div><p>Spring Cloud AWS does not modify the default resource loader unless it encounters an explicit configuration with an XML namespace element.
|
|
The configuration consists of one element for the whole application context that is shown below:</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"><beans</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-attribute">xmlns:xsi</span>=<span xmlns:d="http://docbook.org/ns/docbook" class="hl-value">"http://www.w3.org/2001/XMLSchema-instance"</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-attribute">xmlns:aws-context</span>=<span xmlns:d="http://docbook.org/ns/docbook" class="hl-value">"http://www.springframework.org/schema/cloud/aws/context"</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-attribute">xsi:schemaLocation</span>=<span xmlns:d="http://docbook.org/ns/docbook" class="hl-value">"http://www.springframework.org/schema/cloud/aws/context
|
|
http://www.springframework.org/schema/cloud/aws/context/spring-cloud-aws-context.xsd"</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag">></span>
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"><aws-context:context-credentials></span>
|
|
...
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"></aws-context:context-credentials></span>
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"><aws-context:context-resource-loader/></span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"></beans></span></pre></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_downloading_files" href="#_downloading_files"></a>9.2 Downloading files</h2></div></div></div><p>Downloading files can be done by using the <code class="literal">s3</code> protocol to reference Amazon S3 buckets and objects inside their bucket. The
|
|
typical pattern is <code class="literal">s3://<bucket>/<object></code> where bucket is the global and unique bucket name and object is a valid object
|
|
name inside the bucket. The object name can be a file in the <span class="emphasis"><em>root</em></span> folder of a bucket or a nested file within a directory
|
|
inside a bucket.</p><p>The next example demonstrates the use of the resource loader to load different resources.</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">public</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">class</span> SimpleResourceLoadingBean {
|
|
|
|
<em><span class="hl-annotation" style="color: gray">@Autowired</span></em>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">private</span> ResourceLoader resourceLoader;
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">public</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">void</span> resourceLoadingMethod() <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">throws</span> IOException {
|
|
Resource resource = <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">this</span>.resourceLoader.getResource(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"s3://myBucket/rootFile.log"</span>);
|
|
Resource secondResource = <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">this</span>.resourceLoader.getResource(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"s3://myBucket/rootFolder/subFile"</span>);
|
|
|
|
InputStream inputStream = resource.getInputStream();
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">//read file</span>
|
|
}
|
|
}</pre></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_uploading_files" href="#_uploading_files"></a>9.3 Uploading files</h2></div></div></div><p>Since Spring Framework 3.1 the resource loader can also be used to upload files with the <code class="literal">org.springframework.core.io.WritableResource</code>
|
|
interface which is a specialization of the <code class="literal">org.springframework.core.io.ResourceLoader</code> interface. Clients can upload files
|
|
using the <code class="literal">WritableResource</code> interface. The next example demonstrates an upload of a resource using the resource loader.</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">public</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">class</span> SimpleResourceLoadingBean {
|
|
|
|
<em><span class="hl-annotation" style="color: gray">@Autowired</span></em>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">private</span> ResourceLoader resourceLoader;
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">public</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">void</span> writeResource() <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">throws</span> IOException {
|
|
Resource resource = <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">this</span>.resourceLoader.getResource(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"s3://myBucket/rootFile.log"</span>);
|
|
WritableResource writableResource = (WritableResource) resource;
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">try</span> (OutputStream outputStream = writableResource.getOutputStream()) {
|
|
outputStream.write(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"test"</span>.getBytes());
|
|
}
|
|
}
|
|
}</pre><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_uploading_multi_part_files" href="#_uploading_multi_part_files"></a>9.3.1 Uploading multi-part files</h3></div></div></div><p>Amazon S3 supports <a class="link" href="http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html" target="_top">multi-part uploads</a> to
|
|
increase the general throughput while uploading. Spring Cloud AWS by default only uses one thread to upload the files and
|
|
therefore does not provide parallel upload support. Users can configure a custom <code class="literal">org.springframework.core.task.TaskExecutor</code>
|
|
for the resource loader. The resource loader will queue multiple threads at the same time to use parallel multi-part uploads.</p><p>The configuration for a resource loader that uploads with 10 Threads looks like the following</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"><beans</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-attribute">...></span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-attribute"><aws-context:context-resource-loader</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-attribute">task-executor</span>=<span xmlns:d="http://docbook.org/ns/docbook" class="hl-value">"executor"</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"> /></span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"><task:executor</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-attribute">id</span>=<span xmlns:d="http://docbook.org/ns/docbook" class="hl-value">"executor"</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-attribute">pool-size</span>=<span xmlns:d="http://docbook.org/ns/docbook" class="hl-value">"10"</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-attribute">queue-capacity</span>=<span xmlns:d="http://docbook.org/ns/docbook" class="hl-value">"0"</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-attribute">rejection-policy</span>=<span xmlns:d="http://docbook.org/ns/docbook" class="hl-value">"CALLER_RUNS"</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"> /></span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"></beans></span></pre><div class="warning" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Warning"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Warning]" src="images/warning.png"></td><th align="left">Warning</th></tr><tr><td align="left" valign="top"><p>Spring Cloud AWS consumes up to 5 MB (at a minimum) of memory per thread. Therefore each parallel thread will incur
|
|
a memory footprint of 5 MB in the heap, and a thread size of 10 will consume therefore up to 50 mb of heap space. Spring Cloud
|
|
AWS releases the memory as soon as possible. Also, the example above shows that there is no <code class="literal">queue-capacity</code> configured,
|
|
because queued requests would also consume memory.</p></td></tr></table></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_uploading_with_the_transfermanager" href="#_uploading_with_the_transfermanager"></a>9.3.2 Uploading with the TransferManager</h3></div></div></div><p>The Amazon SDK also provides a high-level abstraction that is useful to upload files, also with multiple threads using
|
|
the multi-part functionality. A <code class="literal">com.amazonaws.services.s3.transfer.TransferManager</code> can be easily created in the application
|
|
code and injected with the pre-configured <code class="literal">com.amazonaws.services.s3.AmazonS3</code> client that is already created
|
|
with the Spring Cloud AWS resource loader configuration.</p><p>This example shows the use of the <code class="literal">transferManager</code> within an application to upload files from the hard-drive.</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">public</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">class</span> SimpleResourceLoadingBean {
|
|
|
|
<em><span class="hl-annotation" style="color: gray">@Autowired</span></em>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">private</span> AmazonS3 amazonS3;
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">public</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">void</span> withTransferManager() {
|
|
TransferManager transferManager = <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">new</span> TransferManager(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">this</span>.amazonS3);
|
|
transferManager.upload(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"myBucket"</span>,<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"filename"</span>,<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">new</span> File(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"someFile"</span>));
|
|
}
|
|
}</pre></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_searching_resources" href="#_searching_resources"></a>9.4 Searching resources</h2></div></div></div><p>The Spring resource loader also supports collecting resources based on an Ant-style path specification. Spring Cloud AWS
|
|
offers the same support to resolve resources within a bucket and even throughout buckets. The next example shows the resource
|
|
resolution by using different patterns.</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">public</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">class</span> SimpleResourceLoadingBean {
|
|
|
|
<em><span class="hl-annotation" style="color: gray">@Autowired</span></em>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">private</span> ResourcePatternResolver resourcePatternResolver;
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">public</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">void</span> resolveAndLoad() <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">throws</span> IOException {
|
|
Resource[] allTxtFilesInFolder = <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">this</span>.resourcePatternResolver.getResources(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"s3://bucket/name/*.txt"</span>);
|
|
Resource[] allTxtFilesInBucket = <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">this</span>.resourcePatternResolver.getResources(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"s3://bucket/**/*.txt"</span>);
|
|
Resource[] allTxtFilesGlobally = <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">this</span>.resourcePatternResolver.getResources(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"s3://**/*.txt"</span>);
|
|
}
|
|
}</pre><div class="warning" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Warning"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Warning]" src="images/warning.png"></td><th align="left">Warning</th></tr><tr><td align="left" valign="top"><p>Resolving resources throughout all buckets can be very time consuming depending on the number of buckets a user owns.</p></td></tr></table></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_using_cloudformation_3" href="#_using_cloudformation_3"></a>9.5 Using CloudFormation</h2></div></div></div><p>CloudFormation also allows to create buckets during stack creation. These buckets will typically have a generated name
|
|
that must be used as the bucket name. In order to allow application developers to define <span class="emphasis"><em>static</em></span> names inside their
|
|
configuration, Spring Cloud AWS provides support to resolve the generated bucket names.
|
|
Application developers can use the <code class="literal">org.springframework.cloud.aws.core.env.ResourceIdResolver</code> interface to resolve the
|
|
physical names that are generated based on the logical names.</p><p>The next example shows a bucket definition inside a CloudFormation stack template. The bucket will be created with a name
|
|
like <span class="emphasis"><em>integrationteststack-sampleBucket-23qysofs62tc2</em></span></p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">{</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"Resources"</span>: <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">{</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"sampleBucket"</span>: <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">{</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"Type"</span>: <span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"AWS::S3::Bucket"</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">}</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">}</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">}</span></pre><p>Application developers can resolve that name and use it to load resources as shown in the next example below.</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">public</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">class</span> SimpleResourceLoadingBean {
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">private</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">final</span> ResourceLoader loader;
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">private</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">final</span> ResourceIdResolver idResolver;
|
|
|
|
<em><span class="hl-annotation" style="color: gray">@Autowired</span></em>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">public</span> SimpleResourceLoadingBean(ResourceLoader loader, ResourceIdResolver idResolver) {
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">this</span>.loader = loader;
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">this</span>.idResolver = idResolver;
|
|
}
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">public</span> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">void</span> resolveAndLoad() {
|
|
String sampleBucketName = <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">this</span>.idResolver.
|
|
resolveToPhysicalResourceId(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"sampleBucket"</span>);
|
|
Resource resource = <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">this</span>.loader.
|
|
getResource(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"s3://"</span> + sampleBucketName + <span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"/test"</span>);
|
|
}
|
|
}</pre></div></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="multi__sending_mails.html">Prev</a> </td><td width="20%" align="center"> </td><td width="40%" align="right"> </td></tr><tr><td width="40%" align="left" valign="top">8. Sending mails </td><td width="20%" align="center"><a accesskey="h" href="multi_spring-cloud-aws.html">Home</a></td><td width="40%" align="right" valign="top"> </td></tr></table></div></body></html> |