Uploading To S3 With AngularJS
Share
A little while back I found myself needing to handle file uploads without touching the application server. This is a common scenario for people using Heroku as they limit all requests to 30 seconds, which in theory sounds fine for most requests, but destroys any chance you have at directly handling file uploads within your application.
I restricted my allowed attachment size down to 2MB but still there were people in more remote parts of the world who were hitting that 30 second timeout. Heroku’s response is to simply not upload to the app server and instead go direct to something like AWS S3.
This got me thinking – can I upload a large file to Amazon S3 using the AWS-JS-SDK with JavaScript and work it into my existing AngularJS application?
As it turns out, yes you can and just in case you want to jump ahead and skip the blog post, I’ve put up a sample application to demonstrate it along with the full source code here;
See A Live Demo Here
http://cheynewallace.github.io/angular-s3-upload/
Full Source Code And Sample Project Here
https://github.com/cheynewallace/angular-s3-upload
UPDATE – July 14th 2016
This is been a very popular article for several years now, how ever many people have asked how to avoid using the restricted public keys method shown in this article. I have created a new post based on this original one that explains how to use Pre-Signed URLs instead.
If you are creating a new public web application you should use this new post as a guide and consider this original one deprecated
Please see
Uploading To S3 With AngularJS and Pre-Signed URLs
Scenario
The scenario we’re going to build for here will be to upload a file (of any size) directly to AWS S3 into a temporary bucket that we will access using a restricted and public IAM account.
The purpose of this front end application will be to get files into AWS S3, using only JavaScript libraries from our browser.
We can then kick off a background process with our application server to process these uploaded files at a later time.
A good example of this is having a user upload a file from a web form for which your application server will then pull back down, encrypt or resize before pushing it back into a more permanent bucket for storage.
By uploading directly to S3 we will be taking load off our application server by not keeping long running connections open while slow clients upload large files. A problem which is especially visible when using services like Heroku.
Step 1: Add The AWS JS SDK To Your Project
This is Amazons JavaScript SDK and it won’t take long before you notice the file size of this library. It’s pretty obvious that this is more of a backend NodeJS library than a browser JS library, even though they have a “browser” version, it still weighs in at about 230KB.
The easiest way to get this library is to simply “bower install aws-sdk-js” and you’ll get the latest “browser” version.
If you’re already feeling the weight from too many libraries on your app, you can clone the NodeJS repo and compile it your self which I found saved me about 50KB in file size.
More info can be found on the AWS Docs under Compiling The AWS SDK
You can compile and minify just the S3 component of the SDK from the NodeJS modules using the following command;
MINIFY=1 node dist-tools/browser-builder.js s3 > aws-sdk.min.js
You can also just use my compiled custom version found here
Step 2: Add the file directive
The file directive simply takes the attributes from a file input type and binds it to the $scope.file object so you can easily work with the filename, file size etc from your controllers.
As soon as you select a file, you can access $scope.file.name or $scope.file.size to get the filename and size for handling client side validation and unique S3 object names.
This means you can validate the file size from the browser with something simple like;
if($scope.file.size > 10585760) {
alert('Sorry, file size must be under 10MB');
return false;
}
Go ahead and include the following directive in your project;
directives.directive('file', function() {
return {
restrict: 'AE',
scope: {
file: '@'
},
link: function(scope, el, attrs){
el.bind('change', function(event){
var files = event.target.files;
var file = files[0];
scope.file = file;
scope.$parent.file = file;
scope.$apply();
});
}
};
});
Step 3: Setup The AWS Credentials
You’re probably wondering how to lock down this new upload functionality considering it’s all JavaScript. There is a few ways you can do this, those being by use of Pre-Signed URL’s or by creating a “public” IAM account.
The use of Pre-Signed URL’s involves making a call to your application server first and retrieving a “pre signed” location for the upload which your server and AWS negotiate on the fly. To keep things simple, we’re not going to use this method today.
The second method, which is the one we’re going to use today is by using a public IAM account, by that I mean a regular IAM account that is heavily restricted to do only 1 particular function, in this case, it will only have permission to PUT files into a particular AWS Bucket and nothing else.
This users API key will be public, so anyone will be able to upload to your bucket if they use this key, which is why we will want to configure the bucket to expire all objects within 24 hours, so even if someone did try and upload a 10 Gigabyte file to screw with you, it would only sit there for a few hours. We will also configure CORS which will prevent people uploading content from anywhere other than your website (more on that later)
Once you upload a file to this temporary bucket from your application, you will want to ping your application server with details of the new file and move it into a new permanent bucket. It’s right here where you will be able to perform any transformations, encryption, resizing or processing.
Create The User
Go into your AWS console and visit the “Security Credentials” section. Create a new user and call it something like “app_public”. Make sure you download the key information when it is presented, this is what we’ll be feeding into our app later to upload with.
Under the permissions section, click “attach a new policy“, then select the policy generator.
Select Amazon S3 as the service and only select the PutObject action from the drop down list.
The ARN is an Amazon Resource Name. This is going to look like;
arn:aws:s3:::your_bucket_name
Click “add statement”, then save and apply policy. Now your user has write-only access to the bucket.
Your policy is going to look something like this;
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt126637111000",
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::your_bucket_name"
]
}
]
}
Step 4: Configure CORS And Expiry On The Bucket
CORS or “Cross Origin Resource Sharing” allows us to restrict the operations that can be performed on a bucket to a specific domain, like your websites domain. Typically a CORS Ajax request will first initiate an OPTIONS HTTP request to the server which will return the allowed options for that endpoint before the real Ajax request actually happens. Think of it like an access request, the server will inspect where the request originated from and return a set of allowed options (or none) for that origin.
Don’t worry, you won’t have to make that request your self, Angular will handle all of that for you, but it’s good to have a basic understanding of whats happening during the lifetime of the request.
Add The CORS Policy
From Your AWS console, under S3 click into your bucket then click the Properties button. There you will see a “Add CORS Configuration” button. It’s here that you’ll configure your bucket to only allow PUT requests from particular origins.
You can use the following sample config – just edit to reflect your development, production and staging environments.
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>http://localhost:3000</AllowedOrigin>
<AllowedOrigin>https://www.yourdomain.com</AllowedOrigin>
<AllowedOrigin>http://staging.yourdomain.com</AllowedOrigin>
<AllowedMethod>PUT</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<ExposeHeader>x-amz-server-side-encryption</ExposeHeader>
<ExposeHeader>x-amz-request-id</ExposeHeader>
<ExposeHeader>x-amz-id-2</ExposeHeader>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
It’s a good idea to split these into other buckets, but for simplicity we’ll just use the one bucket.
Configure Object Expiry
It’s a good idea to expire the objects in this bucket after some short period to prevent people from just uploading huge objects to screw with you. Your server side code should handle moving and deleting valid files so you can assume those that are left after 24 hours are not meant to be there.
From your S3 console, view a bucket and then click Properties, expand the “Lifecycle Rules” section and follow the prompts. Set the action to “Permanently Delete Only” and set it for 1 day which will delete any objects in the bucket that are older than 1 day permanently.
Now you’re ready to lay down some code.
Step 5: Write Some Angular
It’s worth mentioning again that there is a complete sample application available here that you can pull down and play around with or a live demo application here that you can try out. Simply provide the bucket name, access key and secret access key in the form to upload.
So, if we’ve configured everything mentioned above correctly, we’re ready to see this in action. The following snippet is a simplified version of the sample application I mentioned earlier. It does the basic operations we need to upload a file to S3, which includes:
- Configure the AWS S3 credentials and bucket object
- Check to make sure a file is selected
- PUT the object into the S3 bucket whilst displaying progress information to the console
- Alert with any errors or config issues.
Here’s what the method in your controller is going to look like
$scope.creds = {
bucket: 'your_bucket',
access_key: 'your_access_key',
secret_key: 'your_secret_key'
}
$scope.upload = function() {
// Configure The S3 Object
AWS.config.update({ accessKeyId: $scope.creds.access_key, secretAccessKey: $scope.creds.secret_key });
AWS.config.region = 'us-east-1';
var bucket = new AWS.S3({ params: { Bucket: $scope.creds.bucket } });
if($scope.file) {
var params = { Key: $scope.file.name, ContentType: $scope.file.type, Body: $scope.file, ServerSideEncryption: 'AES256' };
bucket.putObject(params, function(err, data) {
if(err) {
// There Was An Error With Your S3 Config
alert(err.message);
return false;
}
else {
// Success!
alert('Upload Done');
}
})
.on('httpUploadProgress',function(progress) {
// Log Progress Information
console.log(Math.round(progress.loaded / progress.total * 100) + '% done');
});
}
else {
// No File Selected
alert('No File Selected');
}
}
and the file input element using the file directive
<input name="file" type="file" file />
The $scope.upload method here could be broken out into a service or factory to clean things up a little, but you could also just drop this method into your controller and with a few minor tweaks be up and running.
The upload is broken down into parts and uploaded one piece at a time.
The ‘httpUploadProgress’ event is fired after each time a part has finished uploading which is where we can update our progress bars or percentage counters for a more aesthetic UI / UX.
In the snippet above we’re simply logging this to the console, but in the sample application I mentioned earlier i’v used a bootstrap progress bar to indicate the overall progress.
Adding Folders
If you want to arrange the uploads into folders, you can do this by simply adding the folder name to the end of the bucket name. So for example, setting $scope.creds.bucket to “MYBUCKET/user1” would upload the file into the MYBUCKET bucket under the folder user1.
putObject Configuration
The putObject params object can hold a lot more configuration than what is shown in this example. Setting content expiry, content type and ACL information is just a few examples of what can be done by adding attributes to the params object. You can read more about configuring these here
A more comprehensive version of the above code, including file size validation, unique file names and proper notifications can be seen here: https://github.com/cheynewallace/angular-s3-upload/blob/master/js/controllers.js
Step 6: Processing The Upload
This step is really going to depend on what you want to do with the file and is going to vary depending on your application server but, generally speaking, now all you need is the path to the file in S3 (which is basically just the bucket name plus the object name) and you can pull the file down, process it and push it back to another location from the server in some sort of background process that doesn’t tie up the front end.
I won’t expand on this too much as this article was more focused on uploading using purely JavaScript but you can see how we’re able to POST details of this files new location in S3 fairly easily to the server.
$scope.s3_path = $scope.creds.bucket + '/' + $scope.file.name; // mybucket/document.pdf
As an example, in Ruby On Rails, using the AWS SDK for Ruby you could pull the file down, transform it and push it back up to another bucket with something like this
# Get The Temporary Upload
s3 = AWS::S3.new
temp_obj = s3.buckets['YOUR_TEMP_BUCKET'].objects[params[:uploaded_file]]
begin
# Read File Size From S3 For Server Size Validation
size = temp_obj.content_length
# Assign A Local Temp File
local_file = "#{Rails.root}/tmp/#{params[:uploaded_file]}"
# Read In File From S3 To Local Path
File.open(local_file, 'wb') do |file|
temp_obj.read do |chunk|
file.write(chunk)
end
end
#########################################
# Perform Some Local Transformation Here
#########################################
# Now Write Back The Transformed File
perm_obj = s3.buckets['YOUR_PERMANENT_BUCKET'].objects[params[:uploaded_file]]
perm_obj.write(File.open(local_file))
rescue StandardError => exception
# That's A Fail
ensure
# Delete The Original File
temp_obj.delete
end
end
Summary
We’ve seen now how we can upload files directly to AWS S3 using only JavaScript. It may seem like a lot of work from the first few steps in this article, but they are necessary in order to prevent people abusing your S3 bucket and your app so I would avoid creating an open bucket or being lazy with the IAM policy.
I’v been using this technique for a while now and it’s been pretty solid. It certainly solved my issue with Heroku H12 timeouts which was causing me endless headaches.
Have any suggestions how to improve on this technique? Let me know in the comments section below