Android upload to Amazon S3

Willem Vermeer,one of our main community members, wrote this piece while strugglingwith our open source Android application. Let’s go!Programming for Android devices can be a lot of fun but every now andyou’re faced with a task which seems simple at first glance but gets you hitting a few walls before you finally find a satisfying solution. This time the requirement for me was to upload a file from an android device to a bucket in the Amazon Simple Storage Service (S3). The progress of the file upload should be accurately visualized by a linear progressbar. Sounds simple right?Well here’s the path I followed. Luckily, this story has a happy ending.Before I got started I needed to read up on how to upload a file toAmazon S3 in the first place. This is pretty well documented in the Amazondeveloper documentation and their gettingstarted docs. You basically need to first sign up for Amazon S3,create a bucket and then perform an Http multipart POST. This POST should go to Inside yourbucket, you are free to create subdirectories by including them in theso-called object key. For example, you can POST a file to images/car.jpgand the image becomes available at far so good. Actually, things are a little more complicated than justPOSTing a file to a certain URL because of policy files and the like butwe leave that out of this discussion for now. How do we perform a HttpPOST from an Android application? We could use the WebView, create anHTML page and POST a form, but then we would have no control over thefile upload progress. Next idea: to use the built-in HttpClient in thepackage org.apache.commons.httpclient. We soon discover that thisHttpClient does not support multipart file upload out of the box. A bitweird, uploading a file through a Http POST seems like a quite regularrequirement to me but by default it’s not included in the java forkpresented to you by Google in the android libraries.After a bit of searching a simple solution presents itself: to include aseparate Apache library HttpMime which does contain the multipart fileupload as described here.I put together some code to test it and all seemed well until I startedreceiving Http error codes from Amazon. As it turns out, the HttpClientdoes not specify the Content-Length header in the POST request. This isa hard requirement imposed by Amazon S3 as described here.So we hit a dead end.HttpClient is really a convenience class, hiding the low levelcomplexity of manually managing an HttpUrlConnection. So if HttpClientdoesn’t do the job for us, we will have to dig one step deeper and workdirectly with an HttpUrlConnection. It means we will have to step bystep construct the multipart request with its boundaries and headers.It’s a dirty job but certainly not impossible. A clean example ofwhat this request should look like is readily available in the Amazondocs.This all works like a charm; the file is uploaded to the Amazon bucket.But wait, we forgot one piece of the requirement: to display a progressbar. No problem because Android contains the cool ProgressBar class. Wecreate an Activity, define a ProgressBar in the layout XML, subclassAsyncTask where we will perform the upload asynchronously and write awhile loop where we send chunks of say 4096 bytes to HttpUrlConnectionand after every chunk we publish the progress to the progress bar andthat’s it! Yes, but of course not quite. It turns out we’ve run into an issue3164 of the pre-froyo Android platform. Thanks to this bug allcontent in the file upload is buffered and only gets sent to the serverat the connection.flush() in one big chunk at the time. Of course thistakes forever with my T-Mobile contract. The progress bar indicates thatthe file upload has almost finished (because it got updated after each4096 byte chunk) but then the waiting starts. I experimented withsetting connection.setChunkedStreamingMode to true but this is notaccepted by Amazon S3 because in that case it violates the requirementwe saw before to mandatorily specify the Content-Length up front.Almost about to give up I got inspired by the movie Inception wherecriminals invade each other’s dreams up to three levels deep. Amazingstuff. Time to sink one level deeper into the HttpUrlConnection byworking directly onto a This proved to be the finalsolution. We open a Socket onto toport 80 and write the multipart POST directly into this socketconnection. In this case we must manually pass the required Http headerswhich we could previously specify through the HttpUrlConnection object.It’s now possible to send a chunk of 4096 bytes, update the progress barand see the real progress. After the final chunk all data has reallybeen sent to the server and the state of the progress bar correctlyreflects the progress of the upload: Done!Now for those of you interested in the details, here we go.First the definition of the ProgressBar in one of your layout XMLs:

<ProgressBar  android:id="@+id/progressBarUpload"  android:layout_width="150dip"  android:layout_height="15dip"  android:layout_centerInParent="true"  android:layout_marginBottom="15dip"  android:layout_marginLeft="10dip"  android:layout_marginRight="10dip"  style="?android:attr/progressBarStyleHorizontal"/>

Note the style attribute, this is the way to explain to Android that youwant a horizontal progress bar instead of a spinning image. Now let’sinflate the ProgressBar inside our Activity:

progressBar = (ProgressBar) findViewById(;progressBar.setMax(100);progressBar.setProgress(0);

Then, when we’re ready to launch the upload task:

new PutOrderFilesTask(orderParams, getApplicationContext(), progressBar, uploadFilesHandler)



…which is a subclass of AsyncTask defined like this:

public class PutOrderFilesTask extends AsyncTask<String, Long, Integer> {

…and we should override the method doInBackground:

@Overrideprotected Integer doInBackground(String... unused) {   Map params = new HashMap();   Uri uri = Uri.parse("the-uri-to-your-file");   params.put("AWSAccessKeyId", "our-aws-access-key");   params.put("Content-Type", "image/jpeg");   params.put("policy", "some-policy-defined-by-yourself");   params.put("Filename", "photo.jpg");   params.put("key", "images/photo.jpg");   params.put("acl", "private");   params.put("signature", "some-signature-defined-by-yourself");   params.put("success_action_status", "201");   try {      HttpRequest.postSocket("",


params,         context.getContentResolver().openInputStream(uri)         fileSize, this, 10, 70, "photo.jpg", "image/jpeg");   } catch (Exception e) {      return -1;   }   return 1;}

The HttpRequest class contains all the low level details of actuallyperforming the upload:

   public class HttpRequest { private static final String boundary = "-----------------------******";


    private static final String newLine = "\r\n";    private static final int maxBufferSize = 4096;    private static final String header =      "POST / HTTP/1.1\n" +  "Host: %s\n" +  "User-Agent: Mozilla/5.0 (Windows; U;   Windows NT 5.1; en-US; rv: Gecko/20071115 Firefox/\n" +  "Accept: text/xml,application/xml,application/xhtml+xml,   text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\n" +  "Accept-Language: en-us,en;q=0.5\n" +  "Accept-Encoding: gzip,deflate\n" +  "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\n" +  "Keep-Alive: 300\n" +  "Connection: keep-alive\n" +  "Content-Type: multipart/form-data; boundary=" + boundary + "\n" +  "Content-Length: %s\n\n"; public static void postSocket(String sUrl, Map params, InputStream stream, long streamLength,       PutOrderFilesTask task, int startProgress, int endProgress, String fileName, String contentType) {      OutputStream writer = null;      BufferedReader reader = null;      Socket socket = null;            try {                int bytesAvailable;                int bufferSize;                int bytesRead;                int totalProgress = endProgress - startProgress;                task.myPublishProgress(new Long(startProgress));                                String openingPart = writeContent(params, fileName, contentType);                String closingPart = newLine + "--" + boundary + "--" + newLine;                long totalLength = openingPart.length() 		+ closingPart.length() + streamLength;                // strip off the leading http:// otherwise the Socket will not work                String socketUrl = sUrl;                if (socketUrl.startsWith("http://")) {                 socketUrl = socketUrl.substring("http://".length());                }                                socket = new Socket(socketUrl, 80);                socket.setKeepAlive(true);             writer = socket.getOutputStream();             reader = new BufferedReader(new 		InputStreamReader(socket.getInputStream()));             writer.write(String.format(header, socketUrl, 		Long.toString(totalLength)).getBytes());                writer.write(openingPart.getBytes());                bytesAvailable = stream.available();                bufferSize = Math.min(bytesAvailable, maxBufferSize);                byte[] buffer = new byte[bufferSize];                bytesRead =, 0, bufferSize);                int readSoFar = bytesRead;                task.myPublishProgress(new Long(startProgress + 		Math.round(totalProgress * readSoFar / streamLength)));                while (bytesRead > 0) {                    writer.write(buffer, 0, bufferSize);                    bytesAvailable = stream.available();                    bufferSize = Math.min(bytesAvailable, maxBufferSize);                    bytesRead =, 0, bufferSize);                    readSoFar += bytesRead;                    task.myPublishProgress(new Long(startProgress + 			Math.round(totalProgress * readSoFar / streamLength)));                }                stream.close();                writer.write(closingPart.getBytes());                Log.d(Cards.LOG_TAG, closingPart);                writer.flush();                                // read the response                String s = reader.readLine();                // do something with response s            } catch (Exception e) {             throw new HttpRequestException(e);            } finally {             if (writer != null) { try { writer.close(); 		writer = null;} catch (Exception ignore) {}}             if (reader != null) { try { reader.close(); 		reader = null;} catch (Exception ignore) {}}             if (socket != null) { try {socket.close(); 		socket = null;} catch (Exception ignore) {}}            }            task.myPublishProgress(new Long(endProgress));        } /**     * Populate the multipart request parameters      * into one large stringbuffer which will later allow us to      * calculate the content-length header which      * is mandatotry when putting objects in an S3     * bucket     *      * @param params     * @param fileName the name of the file to be uploaded     * @param contentType the content type of the file to be uploaded     * @return     */ private static String writeContent(Map params, String fileName, String contentType) {    StringBuffer buf = new StringBuffer();  Set keys = params.keySet();        for (String key : keys) {            String val = params.get(key);            buf.append("--")             .append(boundary)             .append(newLine);            buf.append("Content-Disposition: form-data; name=\"")             .append(key)             .append("\"")             .append(newLine)             .append(newLine)             .append(val)             .append(newLine);        }        buf.append("--")         .append(boundary)         .append(newLine);        buf.append("Content-Disposition: form-data; 	name=\"file\"; filename=\"")         .append(fileName)         .append("\"")         .append(newLine);        buf.append("Content-Type: ")         .append(contentType)         .append(newLine)         .append(newLine);                return buf.toString(); }}

Read theoriginal article at Wilpower.