How to manage Microsoft Azure Blob Storage with Node.js

[Infragistics] Mihail Mateev / Friday, August 1, 2014

Node.js is one one of the the most popular growing platforms for development. The language, used with Node.js is JavaScript - he #1 most-used language on GitHub, and this trend is only going to increase (Forbes, 14 of July, 2014) :

I started several articles, describing some details how to use Node.js with Microsoft SQL Server ( you can see part 1 and part 2 ) and Microsoft Azure Storage (the post about Azure Table Storage is available here) . This blog is about how you can use Node.js and Microsoft Azure Blob Storage .

You will learn how to start with Node and Blob Storage, covering the most often seen cases.

Microsoft Azure Blob Storage

Microsoft Azure storage services allow us to store/retrieve the NON RELATIONAL data to/from Windows Cloud environment. In Microsoft Azure Storage, the data can be stored in 4 different formats (v.i.z. Blobs, Tables and Queues, File Storage (in preview )). More details about what is Microsoft Azure Storage in general you can read in my previous post.

What is a Blob?

Azure Blob storage is a service for storing large amounts of unstructured data that can be accessed from anywhere in the world via HTTP or HTTPS. A single blob can be hundreds of gigabytes in size.

The Blob service contains the following components:

Storage Account: All access to Azure Storage is done through a storage account.
Container: A container provides a grouping of a set of blobs. All blobs must be in a container. An account can contain an unlimited number of containers. A container can store an unlimited number of blobs.
Blob: A file of any type and size. There are two types of blobs that can be stored in Azure Storage: block and page blobs. Most files are block blobs. A single block blob can be up to 200 GB in size. This tutorial uses block blobs. Page blobs, another blob type, can be up to 1 TB in size, and are more efficient when ranges of bytes in a file are modified frequently.

Blob stands for 'binary large object', which is an array of raw bytes. The following figure shows the hierarchical structure used for the Blob storage in Windows Azure:

Azure Storage Account can have multiple Containers. A Container can be considered as an array or collection of one or more Blobs. Also, each Blob can have one or more metadata properties to define what the Blob content is all about. The metadata properties are Name-Value collection of strings.

As mentioned before, the contents of every Blob on Microsoft Azure can be accessed by browsing its corresponding URI (REST). The URI is usually of the following format:

https://<Account>.blob.core.windows.net/<Container>/<BlobName>

How to use Blob Storage and Node.JS?

You can use different NodeJS packages to handle Azure Blob Storage. In this post we will cover azure Node package ( an official Microsoft Azure SDK ).

Microsoft Azure SDK for Node.js

The official Microsoft Azure SDK (npm page: https://www.npmjs.org/package/azure , homepage: http://github.com/WindowsAzure/azure-sdk-for-node ) for Node.js. provides a Node.js package that makes it easy to consume and manage Microsoft Azure Services.

The listed snippets below demonstrate how to manage Azure Blob service using Microsoft Azure SDK for Node.js.

Create an Azure Blob Service

var azure = require('azure');

  

var accessKey = '[accountKey]';

var storageAccount = '[accountName]';

var containerName = 'nodejs';

  

//create a blob service set explicit credentials

var blobService = azure.createBlobService(storageAccount, accessKey);

You can create Azure Blob Service using the environment variables to store the credentials in the same way like for Azure Table service ( look at this post ).

List blobs in Azure Blob Storage

JavaScript code:

var azure = require('azure');

  

exports.blobs = function (request, response) {

  

    var accessKey = '[accountKey]';

    var storageAccount = '[accountName]';

    var container = 'nodejs';

  

    var blobService = azure.createBlobService(storageAccount, accessKey);

    //render blobs with blobs.jade view

    blobService.listBlobs(container, function (error, blobs) {

        response.render('blobs', {

            error: error,

            container: container,

            blobs: blobs

        });

    });

}

A jade view ( using Express.js )

extends layout

 

block content

    h1 Blob listing for #{container}

 

    if error

        h3= error

 

    if blobs

        h3= container

        table

            tr

                th Name

                th Properties

            - each blob in blobs

                tr

                    td= blob.name

                    td

                        ul

                            - each value, name in blob.properties

                                if value

                                    li= name + ":" + value

Upload file to Azure Blob Storage

One approach is to use multiparty node module. It is multipart/form-data parser which supports streaming. Using Multiparty(npm install multiparty), a fork of Formidable, we can access the multipart data. This module will not stream the file to disk unless you tell it to.

//upload a file to azure blob storage

app.get('/upload', function (req, res) {

    res.send(

    '<form action="/upload" method="post" enctype="multipart/form-data">' +

    '<input type="file" name="snapshot" />' +

    '<input type="submit" value="Upload" />' +

    '</form>'

);

});

  

app.post('/upload', function (req, res) {

    var multiparty = require('multiparty');

    var accessKey = '[accountKey]';

    var storageAccount = '[accountName]';

  

    var container = 'nodejs';    

    var blobService = azure.createBlobService(storageAccount, accessKey);

    var form = new multiparty.Form();

    

    form.on('part', function (part) {

        if (part.filename) {

            

            var size = part.byteCount - part.byteOffset;

            var name = part.filename;

            

            blobService.createBlockBlobFromStream(container, name, part, size, function (error) {

                if (error) {

                    res.send(' Blob create: error ');

                }

            });

        } else {

            form.handlePart(part);

        }

    });

    form.parse(req);

    res.send('OK');

});

//end of upload a file to azure blob storage

Screens below demonstrate how to upload file to Azure Blob Storage using the sample demo application.

Download files from Azure Blob Storage

To download the blob and write it to the file system, a similar getBlob or getBlobToFile methods can be used.

The snipped below demonstrates how to use getBlob:

var azure = require('azure');

  

var blobService = azure.createBlobService();

  

//using getBlob to save a file from Azure Blob

blobService.getBlob('[containerName]', '[blobName]').pipe(fs.createWriteStream('[myFileName]'));

Code below is part of the demo application, demonstrating how to download file from Azure Blob Storage using getBlobToFile :

  

//download azure blob storage cotent

app.get('/downloadBlob', function (req, res) {

    res.send(

    '<form action="/downloadBlob" method="post" >' +

    '<input type="text" name="blobFile" value="C:\\temp" />' +

    '<input type="submit" value="Download" />' +

    '</form>'

);

});

  

app.post('/downloadBlob', function (req, res) {

    var fs = require('fs');

    

    if (!fs.existsSync) {

        fs.existsSync = require('path').existsSync;

    }

    var destinationDirectoryPath = req.body.blobFile;

    var accessKey = '[accountKey]';

    var storageAccount = '[accountName]';

    var containerName = 'nodejs';

  

    var blobService = azure.createBlobService(storageAccount, accessKey); //ok

    

    if (!fs.existsSync(destinationDirectoryPath)) {

        console.log(destinationDirectoryPath + ' is an invalid directory path.');

    } else {

        downloadFilesParallel(res, blobService, containerName, destinationDirectoryPath);

    }

  

});

downloadFilesParallel function, used in the code above:

function downloadFilesParallel(res, blobService, containerName, destinationDirectoryPath) {

    blobService.listBlobs(containerName, function (error, blobs) {

        if (error) {

            console.log(error);

        } else {

            var blobsDownloaded = 0;

            res.writeHead(200, { 'Content-Type': 'text/plain' })

            blobs.forEach(function (blob) {

                blobService.getBlobToFile(containerName, blob.name, destinationDirectoryPath + '/' + blob.name, function (error2) {

                    blobsDownloaded++;

                    

                    if (error2) {

                        console.log(error2);

                    } else {

                        res.write('\nBlob ' + blob.name + ' download finished.');

                        

                        if (blobsDownloaded === blobs.length) {

                            // Wait until all workers complete and the blobs are downloaded

                            res.end('\nAll files downloaded');

                        }

                    }

                });

            });

        }

    });

}

Screenshots from the sample app, demonstrating how to download a file from Blob Storage.

Specialists can learn much more details about Azure Blob services and Node. Microsoft continues to grow and improve blob storage. This post describes how to start, covering the base cases and most popular Node.js modules for Azure Blob Storage. It will be useful for both – JavaScript developers who don’t have experience with Microsoft Azure and Azure developers who have less experience with JavaScript and/or Node.js.

Developers on the Microsoft Azure Platform should become familiar with how Azure Blob Storage works and how to handle unstructured data in the Microsoft cloud. Knowing how blob storage works will help you determine if it is a good fit for your particular requirements.

You can download source code from Git repository .

If you want more information about how to use Microsoft Azure Storage & Node.js feel free to contact me at mmateev@infragistics.com

You can learn more about Node.js , Microsoft Azure and related events like Azure Bootcamp Bulgaria if you follow us on Twitter @mihailmateev and @Infragistics and stay in touch on Facebook, Google+, LinkedIn and Infragistics Friends User Group !