Blog

Using the NetAdvantage jQuery Chart to Explore Over One Million Items of USGS Data

Graham Murray / Thursday, November 3, 2011

The new NetAdvantage jQuery Chart (CTP out very soon) can display and explore over 1 million data items smoothly. The amount of memory available to the JavaScript environment is more limiting than the chart’s performance. In this post I’ll describe a sample were we can show off this performance by loading gigantic amounts of CreepMeter Data from the USGS into the chart for exploration.
The data used in this sample can be obtained here. We are mostly interested in the CreepMeter logs that were taken at 10 minute intervals as this provides the massive amounts of data that we are interested in using to stress the system. Logs that started around the year 1986 provide about 1.3 million records to amply strain the system.
As you will see below, we will spend more time worrying about how to efficiently get the data to the chart than worrying about the chart’s performance itself. The chart should let you explore the data smoothly, and all in pure JavaScript running on the client.


Setting up the Project


Start by creating an ASP.NET MVC 3 Web Application from the New Project menu.

Select Internet Application, View engine: Razor and hit OK.
In my previous post on the chart, we used the chart’s MVC API to configure it, but for this post I’ll show you how you can create and update the chart using its jQuery Widget API. As a result, so we can skip adding the reference to Infragistics.Web.Mvc.
The project template for an MVC 3 Internet Application will have already created a folder named Content and a folder named Scripts in your project. Next, we need to add the appropriate NetAdvantage jQuery bits to these folders.
If you used the default install path for the product, you should find the css files to add to the Content\themes Folder at:
C:\Program Files (x86)\Infragistics\NetAdvantage 2011.2\jQuery\themes\min
And the requisite script files to add to the Scripts\IG folder at:
C:\Program Files (x86)\Infragistics\NetAdvantage 2011.2\jQuery\js\combined\min


Now we can add the requisite script references to _Layout.cshtml to be able to use the Chart from any of the views in the application:

<!-- Required for Chart -->  
    <link href="@Url.Content("~/Content/themes/base/jquery-ui.min.css")" rel="stylesheet" type="text/css" />  
    <link href=""@Url.Content("~/Content/themes/ig/jquery.ui.custom.min.css")" rel="stylesheet" type="text/css" />  
    <script src="@Url.Content("~/Scripts/jquery-ui-1.8.11.min.js")" type="text/javascript"></script>  
    <link href="@Url.Content("~/Content/themes/base/ig.ui.chart.min.css")" rel="stylesheet" type="text/css" />  
    <script src="@Url.Content("~/Scripts/IG/ig.ui.chart.min.js")" type="text/javascript"></script>

We are also going to use jQuery Templates in the tooltip of the chart, so let’s install that via NuGet.
Right click on the project node and select: Manage NuGet Packages…
Select Online and then search for jQuery Template and then click Install. After which you can add this to the script references we just added:

<script src="@Url.Content("~/Scripts/jQuery.tmpl.min.js")" type="text/javascript"></script>

 

Setting Up the Chart


Now we will set up the chart to receive the data we will download into it. After that, we can worry about getting the actual data.
Open up Index.cshtml and replace its content with:

<style>
    .shadow
    {
        -moz-box-shadow: 5px 5px rgba(0,0,0,0.5);
        -webkit-box-shadow: 5px 5px rgba(0,0,0,0.5);
        box-shadow: 5px 5px rgba(0,0,0,0.5);
    }
</style>
<script id="tooltipTemplate" type="text/x-jquery-tmpl">
        <div class="shadow ui-widget-content ui-corner-all">
        <table id="tooltip">
            <tr>
                <td>
                    <div>
                        <span id="dateValue">Date: ${ item.d }</span><br />
                        <span id="tooltipValue">Value: ${ item.v }</span>
                    </div>
                </td>
            </tr>
        </table>
        </div>
</script>

Here, we are creating a jQuery template that we will use in the tooltip of the chart. When the chart renders a tooltip it provides a “data context” that you can refer to in order to display various values in the tool tip that are relevant to the position of the cursor. In this case, we are reading two values off the data item that is in context and displaying them as text. Using jQuery templates is not the only way to define tooltip content in the chart, but it is the coolest way, so it’s what we will use in the sample today.
Next, insert this content below the above content:

<script>
    $(function () {
        var emptyData = [], interval = -1, logInfo, startTicks = -1;

        $("#tooltipTemplate").template("tooltipTemplate");

        $("#chart1").igDataChart({
            horizontalZoomable: true,
            verticalZoomable: true,
            windowResponse: "immediate",
            width: "500px",
            height: "500px",
            dataSource: emptyData,
            axes: [{
                name: "xAxis",
                type: "categoryX",
                label: "d",
                labelExtent: 50
            }, {
                name: "yAxis",
                type: "numericY",
                labelExtent: 50
            }],
            series: [{
                name: "creepMeterData",
                type: "line",
                xAxis: "xAxis",
                yAxis: "yAxis",
                title: "Creep Meter Data",
                valueMemberPath: "v",
                showTooltip: true,
                tooltipTemplate: "tooltipTemplate"
            }]
        });
    });
</script>
In this sample you can select some <a href="http://en.wikipedia.org/wiki/Creepmeter">Creepmeter</a> data <a href="http://earthquake.usgs.gov/monitoring/deformation/data/download/">provided by the USGS</a> which will download 100,000 to 1000,000+ records (depending on data set) and display them in the Chart. 
You can observe the timings for downloading the data (which is understandably large) and see how quickly, in comparison, the chart is able to render and explore the data.
<div id="chart1">
</div>

Let’s break down what’s going on here. First we have:

var emptyData = [], interval = -1, logInfo, startTicks = -1;

$("#tooltipTemplate").template("tooltipTemplate");

We are first defining some variables which we will use later, then we are indicating that we want to use the content of the script block with id “tooltipTemplate” as a jQuery template.
Next we have:

$("#chart1").igDataChart({
            horizontalZoomable: true,
            verticalZoomable: true,
            windowResponse: "immediate",
            width: "500px",
            height: "500px",
            dataSource: emptyData,
            axes: [{
                name: "xAxis",
                type: "categoryX",
                label: "d",
                labelExtent: 50
            }, {
                name: "yAxis",
                type: "numericY",
                labelExtent: 50
            }],
            series: [{
                name: "creepMeterData",
                type: "line",
                xAxis: "xAxis",
                yAxis: "yAxis",
                title: "Creep Meter Data",
                valueMemberPath: "v",
                showTooltip: true,
                tooltipTemplate: "tooltipTemplate"
            }]
        });

Here we are:

  • Finding the DOM element with id “chart1” (the div, further down)
  • Calling the Chart widget to transform the element into a chart
  • Specifying that the chart should be zoomable
  • Specifying that the chart should respond immediately to events rather than waiting for the user’s gesture to finish (mostly applies to panning).
  • Setting the width and height of the chart to 500 pixels.
  • Associating an empty data source with the chart.
  • Defining a category x axis for the chart
    • Indicating that the x axis should get its labels from a property called “d” on the items
    • Reserving 50 pixels of vertical space for the x axis’ labels.
  • Defining a numeric y axis for the chart
    • Reserving 50 pixels of horizontal space for the y axis’ labels.
  • Defining a line series for the chart.
    • Indicating that the series type is line.
    • Associating an x axis and y axis with the series.
    • Setting the title of the series (displayed in legends, etc.) to Creep Meter Data.
    • Indicating that the series should get its values from a property called “v”,
    • Indicating that the series should display a tooltip.
    • Setting the name of the jQuery template that will be used for the tooltip.

Lastly, here we have some description of the sample and the div element that will be transformed into the chart.

In this sample you can select some <a href="http://en.wikipedia.org/wiki/Creepmeter">Creepmeter</a> data <a href="http://earthquake.usgs.gov/monitoring/deformation/data/download/">provided by the USGS</a> which will download 100,000 to 1,000,000+ records (depending on data set) and display them in the Chart. 
You can observe the timings for downloading the data (which is understandably large) and see how quickly, in comparison, the chart is able to render and explore the data.
<div id="chart1">
</div>

Which shows an empty chart, and the descriptive text that we entered. Let’s put in the pieces that we will need to load some data into the chart now. Along with the loading of the data, we are going to include some logging so that we can tell the difference between the amount of time it takes to download and parse the JSON data we are going to be loading into the chart vs. the amount of time it takes the chart to load and render the data it is given. This is important because, with the quantity of data we are dealing with here, the network is going to be the limiting factor by far. You will also be able to see this by how smoothly you can interact with the chart after the data is loaded, vs. how long it takes to pull the data over the wire.
Please note, when running the project from here on in, please "Start Without Debugging" (Ctrl-F5) If you run with the debugger attached, it will hinder the performance and you won't see the Chart performing at its best. Also, if you are using Chrome v15, it appears they have currently done something to severely hinder their canvas path rendering performance. Hopefully they will resolve it in Chrome 16, but until then, you will be better off using a different browser, or running Chrome with this command line argument: --disable-accelerated-2d-canvas. The Chromium team has been made aware of the problem, and they are looking into it.
Now, let’s add the pieces we need. Insert this to the bottom of Index.cshtml:

@Html.DropDownList("AvailableFiles")
<input type="button" id="loadData" value="Load Data" />
<div class="ui-widget-content ui-corner-all" style="width: 500px; height: 200px; overflow: auto">
    <ul id="infoBox">
    </ul>
</div>

Here we are defining a DropDownList that will allow us to select which data set we want to add to the chart. We will see how we will populate this later. We are also defining a button that we will use to cause the data to be downloaded and the chart to populate, and lastly we are defining a box for our logging statements to collect in, so that we can observe the timing information of the load.
Next, modify the definition of the chart so that it looks like this:

$("#chart1").igDataChart({
            horizontalZoomable: true,
            verticalZoomable: true,
            windowResponse: "immediate",
            width: "500px",
            height: "500px",
            dataSource: emptyData,
            axes: [{
                name: "xAxis",
                type: "categoryX",
                label: "d",
                labelExtent: 50
            }, {
                name: "yAxis",
                type: "numericY",
                labelExtent: 50
            }],
            series: [{
                name: "creepMeterData",
                type: "line",
                xAxis: "xAxis",
                yAxis: "yAxis",
                title: "Creep Meter Data",
                valueMemberPath: "v",
                showTooltip: true,
                tooltipTemplate: "tooltipTemplate"
            }],
            refreshCompleted: function () {
                if (startTicks > -1) {
                    logInfo("chart load finished");
                    startTicks = -1;
                }
            }
        });

The new part we have added is the refreshCompleted handler. The chart will raise this event whenever it is done rendering. Since the chart does some work asynchronously this is the only way we can measure how long the work takes between when we assign it some data and when it has finished its work.
Next, below, we will define a method to help us log timing information to the info box we declared above:

logInfo = function (info) {
            if (startTicks > -1) {
                var date = new Date();
                var diff = date.getTime() - startTicks;
                startTicks = date.getTime();
                $("#infoBox").append($("<li>" + diff + "ms: " + info + "</li>"));
            }
        }

This records the delta timing since the previous log statement.
Below that, we will define the click handler for the button such that it downloads and loads the new data into the chart:

$("#loadData").click(function () {
            var fileName = $("#AvailableFiles").val();

            startTicks = new Date().getTime();
            logInfo("download and JSON parse started");

            $.ajax({
                url: '/Home/CreepMeterData?fileName=' + fileName,
                type: 'get',
                dataType: 'json',
                success: function (data) {
                    logInfo("download and JSON parse finished");
                    logInfo("chart load started");
                    $("#chart1").igDataChart({ dataSource: data });
                    $("#chart1").igDataChart("resetZoom");
                },
                error: function (data) {
                    console.log('error parsing json');
                }
            });
        });

In the above we:

  • Get the filename to load from the AvailableFiles DropDownList.
  • Initialize our timing for the log.
  • Log that we are starting the download.
  • Use jQuery to asynchronously download the JSON hosted at: /CreepMeterData/Home/CreepMeterData?fileName=' + filename
    • We will see how we define this action later
  • Create a success handler that will
    • Log that the download and parse of the JSON data has completed.
    • Log that the chart load is starting
    • Set the data as the dataSource property of the chart.
    • Reset the zoom level of the chart in case the user had zoomed into the previous data loaded in the chart.
    • Define an error handler that will log if we had problems with the ajax get.


Ok. Our client portion is essentially finished, all we need is the actual JSON data and the content for the AvailableFiles DropDownList. If you try to run the project now you will get errors from said list due to the fact that we have not created a ViewBag property defining its content.

Providing the Data

First lets obtain a data file to use to test the above. Open a web browser and navigate to:
http://earthquake.usgs.gov/monitoring/deformation/data/download/
Scroll down to the row that says Taylor Ranch and click the link in the 3rd column, which contains the 10 minute interval data. This should download a file called xta1.10Min.gz. Its gzipped, so use your favorite compression tool to decompress it first.
Now, create a folder in your App_Data folder named DataFiles, and paste xta1.10Min- into it. If you open xta1.10Min- in a text editor, you will see that it consists of many rows that have 3 space delimited columns. The first 2 columns seem to be indicating the year and the time offset into that year. The 3rd column represents the CreepMeter value.
We’d like to transform this data into a JSON representation that we can easily download from the client portion of the application. Because the uncompressed JSON will be large, we also want to make sure that we are using gzip compression to pass the resulting JSON to the client.
To make things more seamless, let’s use ASP.NET MVC to do the file conversion upon request and cache the converted file. We could configure IIS to serve the resulting file gzipped and the browser could figure out to decompress it, but for simplicity's sake, lets also have our MVC action store a compressed version of the JSON data and serve that up with the appropriate response headers to have the browser automatically decompress it on arrival.
First, let’s populate the AvailableFiles ViewBag property so that are DropDownList will work. Add the following to the Index method in HomeController.cs.

var availableFiles = new List<SelectListItem>();
            foreach (var file in System.IO.Directory.GetFiles(Server.MapPath("~/App_Data/DataFiles/")))
            {
                if (!file.EndsWith(".gz") && !file.EndsWith(".title"))
                {
                    var fileName = System.IO.Path.GetFileName(file);
                    var title = fileName;
                    if (System.IO.File.Exists(file + ".title"))
                    {
                        title = System.IO.File.ReadAllText(file + ".title");
                    }

                    availableFiles.Add(new SelectListItem() { Text = title, Value = fileName });
                }
            }
            ViewBag.AvailableFiles = availableFiles;

Here we are looking through all the untransformed raw data files from the USGS that we have put in the DataFiles directory, skipping any transformed gzipped files that we will be creating. If there is a file with the same name as a raw data file but with “.title” appended we will use its content as a title for the raw file.

Now, all that remains is to define the CreepMeterData action that we are calling from the client to get the JSON data. So lets insert this method into the HomeController:

public ActionResult CreepMeterData(string fileName)
        {
            var path = Server.MapPath(System.IO.Path.Combine("~/App_Data/DataFiles", fileName));

            if (!System.IO.File.Exists(path))
            {
                return HttpNotFound();
            }

            if (!System.IO.File.Exists(path + ".gz"))
            {
                CreateCompressedJsonVersion(path);
            }

            Response.AddHeader("content-encoding", "gzip");
            return File(path + ".gz", "application/json");
        }

We are checking to see if we already have a gzipped JSON version of the data, and if not, calling CreateCompressedJsonVersion for the requested file.
Then we are making sure that the Response headers contain:
content-encoding: gzip
This is to alert the browser that the response is compressed and needs to be decompressed before being passed along. Again, IIS can perform this sort of thing for you automagically if you configure it to, but this way is more concise for the purposes of this demo, and makes sure we never send the uncompressed version over the wire, which would be unpleasant.
Now all that remains is to be able to transform the raw data into JSON and gzip the result:

private void CreateCompressedJsonVersion(string path)
        {
            using (var fileStream = new System.IO.FileStream(path + ".gz", System.IO.FileMode.Create))
            {
                using (var gzipStream = new GZipStream(fileStream, CompressionMode.Compress))
                {
                    using (var memStream = new System.IO.MemoryStream())
                    {
                        using (var streamWriter = new System.IO.StreamWriter(memStream))
                        {
                            streamWriter.Write("[");

                            using (var reader = new System.IO.StreamReader(path))
                            {
                                string line = reader.ReadLine();
                                bool first = true;

                                while (line != null)
                                {
                                    if (first)
                                    {
                                        first = false;
                                    }
                                    else
                                    {
                                        streamWriter.WriteLine(",");
                                    }

                                    var vals = line.Split(' ');
                                    if (vals.Length < 3)
                                    {
                                        break;
                                    }
                                    int year = 0;
                                    double offset = 0;
                                    double value = 0;

                                    int.TryParse(vals[0], out year);
                                    double.TryParse(vals[1], out offset);
                                    double.TryParse(vals[2], out value);

                                    DateTime date = new DateTime(year, 1, 1);
                                    date = date.AddDays(offset - 1.0);

                                    streamWriter.Write("{\"d\": \"" + date.ToString("M/dd/yy") + " " + date.ToString("HH:mm") + "\", \"v\": " + value + "}");

                                    line = reader.ReadLine();
                                }
                            }

                            streamWriter.WriteLine("]");
                            streamWriter.Flush();
                            memStream.Seek(0, System.IO.SeekOrigin.Begin);
                            memStream.CopyTo(gzipStream);
                        }
                    }
                }
            }
        }

You will need to add this to the using statements:

using System.IO.Compression;

The above uses a GZipStream to create a gzip compressed version of the data in JSON format. It simply parses each row of the space delimited file and writes out a usable JSON representation. The resulting file is written to disk in the same folder as the raw data file, and will simply be read from disk next time rather than retransformed.


As you can see, it takes a short while to download and parse the over 1 million records of data, but the load into the chart takes less than a second, and exploration of the data in the chart is fast and fluid.
Adding additional files to the app only requires downloading them from the USGS, unzipping them, and placing them into the DataFiles folder. Have fun!

You can download the sample project here.

-Graham