Documentation

Incremental on-demand data loading

This tutorial will explore how we can implement fully automated on-demand incremental loading of hourly/daily/monthly data from a backend, on a Stock-like chart shown above.

Working demo

If you'd like to jump straight in, here's a fully working chart. Further sections in this tutorial will explain main pieces of functionality used in this demo.

See the Pen amCharts 5: Incremental on-demand data loading Stock chart by amCharts team (@amcharts) on CodePen.

Chart setup

We start our chart with data for the last 50 days.

The goal is to enable user to drag plot area to right, revealing earlier dates, which would subsequently be populated with missing data for those days.

User could do this repeatedly, gradually loading more ant more data.

To do that we need to:

  • Enable panning of plot area.
  • Enable over-zoom (ability to drag the chart outside of the actual data scope).
  • Set up event handlers to trigger data loading for newly revealed range of date/time.

Enabling panning

Normally, chart would perform zoom if pushed and dragged cursor or touch across it.

To enable panning instead, we need to use chart's panX setting. Since we only want horizontal panning, we'll just disable panY.

We also enable panning using mouse wheel using wheelX setting:

let chart = root.container.children.push(am5xy.XYChart.new(root, {
  panX: true,
  panY: false,
  wheelX: "panX",
  wheelY: "zoomX",
  layout: root.verticalLayout
}));
var chart = root.container.children.push(am5xy.XYChart.new(root, {
  panX: true,
  panY: false,
  wheelX: "panX",
  wheelY: "zoomX",
  layout: root.verticalLayout
}));

MORE INFOFor more information on how pan/zoom/wheel works on an XYChart, visit "Zoom and pan" tutorial.

Enabling over-zoom

Over-zoom means that chart will allow panning it outside of the actual range of data.

For over-zoom is responsible our X axis (date axis), and more specifically it's maxDeviation setting.

It accepts relative value of how much of axis width we want to allow to allow overzoom.

For our purpose we'll set it to 1 (one) which means that we allow user to completely drag the visual axis scope outside of the plot area.

let dateAxis = chart.xAxes.push(am5xy.DateAxis.new(root, {
  maxDeviation: 1,
  baseInterval: { timeUnit: "day", count: 1 },
  renderer: am5xy.AxisRendererX.new(root, {}),
  tooltip: am5.Tooltip.new(root, {})
}));
var dateAxis = chart.xAxes.push(am5xy.DateAxis.new(root, {
  maxDeviation: 1,
  baseInterval: { timeUnit: "day", count: 1 },
  renderer: am5xy.AxisRendererX.new(root, {}),
  tooltip: am5.Tooltip.new(root, {})
}));

MORE INFOFor more information, visit "Zoom and pan: Over-zooming".

Setting up events

We'll get to this later in the tutorial in "On-demand loading" section.

Backend

For the purpose of this demo we are using a sample backend service which takes parameters for start, end timestamps as well as required data granularity: hourly, daily, or monthly.

Our sample backend returns randomly generated data.

In real applications you may need to use some existing or custom backend, but it needs to be capable of returning just part of the data - so that we can implement fast incremental loading - no need to load data we already have.

Loading data

We are using amCharts 5 built-in data loader and parser to load CSV data from our backend.

The following custom function is responsible for the actual request, load, parsing of data:

function loadData(unit, min, max, side) {

  // round min so that selected unit would be included
  min = am5.time.round(new Date(min), unit, 1).getTime();

  // Load external data
  let url = "https://www.amcharts.com/tools/data/?unit=" + unit + "&start=" + min + "&end=" + max;

  // Handle loaded data
  am5.net.load(url).then(function(result) {

    // Parse loaded data
    let data = am5.CSVParser.parse(result.response, {
      delimiter: ",",
      reverse: false,
      skipEmpty: true,
      useColumnNames: true
    });

    // Process data (convert dates and values)
    let processor = am5.DataProcessor.new(root, {
      numericFields: ["date", "open", "high", "low", "close", "volume"]
    });
    processor.processMany(data);

    // Update chart elements with newly loaded data
    // ...
  });

}
function loadData(unit, min, max, side) {

  // round min so that selected unit would be included
  min = am5.time.round(new Date(min), unit, 1).getTime();

  // Load external data
  var url = "https://www.amcharts.com/tools/data/?unit=" + unit + "&start=" + min + "&end=" + max;

  // Handle loaded data
  am5.net.load(url).then(function(result) {

    // Parse loaded data
    var data = am5.CSVParser.parse(result.response, {
      delimiter: ",",
      reverse: false,
      skipEmpty: true,
      useColumnNames: true
    });

    // Process data (convert dates and values)
    var processor = am5.DataProcessor.new(root, {
      numericFields: ["date", "open", "high", "low", "close", "volume"]
    });
    processor.processMany(data);

    // Update chart elements with newly loaded data
    // ...
  });

}

The function is pretty-self explanatory: it takes unit ("hour", "day", or "month"), as well as start and end timestamps of the data we want to load data for.

The fourth parameter will be used when updating series data: we need to know where to append data to: to the beginning, end, or completely replace the data with a new one. We'll get to that later.

MORE INFOFor more information about loading and parsing external data, visit "Data: External data".

On-demand loading

In order to handle on-demand loading we need to know when and how much user has panned the chart.

For that we will use chart's panended and wheelended events. Those kick in when chart is panned using mouse/touch and mouse wheel respectively.

While panended event will kick only when user releases the chart after dragging, wheelended will be triggered multiple times, so we will need to "debounce" it (trigger actual load only when turning has stopped, i.e. significant time has passed between turns).

// load data when panning ends
chart.events.on("panended", function() {
  loadSomeData();
});


let wheelTimeout;
chart.events.on("wheelended", function() {
  // load data with some delay when wheel ends, as this is event is fired a lot
  // if we already set timeout for loading, dispose it
  if(wheelTimeout){
    wheelTimeout.dispose();
  }

  wheelTimeout = chart.setTimeout(function() {
    loadSomeData();  
  }, 50)  
});
// load data when panning ends
chart.events.on("panended", function() {
  loadSomeData();
});


var wheelTimeout;
chart.events.on("wheelended", function() {
  // load data with some delay when wheel ends, as this is event is fired a lot
  // if we already set timeout for loading, dispose it
  if(wheelTimeout){
    wheelTimeout.dispose();
  }

  wheelTimeout = chart.setTimeout(function() {
    loadSomeData();  
  }, 50)  
});

Notice, how we don't care about actual range of pan/selection, yet. When panning ends we just invoke a custom function loadSomeData() which will take actual range of the panned date axis, and use that information to load missing data.

function loadSomeData() {
  let start = dateAxis.get("start");
  let end = dateAxis.get("end");

  let selectionMin = Math.max(dateAxis.getPrivate("selectionMin"), absoluteMin);
  let selectionMax = Math.min(dateAxis.getPrivate("selectionMax"), absoluteMax);

  let min = dateAxis.getPrivate("min");
  let max = dateAxis.getPrivate("max");

  // if start is less than 0, means we are panning to the right, need to load data to the left (earlier days)
  if (start < 0) {
    loadData(currentUnit, selectionMin, min, "left");
  }
  // if end is bigger than 1, means we are panning to the left, need to load data to the right (later days)
  if (end > 1) {
    loadData(currentUnit, max, selectionMax, "right");
  }
}
function loadSomeData() {
  var start = dateAxis.get("start");
  var end = dateAxis.get("end");

  var selectionMin = Math.max(dateAxis.getPrivate("selectionMin"), absoluteMin);
  var selectionMax = Math.min(dateAxis.getPrivate("selectionMax"), absoluteMax);

  var min = dateAxis.getPrivate("min");
  var max = dateAxis.getPrivate("max");

  // if start is less than 0, means we are panning to the right, need to load data to the left (earlier days)
  if (start < 0) {
    loadData(currentUnit, selectionMin, min, "left");
  }
  // if end is bigger than 1, means we are panning to the left, need to load data to the right (later days)
  if (end > 1) {
    loadData(currentUnit, max, selectionMax, "right");
  }
}

The loadSomeData() function above glues everything together.

It uses a number date axis' settings to track its current zoom:

  • start and end indicates relative zoom position. We are using it to determine which direction user has panned the chart to: if start is lower then zero - it means that user has overzoomed to the left; if end is higher than 1 (one) it means user overzoomed to the right. Fully zoomed out axis has start and end set at 0 and 1 respectively.
  • Private settings selectionMin and selectionMax will hold actual timestamps for start and end position of the axis, in its current panned state. We use those to specify the range of data to load.
  • Private settings min and max hold timestamps for earliest and latest data items we have data for. We use them to make our data loader load only data we don't have.

Finally, the function invokes loadData() function we have looked at already, which then loads, parses, and updates the chart accordingly.

Updating data

Let's go back to loadData() function.

We already looked how it loads raw data using am5.net.load() utility, then parses it using data parser.

All we have left to do is to update chart elements with the newly-loaded data.

We will need to update data for:

  • Both series of the chart.
  • Series used in the scrollbar.
  • Update date axis' min and max settings to our new data range.

Updating data for series is super easy: we just need to either push() data object into its data (if we want to add data points to the end), or unshift() if we're adding data to the beginning. E.g.:

for (let i = data.length - 1; i >= 0; i--) {
  let date = data[i].date;
  // only add if first items date is bigger then newly added items date
  if (seriesFirst[valueSeries.uid] > date) {
    valueSeries.data.unshift(data[i]);
  }
  if (seriesFirst[volumeSeries.uid] > date) {
    volumeSeries.data.unshift(data[i]);
  }
  if (seriesFirst[sbSeries.uid] > date) {
    sbSeries.data.unshift(data[i]);
  }
}
for (var i = data.length - 1; i >= 0; i--) {
  var date = data[i].date;
  // only add if first items date is bigger then newly added items date
  if (seriesFirst[valueSeries.uid] > date) {
    valueSeries.data.unshift(data[i]);
  }
  if (seriesFirst[volumeSeries.uid] > date) {
    volumeSeries.data.unshift(data[i]);
  }
  if (seriesFirst[sbSeries.uid] > date) {
    sbSeries.data.unshift(data[i]);
  }
}

Normally, when we load new data, the chart would animate to new range. Since we don't want any additional bouncing around, we also fix series min and/or max settings to the current data range.

We also want to keep current zoom level, which means that we need to to update start and end settings, too.

// update axis min
min = Math.max(min, absoluteMin);
dateAxis.set("min", min);
dateAxis.setPrivate("min", min); // needed in order not to animate

// recalculate start and end so that the selection would remain
dateAxis.set("start", 0);
dateAxis.set("end", (end - start) / (1 - start));
// update axis min
min = Math.max(min, absoluteMin);
dateAxis.set("min", min);
dateAxis.setPrivate("min", min); // needed in order not to animate

// recalculate start and end so that the selection would remain
dateAxis.set("start", 0);
dateAxis.set("end", (end - start) / (1 - start));

The above is code snippet is relative for the situation where we are adding data to the left of the data set.

Other situations will need different setup of settings. Please refer to the full code in the CodePen example at the beginning of this tutorial.

Data granularity

The other concept of the demo is that it allows switching between granularity of the data easily.

Basically, when specific granularity button is pressed, the code will load data for the current visible range anew.

It will use the loadData() function we're already familiar with, except that it will instruct it to replace the whole data set, rather than append data.

loadData("month", dateAxis.getPrivate("selectionMin"), dateAxis.getPrivate("selectionMax"), "none");
loadData("month", dateAxis.getPrivate("selectionMin"), dateAxis.getPrivate("selectionMax"), "none");
Posted in Uncategorized