Loading External Data

amCharts 4 has built-in external data loader and parser, which you can use to automatically load chart data in JSON or CSV formats.

This article will show how to configure and use this built-in data loader.

Prerequisites

If you haven't done so already, before we begin, we suggest you read our "Data" article, which explains the concepts of data in amCharts 4.

Automatic loading

Each chart type has two data-related properties: data (holds the data for the chart) and dataSource (holds instructions for external data).

The former (data) can be used to set data directly to the chart. If you have the data already you can set, you don't need any external data.

The subject of this article - second property (dataSource) - is an object of type DataSource which describes instructions on how to load external data.

Enabling external data

Depending on data format of your external source, you may need to one or more configuration options set on dataSource.

To enable external data loading, at the very least you need to set dataSource.url property, which will contain an URL of that source.

The URL can either be absolute, or relative to the current page.

chart.dataSource.url = "/data/myData.php";
chart.dataSource.url = "/data/myData.php";
{
  // ...
  "dataSource": {
    "url": "/data/myData.php"
  }
}

This instructs the chart to hold off from rendering, trigger loading and subsequent parsing of external data. While all of this is happening, the chart will show a loading indicator.

Since we haven't specified any format, yet, the loader will assume our data is in JSON format, and will try to parse loaded information as such.

Once the data is loaded and parsed, the loader itself will set the data property at which point the actual chart will be rendered.

Supported formats

Built-in data loader currently supports two formats: JSON (default) and CSV.

If not set otherwise, loader will assume JSON.

To set which format is used we use dataSource.parser property.

JSON parser is an object of JSONParser class. CSV parser is predictably CSVParser.

Here's how we would modify our example to explicitly instruct to use JSON parser:

chart.dataSource.url = "/data/myData.php";
chart.dataSource.parser = new am4core.JSONParser();
chart.dataSource.url = "/data/myData.php";
chart.dataSource.parser = new am4core.JSONParser();
{
  // ...
  "dataSource": {
    "url": "/data/myData.php",
    "parser": {
      "type": "JSONParser"
    }
  }
}

Of course, specifying JSON as parser above is just to illustrate the point. JSON is default, so we would have achieved the same effect without specifying any parser at all.

Each parser has its own set of configuration options, relevant to this specific format. You set them directly on parser's options properties.

chart.dataSource.url = "/data/myData.php";
chart.dataSource.parser = new am4core.JSONParser();
chart.dataSource.parser.options.emptyAs = 0;
chart.dataSource.url = "/data/myData.php";
chart.dataSource.parser = new am4core.JSONParser();
chart.dataSource.parser.options.emptyAs = 0;
{
  // ...
  "dataSource": {
    "url": "/data/myData.php",
    "parser": {
      "type": "JSONParser",
      "options": {
        "emptyAs": 0
      }
    }
  }
}

Let's explore each available options for each individual format.

JSON

Setting Type Default Comment
dateFields string[] [] (empty array) Specifies a list of fields that hold date information.

 

If set, the parser will use chart's dateFormatter to parse strings in those fields to proper Date objects.

NOTE If you have any series that bind their "date" data fields to data, this setting will automatically contain these field names.

dateFormat string - If set, parser will assume this format when parsing dates from dateFields.

 

MORE INFO Date formatting docs.

dateFormatter DateFormatter - In case you don't specify dateFormat and do not want parser to user chart's DateFormatter, you can set a new instance of DateFormatter here.

 

Normally, you would not set property.

delimiter string "," (comma) A symbol (or symbols) separating columns in each row.
emptyAs any - If set, any "empty" values will be replaced by this value.

 

I.e. you can set this to 0 (zero) to treat all null or omitted values as numeric zeros, rather than gap in data.

numberFields string[] [] (empty array) A list of fields that hold numeric fields.

 

Parser will try to convert any values, contained in such fields to number.

NOTE If you have any series that bind their "value" data fields to data, this setting will automatically contain these field names.

JSON example

Let's create a sample Line chart with external data in JSON format.

We'll use this file as our sample data.

It looks like this:

[{
  "year": "1994",
  "cars": 1587,
  "motorcycles": 650,
  "bicycles": 121
}, {
  "year": "1995",
  "cars": 1567,
  "motorcycles": 683,
  "bicycles": 146
}, {
  // ...
}]

As you can see, each data point in it contains a category ("year"), and three value fields.

We'll create a chart which has a Category axis bound to "year" as well as three Line series bound to each of the value fiels. ("cars", "motorcycles", and "bicycles")

Here's how it looks like:

See the Pen amCharts V4: Data loading (JSON) by amCharts (@amcharts) on CodePen.24419

CSV

CSV while offering the least possible overhead in formatting, requires more options for the parser to handle it.

Setting Type Default Comment
dateFields string[] [] (empty array) Specifies a list of fields that hold date information.

 

If set, the parser will use chart's dateFormatter to parse strings in those fields to proper Date objects.

NOTE If you have any series that bind their "date" data fields to data, this setting will automatically contain these field names.

dateFormat string - If set, parser will assume this format when parsing dates from dateFields.

 

MORE INFO Date formatting docs.

dateFormatter DateFormatter - In case you don't specify dateFormat and do not want parser to user chart's DateFormatter, you can set a new instance of DateFormatter here.

 

Normally, you would not set property.

delimiter string "," (comma) A symbol (or symbols) separating columns in each row.
emptyAs any - If set, any "empty" values will be replaced by this value.

 

I.e. you can set this to 0 (zero) to treat all null or omitted values as numeric zeros, rather than gap in data.

numberFields string[] [] (empty array) A list of fields that hold numeric fields.

 

Parser will try to convert any values, contained in such fields to number.

NOTE If you have any series that bind their "value" data fields to data, this setting will automatically contain these field names.

reverse boolean false Set to true if your data is in reverse (descending) order. (older dates are lower in the list)
skipEmpty boolean true Parser will skip empty rows by default. Set this to false to instead create empty data points.
skipRows number 0 Set how many rows to skip from the beginning of the data.

 

If you use useColumnNames you don't need to set this to 1. The parser will automatically assume that first row is column names, not data.

useColumnNames boolean false Normally, the parser will assume the following column names: "col0", "col1", etc.

 

If you set useColumnNames = true parser will use the first row in data, to figure out column names. E.g.:

Date,Value
2018-01-01,100
2018-01-02,110
2018-01-03,95

Here's how it will look like when parsed without useColumnNames:

[{
  "col0": "2018-01-01",
  "col1": 100
}, {
  "col0": "2018-01-02",
  "col1": 110
}, {
  "col0": "2018-01-03",
  "col1": 95
}]

And here's how it looks like with useColumnsNames = true:

[{
  "Date": "2018-01-01",
  "Value": 100
}, {
  "Date": "2018-01-02",
  "Value": 110
}, {
  "Date": "2018-01-03",
  "Value": 95
}]

Using CSV fields names

Important thing to consider is how we are going to be referencing fields in data loaded from CSV sources.

If we don't set useColumnNames = true (our data source might not necessarily contain column names), we'll end up with field names like "col0", "col1", etc. in our final data.

We'll need to use them as such everywhere fields are referenced, e.g. series' dataFields, CSV parser's own settings, like dateFields and numberFields, and just about anywhere else where data binding is going on.

CSV example

Let's rework the same chart from our JSON example.

We do have a version of the same data in CSV format, which looks like this:

year,cars,motorcycles,bicycles
1994,1587,650,121
1995,1567,683,146
1996,1617,691,138
...

As we have already learned, CSV requires a bit more of an effort in configuration.

At the very least we'll need to set up our data loader to use CSVParser as well as make that parser use first row as column names in resulting data.

chart.dataSource.url = "https://s3-us-west-2.amazonaws.com/s.cdpn.io/t-160/sample_data_serial.csv";
chart.dataSource.parser = new am4core.CSVParser();
chart.dataSource.parser.options.useColumnNames = true;
chart.dataSource.url = "https://s3-us-west-2.amazonaws.com/s.cdpn.io/t-160/sample_data_serial.csv";
chart.dataSource.parser = new am4core.CSVParser();
chart.dataSource.parser.options.useColumnNames = true;
{
  // ...
  "dataSource": {
    "url": "/data/myData.php",
    "parser": {
      "type": "CSVParser",
      "options": {
        "useColumnNames": true
      }
    }
  }
}

Let's see how it turned out:

See the Pen amCharts V4: Data loading (CSV) by amCharts (@amcharts) on CodePen.24419

Advanced usage

Repetitive loading

In case your data changes dynamically, you might want to set up the data loader to reload the data at preset intervals, so that your chart is always up-to-date.

To do that, use dataSource.reloadFrequency setting.

It's a number of milliseconds you want your chart data to be reloaded at.

For example if I want my chart to reload its data every 5 seconds, i'll set this to 5000:

chart.dataSource.reloadFrequency = 5000;
chart.dataSource.reloadFrequency = 5000;
{
  // ...
  "dataSource": {
    // ...
    "reloadFrequency": 5000
  }
}

Reloading (different) data

I you need to change the URL of the data after the data is already loaded, you can simply set data source's url property. To trigger a reload, call its load() method:

chart.dataSource.url = "newData.json";
chart.dataSource.load();
chart.dataSource.url = "newData.json";
chart.dataSource.load();

Data source will load new data set and update the chart with it. No further action is required.

Incremental data loading

Sometimes, especially if you have large data sets, it's not worth to reload whole data on each update. If you add a new data point every 5 seconds, loading a thousand of previously-loaded data points again would be a complete waste of bandwidth.

In this case you can specify data loader to use "incremental loading" by setting its setting incremental to true. (default is false)

This will instruct the loader to treat each new load of data as addition to the old data. New data points will be appended to the end of the data rather completely replace it.

Modifying URL for each incremental load

Incremental loading, if done correctly, might save a lot, both in load speed and in subsequent data parsing.

To make it useful, you might also need to pass in a time of your last load to your data source via URL. We can use an adapter for that. (If you're not sure what adapters are and how to use them, make sure to read this article)

Data source stores the date/time of its last load in dataSource.lastLoad. We'll use it to in an adapter for url to slap a timestamp of the last load:

chart.dataSource.reloadFrequency = 5000;
chart.dataSource.incremental = true;
chart.dataSource.adapter.add("url", function(url, target) {
  // "target" contains reference to the dataSource itself
  if (target.lastLoad) {
    url .= "?lastload=" + target.lastLoad.getTime();
  }
  return url;
});
chart.dataSource.reloadFrequency = 5000;
chart.dataSource.incremental = true;
chart.dataSource.adapter.add("url", function(url, target) {
  // "target" contains reference to the dataSource itself
  if (target.lastLoad) {
    url .= "?lastload=" + target.lastLoad.getTime();
  }
  return url;
});
{
  // ...
  "dataSource": {
    // ...
    "reloadFrequency": 5000,
    "incremental": true,
    "adapter": {
      "url": function(url, target) {
        // "target" contains reference to the dataSource itself
        if (target.lastLoad) {
          url .= "?lastload=" + target.lastLoad.getTime();
        }
        return url;
      }
    }
  }
}

Another way to attach parameters to incremental loads is to use incrementalParams property. It's a powerful property allowing completely customize the URL of the request for each incremental load.

It's an object whose key and value pairs will be used as query string parameters. E.g.:

chart.dataSource.url = "/data/myData.php";
chart.dataSource.incremental = true;
chart.dataSource.reloadFrequency = 5000;
chart.dataSource.incrementalParams = {
  incremental: "y",
  something: "else"
}
chart.dataSource.url = "/data/myData.php";
chart.dataSource.incremental = true;
chart.dataSource.reloadFrequency = 5000;
chart.dataSource.incrementalParams = {
  incremental: "y",
  something: "else"
}
{
  // ...
  "dataSource": {
    "url": "/data/myData.php",
    "incremental": true,
    "reloadFrequency": 5000,
    "incrementalParams": {
      "incremental": "y",
      "something": "else"
    }
  }
}

The above will call /data/myData.php on the first load, and /data/myData.php?incremental=y&something=else on any subsequent loads.

We can also use adapters on it as well:

chart.dataSource.url = "/data/myData.php";
chart.dataSource.incremental = true;
chart.dataSource.reloadFrequency = 5000;
chart.dataSource.adapter.add("incrementalParams", function(params) {
  params.lastload = target.lastLoad.getTime();
  params.incremental = "y";
  return params;
});
chart.dataSource.url = "/data/myData.php";
chart.dataSource.incremental = true;
chart.dataSource.reloadFrequency = 5000;
chart.dataSource.adapter.add("incrementalParams", function(params) {
  params.lastload = target.lastLoad.getTime();
  params.incremental = "y";
  return params;
});
{
  // ...
  "dataSource": {
    "url": "/data/myData.php",
    "incremental": true,
    "reloadFrequency": 5000,
    "adapter": {
      "incrementalParams": function(params) {
        params.lastload = target.lastLoad.getTime();
        params.incremental = "y";
        return params;
      }
    }
  }
}

Now, with each subsequent incremental request, the URL will contain one static parameter ("incremental") and one dynamically filled one ("lastload").

Removing old data items

If you are implementing a "moving chart" - one that is constantly updated with new items, while old ones are being discarded - you might also want to set keepCount = true.

If set, it will ensure that number of data items is kept at constant number. When X new data points are loaded and added to the data set, the same X number of oldest data items will be removed.

Updating existing data

Normally, data loader would automatically replace all of the chart's data or add loaded data points to the end.

However, if your data changes only in values across the loads, much more efficient (and neat) way is to instruct the loader to only update those values.

Version 4.5.5 introduce's a new setting to DataSource: updateCurrentData.

If set to true it will not replace whole of the chart's data but rather individual values, as well as call invalidateRawData() on it.

It will not only result in faster load, but will also have a neat side-effect of series morphing from old values to new ones.

chart.dataSource.url = "/data/myData.php";
chart.dataSource.updateCurrentData = true;
chart.dataSource.reloadFrequency = 5000;
chart.dataSource.url = "/data/myData.php";
chart.dataSource.updateCurrentData = true;
chart.dataSource.reloadFrequency = 5000;
{
  // ...
  "dataSource": {
    "url": "/data/myData.php",
    "updateCurrentData": true,
    "reloadFrequency": 5000
  }
}

Handling events

PREREQUISITE If you're not familiar with event handling in amCharts 4, please read "Event Listeners" article first.

Data loader supports a lot of events, which you can use to add custom functionality, or even manipulate loaded data.

Manipulating loaded data

For example, to manipulate loaded data, we can use parseended event, which kicks in right after the data was parsed but not yet handed over to the chart.

chart.dataSource.events.on("parseended", function(ev) {
  // parsed data is assigned to data source's `data` property
  var data = ev.target.data;
  for (var i = 0; i < data.length; i++) {
    data[i]["year"] = "Year: " + data[i]["year"];
  }
});
chart.dataSource.events.on("parseended", function(ev) {
  // parsed data is assigned to data source's `data` property
  var data = ev.target.data;
  for (var i = 0; i < data.length; i++) {
    data[i]["year"] = "Year: " + data[i]["year"];
  }
});
{
  // ...
  "dataSource": {
    // ...
    "events": {
      "parseended": function(ev) {
        // parsed data is assigned to data source's `data` property
        var data = ev.target.data;
        for (var i = 0; i < data.length; i++) {
          data[i]["year"] = "Year: " + data[i]["year"];
        }
      }
    }
  }
}

The above code is supposed to prefix each category (year) with a "Year: " after the data is loaded. Let's see if it worked:

See the Pen amCharts V4: Data loading (JSON 2) by amCharts (@amcharts) on CodePen.24419

Handling errors

Another common usage for data loading events, is catching and handling errors.

Perhaps you'd like to show an error message to your user, or do something else.

chart.dataSource.events.on("error", function(ev) {
  console.log("Oopsy! Something went wrong");
});
chart.dataSource.events.on("error", function(ev) {
  console.log("Oopsy! Something went wrong");
});
{
  // ...
  "dataSource": {
    // ...
    "events": {
      "error": function(ev) {
        console.log("Oopsy! Something went wrong");
      }
    }
  }
}

Loading chunks of chart config

You can use data loading to even load chunks of chart config in JSON format.

The following example demonstrates how a Gauge chart can use its dataSource to dynamically load its hands:

chart.dataSource.events.on("done", function(ev) {
  ev.target.component.config = {
    hands: ev.data
  }
});
chart.dataSource.events.on("done", function(ev) {
  ev.target.component.config = {
    hands: ev.data
  }
});
{
  // ...
  "dataSource": {
    // ...
    "events": {
      "done": function(ev) {
        ev.target.component.config = {
          hands: ev.data
        }
      }
    }
  }
}

NOTE Notice how we are assigning the Gauge's hands data. This is not just raw data. It's a chart config. So we need to use charts config accessor to invoke JSON config parser. Since target in event handler parameter represents the DataSource, we need to use its property component which holds actual reference to the chart.

Custom request headers

It's possible to add custom request headers to your Data Source requests, using requestOptions.requestHeaders option.

The requestHeaders is an array of objects that have key and value pairs, e.g.:

chart.dataSource.requestOptions.requestHeaders = [
  { key: "Accept", value: "*/*" },
  { key: "Content-Type", value: "text/csv" },
  { key: "Authorization", value: "123456789" }
];
chart.dataSource.requestOptions.requestHeaders = [
  { key: "Accept", value: "*/*" },
  { key: "Content-Type", value: "text/csv" },
  { key: "Authorization", value: "123456789" }
];
{
  // ...
  "dataSource": {
    // ...
    "requestOptions": {
      "requestHeaders": [
        { key: "Accept", value: "*/*" },
        { key: "Content-Type", value: "text/csv" },
        { key: "Authorization", value: "123456789" }
      ]
    }
  }
}

Animating series

Charts with in-line data, apply certain animations to its series on load.

This does not happen when used with external data, because by the time the data is loaded, the chart is already initialized.

To work around that, we can use datavalidated event to trigger initial animations of the those series:

chart.events.on("datavalidated", function(ev) {
  chart.series.each(function(series) {
    series.appear();
  });
});
chart.events.on("datavalidated", function(ev) {
  chart.series.each(function(series) {
    series.appear();
  });
});
{
  // ...
  "events": {
    "datavalidated": function(ev) {
      var chart = this.target;
      chart.series.each(function(series) {
        series.appear();
      });
    }
  }
});

Standalone usage

You can use amCharts 4 awesome data loading and parsing capabilities outside charts.

A DataSource object can be instantiated in a completely standalone mode.

let dataSource = new am4core.DataSource();
dataSource.url = "http://www.mywebsite.com/mydata.php";
dataSource.load();
dataSource.events.on("done", function(ev) {
  // Data loaded and parsed
  console.log(ev.target.data);
});
dataSource.events.on("error", function(ev) {
  console.log("Oopsy! Something went wrong");
});
var dataSource = new am4core.DataSource();
dataSource.url = "http://www.mywebsite.com/mydata.php";
dataSource.load();
dataSource.events.on("done", function(ev) {
  // Data loaded and parsed
  console.log(ev.target.data);
});
dataSource.events.on("error", function(ev) {
  console.log("Oopsy! Something went wrong");
});

NOTE Creating a DataSource will not trigger a load automatically. You'll need to trigger it using its load() method, as per example above.

Common pitfalls

Cross-domain restrictions

Nowadays, browsers are designed to respect data publisher's rights to control access to their data.

This led to a situation that by default browsers will not let you load data across different domains. For example if your web page that displays a chart is located on "www.mywebsite.com", the browser will not let scripts on that page load data from "www.someotherwebsite.com" unless that website explicitly agrees to let data be loaded by everyone or "www.mywebsite.com" in particular.

This is called a Cross-Origin Resource Sharing or CORS in short.

If you take a closer look at the examples above, you'll see that they are all loaded from "codepen.io", while actual data source URLs point to an Amazon server "*.amazonaws.com". Yet they work, because these particular Amazon servers are set up to let "codepen.io" load data.

However, if you move those same examples out of CodePen and placed on your website, they'd stop working. You'd need to move those data files to your server as well, in order not to violate CORS.

MORE INFO For more information on CORS and related configuration options, refer to this Mozilla article.

Local files

For security reasons, browsers will never allow loading of files from a local file system, or when the web page is loaded from a local files system. (URL starts with file:///)

So, if you want to use external data, you always load your web page through a web server (URL starts with http(s)://), even if it's a local web server. (localhost)

Avoiding cache

Browser, being an all friendly and helpful guy, will go out of its way to serve content as fast as possible.

This means, that it will try to cache data as much as possible. This might lead to an older version of the data file being loaded.

If this happens, you can work around it by setting dataSource.disableCache = true. This will make data loader "timestamp" each request so it appears as a different URL every time, effectively disabling cache.

NOTE Use it only if absolutely necessary. Caching is there for a reason.

Related demos