Best Practices for reading csv data with d3 in node. Why gettting a TypeError?

Question

I am trying to read a csv file with d3 in a node-environment. I am kind of confused what the "best-practice"-way therefore is and why I keep on getting a TypeError. I know that for some module of d3a environment that supports fetch must be present. Therefore my code at the moment looks like this:

import * as d3 from "d3";
import fetch from "node-fetch";
global.fetch = fetch;

let data_path = "flightCodes.csv";


// would a d3.csvParse(fs.readFileSync(data_path)) be better?
let data = await d3.csv(data_path, function (err, dat) {
  if (err) {
    console.log(err);
  } else {
    console.log(dat);
  }
});

console.log(data);

The csv file lives in the same folder and looks like this:

orig,dest,orig_codes,dest_codes
Paris,Lyon,"BVA, ORY, CDG",LYS
Paris,Bordeaux,"BVA, ORY, CDG",BOD
Paris,Nantes,"BVA, ORY, CDG",NTE
Madrid,Barcelona,MAD,BCN
Lisboa,Porto,LIS,OPO

My package.json looks like this:

{
  "name": "name",
  "version": "1.0.0",
  "description": "",
  "type": "module",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "d3": "^7.2.0",
    "d3-fetch": "^3.0.1",
    "node-fetch": "^3.1.0"
  }
}

When I do a node script.js, I get the following and I do not completely understand why. Anybody might have an idea?

node:internal/errors:464
    ErrorCaptureStackTrace(err);
    ^

TypeError [ERR_INVALID_URL]: Invalid URL
    at new NodeError (node:internal/errors:371:5)
    at onParseError (node:internal/url:552:9)
    at parse (<anonymous>)
    at new URL (node:internal/url:632:5)

Answer 1

If the file you're referencing is not available through a URL, then d3.csv (which uses fetch) cannot find it. Internally, d3.csv fetches the data and then calls d3.csvParse to parse it. Fetching the data requires an external URL, so you cannot reference a local file.

However, you can use the fs module to read the file locally and then call d3.csvParse to get the same result as d3.csv does!

let data = d3.csvParse(fs.readFileSync(data_path));

---

In addition, there was another problem with your code. The signature d3.csv(path, callback) is deprecated. Instead, it will execute the second argument on every row as a parse function. The correct way to call it would be d3.csv(path).then(callback) or

const data = await d3.csv(path);
callback(data);