Recursive extraction of object values and parent key name using jq

Question

I have a requirement to parse the output of the npm ls --global --json command, such that I get a list of all the installed npm packages in the following format:

$package;$version;js;$resolved

Where:

• $package is the key containing the package name, from each dependencies object.
• $version is the version value taken from each package
• js is just a literal string
• $resolved is the resolved value taken from each package

I have gotten as far as this command syntax and output:

$ jq --raw-output 'select( has("dependencies") ) .dependencies[] | . as $d | "parentkey" + ";" + $d.version + ";js;" + $d.resolved'`
parentkey;5.5.1;js;
parentkey;1.1.3;js;https://registry.npmjs.org/yaml-table/-/yaml-table-1.1.3.tgz

The parts that I am specificly having difficulty with are as follows:

• How can I get the key name value that I am iterating over in .dependencies that contains that package name. It seems that by that point I am looking at the contents of that object itself.

• How can I recurse through ALL dependency objects? At the moment I'm only looking at the top level records in the root .dependencies object. I've discovered .. recursion, but I'm not quite sure how to apply it here.

Based on the example data below, I am trying to reach the following output results:

npm;5.5.1;js;
JSONStream;1.3.1;js;https://registry.npmjs.org/JSONStream/-/JSONStream-1.3.1.tgz
jsonparse;1.3.1;js;https://registry.npmjs.org/jsonparse/-/jsonparse-1.3.1.tgz
through;2.3.8;js;https://registry.npmjs.org/through/-/through-2.3.8.tgz
yaml-table;1.1.3;js;https://registry.npmjs.org/yaml-table/-/yaml-table-1.1.3.tgz
js-yaml;3.4.6;js;https://registry.npmjs.org/js-yaml/-/js-yaml-3.4.6.tgz
argparse;1.0.9;js;https://registry.npmjs.org/argparse/-/argparse-1.0.9.tgz

Some (much reduced) sample output npm ls --global --json that I have used for the above example, is as follows:

{
  "dependencies": {
    "npm": {
      "version": "5.5.1",
      "dependencies": {
        "JSONStream": {
          "version": "1.3.1",
          "from": "JSONStream@~1.3.1",
          "resolved": "https://registry.npmjs.org/JSONStream/-/JSONStream-1.3.1.tgz",
          "dependencies": {
            "jsonparse": {
              "version": "1.3.1",
              "from": "jsonparse@^1.2.0",
              "resolved": "https://registry.npmjs.org/jsonparse/-/jsonparse-1.3.1.tgz"
            },
            "through": {
              "version": "2.3.8",
              "from": "through@>=2.2.7 <3",
              "resolved": "https://registry.npmjs.org/through/-/through-2.3.8.tgz"
            }
          }
        }
      }
    },
    "yaml-table": {
      "version": "1.1.3",
      "from": "yaml-table@latest",
      "resolved": "https://registry.npmjs.org/yaml-table/-/yaml-table-1.1.3.tgz",
      "dependencies": {
        "js-yaml": {
          "version": "3.4.6",
          "from": "[email protected]",
          "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.4.6.tgz",
          "dependencies": {
            "argparse": {
              "version": "1.0.9",
              "from": "argparse@>=1.0.2 <2.0.0",
              "resolved": "https://registry.npmjs.org/argparse/-/argparse-1.0.9.tgz"
            }
          }
        }
      }
    }
  }
}

Answer 1

By using .., it will recurse through all values in the json tree. So you'll want to filter those out by objects that have the structure you're expecting. In this case, things that have a valid dependencies object. Once you've located the objects, you could extract the values you want.

jq -r '.. | .dependencies? | objects
    | to_entries[] | [.key, .value.version, "js", .value.resolved] | join(";")' input.json

produces the results:

npm;5.5.1;js;
yaml-table;1.1.3;js;https://registry.npmjs.org/yaml-table/-/yaml-table-1.1.3.tgz
JSONStream;1.3.1;js;https://registry.npmjs.org/JSONStream/-/JSONStream-1.3.1.tgz
jsonparse;1.3.1;js;https://registry.npmjs.org/jsonparse/-/jsonparse-1.3.1.tgz
through;2.3.8;js;https://registry.npmjs.org/through/-/through-2.3.8.tgz
js-yaml;3.4.6;js;https://registry.npmjs.org/js-yaml/-/js-yaml-3.4.6.tgz
argparse;1.0.9;js;https://registry.npmjs.org/argparse/-/argparse-1.0.9.tgz