Encode complex JSON structure using Perl

Question

I want to encode the result of a MySQL query into a JSON string using JSON::XS. The JSON string needs to look like this

{
    "database" : "dbname"
    "retentionPolicy" : "mytest",
    "tags" : {
        "type" : "generate",
        "location" : "total",
        "source" : "ehz"
    },
    "points" : [{
            "precision" : "ms",
            "timestamp" : "ts1",
            "name" : "power",
            "values" : {
                "value" : "val1"
            }
        }, {
            "precision" : "ms",
            "timestamp" : "ts2",
            "name" : "power",
            "values" : {
                "value" : "val2"
            }
        }, {
            "precision" : "ms",
            "timestamp" : "ts3",
            "name" : "power",
            "values" : {
                "value" : "val3"
            }
        }
    ]
}

The points array with each point's values element is giving me immense headaches.

Here is the code block that generates the JSON

my %json_body = (
                'database' => $db_name,
                'retentionPolicy' => $retention,
                'tags' => {
                    'source' => $metric_source,
                    'type' => $metric_type,
                    'location' => $metric_location
                }
                );

# loop through mysql result
while ( ($timestamp, $value) = $query->fetchrow_array() ) {
    my %json_point1 = (
                        'name' => $series_name,
                        'timestamp' => ($timestamp * 1),
                        'precision' => "ms"
                      );
    %json_point2 = ('value' => $value); 
    %json_values = (%json_point1, 'values' => \%json_point2); 
    push(@all_values, \%json_values);       
}
$query->finish();

# Encode json
my %json_data = (%json_body, "points" => \@all_values);
$influx_json = encode_json(\%json_data);

I think the line push(@all_values, \%json_values) is my problem. If I pass %json_data as a hash reference, only the last value from the while loop is retained. If I use %json_values directly, the encoded JSON is messed up because it loses the structure.

Any hint would be appreciated. And please bear with me: this array and hash references are already making my head explode.

Answer 1

I'm pretty sure you problem will be because you're using a globally scoped hash for %json_point and %json_point2. You see, the root of this is - you simply don't get a list of hashes. You get a list of hash references.

So the problem here is - when you push a reference to your hash into @all_values - you're pushing the same reference each time. But then you're overwriting the contents of the hash that you're referencing.

Try this:

#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;

my %hash_thing;
my @all_values; 

for ( 1..3 ) {
   %hash_thing = ( "test" => $_ );
   push ( @all_values, \%hash_thing ) ; 
}

print join ( "\n", @all_values );
print Dumper \@all_values;

And you'll see you have the same 'value' 3 times:

HASH(0x74478c)
HASH(0x74478c)
HASH(0x74478c)

And so if you dump it, then of course - you don't get the right array - and so your encoded JSON doesn't work either.

$VAR1 = [
          {
            'test' => 3
          },
          $VAR1->[0],
          $VAR1->[0]
        ];

The simplest fix is to use my to scope the hashes to the loop. (And turn on use strict; and use warnings if you haven't.)

Alternatively, you can use a hash reference like this:

my @all_values; 
my $hash_ref;

for ( 1..3 ) {
   $hash_ref =  { "test" => $_ };
   push ( @all_values, $hash_ref ) ; 
}

print @all_values;
print Dumper \@all_values;

Because $hash_ref is a scalar, and it's a reference to an anonymous hash, it can be inserted into the array by value, rather than by reference.