I tried to read HTTP response with axios and parse JSON in stream mode with stream-json` so I can fully fit my database on demand. It works well but if I try to close database connection everything will be crashed because the connection will be closed too soon. The problem is: await doesn't wait for the extract_coins function to complete (even if it's returning the promise) and close database connection in a final scope.
const main = async() => {
const dbcfg = config.get('db.coins');
const db = await coins_db_configure(dbcfg);
try {
console.log('Connected to the database and created coins table!');
await extract_coins('some_url_which_gives_correct_json', coins_emitter);
}
catch(e){
console.error(e);
}
finally {
await db.close();
}
};
main();
extract_coins:
module.exports = async function extract_coins(url, emitter){
return await axios({
method: 'get',
url: url,
responseType: 'stream'
}).then((res) => {
const pipeline = chain([
res.data,
parser(),
pick({filter: 'data'}),
streamArray()
]);
pipeline.on('data', data => {
emitter.emit('coin_extracted', data.value);
});
pipeline.on('end', () => console.log("All the coins were successfully passed!"));
});
};
As the code related to the asynchronous pipeline is not promisified, you currently have no way to return a promise that resolves on getting the "end" event.
Your await does wait for the promise to resolve, but your then callback returns undefined, and so that resolves the promise at that very moment, long before the end event is broadcast.
So change this:
then((res) => {
const pipeline = chain([
res.data,
parser(),
pick({filter: 'data'}),
streamArray()
]);
pipeline.on('data', data => {
emitter.emit('coin_extracted', data.value);
});
pipeline.on('end', () => console.log("All the coins were successfully passed!"));
});
To this:
then((res) => new Promise((resolve) => {
const pipeline = chain([
res.data,
parser(),
pick({filter: 'data'}),
streamArray()
]);
pipeline.on('data', data => {
emitter.emit('coin_extracted', data.value);
});
pipeline.on('end', () => {
console.log("All the coins were successfully passed!");
resolve();
});
}));
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With