Dealing with asynchronicity in nodejs has been a challenge from day one due to its non blocking nature. The evolution has been slow and the node world has moved from callback hell to promises and from promises to generators.
I’ve used async and await in
c# where its addition is still great but its impact is not quite as dramatic.
async and await, you can write asynchronous code that for all intensive purposes looks synchronous by marking functions as
async and and prefixing function invocations with
await to indicate execution will be deferred until a promise is returned from the function call after
Below is a very simple async function:
1 2 3 4 5 6 7 8 9 10 11
The function is marked as async with the
async keyword on line 1. Any function that will
await on the result of a function that returns a promise must be marked as
await keyword allows you to
await on the result of a promise. The
getJSON function that is called on line 5 returns a promise and any function that is called with
It is worth pointing out that promises are the glue that makes all this possilbe and they are still one of the most important primitives in node.
npm install the babel-plugin-transform-async-to-generator package and add a reference to the package in your
Below is my
.babelrc that allows me to use async await and other features:
1 2 3 4 5 6 7
Streams and Async..Await
Csv-parse creates a readable stream that emits
data events each time it encouters a chunk of data. CSV-Parse allows me to bind to a readable event that gets passed a row of a csv file for each row in that particular file.
The package I wrote allows the user to specify transformations from a source csv file to a destination table. Some of these transformations might involve one or more asynchronous actions that could end up as some pretty messy code if I just used promises so
async and await seemed like a great fit. I will post the original code at the end of the post that used promises and it is very hard to follow and deeply nested.
Below is the code that hooks up the csv-parse stream events to member functions in the class that will handle the events:
1 2 3 4 5 6
Line 6 specifies that the
onReadable is bound to the
readable event of the stream.
My first attempt at using async and await with the stream is below where I marked the
onReadable function that gets rows of csv data passed to it as an
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
The code calls
createObjectFrom on line 12 which is itself an async method that returns a promise and adds it to the
this.records array that will be used to persist the transformed values to the database.
Below is a scaled down version
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
The code loops over an array of transformations, one of which might be an async call to
knex on line 9 to retrieve a value from the database.
A promise is returned on line 23 to allow this function to be called with async and await.
I wrote a test to test this function in isolation and I was buoyed when it passed but when running the code for real with stream
readable events being raised, the
this.records array was empty when it the
end event of the stream was reached:
1 2 3
This is because the
onReadable function is not being called async from the csv-parse module that raises the evnets.
After much head scratching the answer was to return a promise from
createObjectFrom. But the interesting part of this solution was that I was able to mark the function that gets passed to the Promise constructor as
async on line 2 of the code below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
On line 2, I mark the anonyous function that gets passed to the Promise constructor as
async and this allows me to invoke functions with
await and only resolve the promise on line 24 when all the processing has finished.
onReadable event that calls this function now looks like this and is no longer async:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
createObjectFrom now returns a promise that will resolve later and
onReadable works as advertised.
Lines 12 and 13 simply add each promise that is returned from
createObjectFrom to the
this.promises array that can be processed later.
onEnd event now uses
Promise.all to wait for all the promises to resolve before inserting the resolved values into the database:
1 2 3 4 5
This works nicely and is much better than the Promise handling code at the bottom of this post.
async and await in tests
Async and await can also be utilised to make your testing code much simpler and synchronous looking.
Below is my test to ensure that the records are being inserted into the database:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Normally when testing promises from mocha, you have to return a promise from the mocha
it function in order for execution to wait until the promise has resolved but as I have marked the anonymous function on line 2 as
async, I can simply
await (line 6) for the asynchoronous csv parsing code to finish before testing the results.
I can also make further asynchronous calls in the anonymous function like I am on line 14 and
await their results before asserting expectations.
This is flatter and synchronous looking code that we all know well.
Below is the code I mentioned previously which used promises instead of async await and is pretty damn nasty.
The async await version is considerably cleaner.
Feedback is welcomed in the comments below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96