How do I deserialize the byte stream from a reqwest response into JSON?

Question

A request via reqwest is expecting a long poll response, so I created a byte stream and tried to deserialize each chunk into JSON. I immediately realized that it is wrong because each chunk can be incomplete so deserialization can fail; even though some of chunks were deserialized into JSON, most of them were failed. How do I do this?

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = reqwest::Client::new();
    let mut stream = client
        .get("https://...")
        .send()
        .await?
        .bytes_stream();

    while let Some(chunk) = stream.next().await {
        match serde_json::from_slice::<Value>(&chunk?){
            Ok(value) => println!("OK: {:?}", value),
            Err(e) => println!("ERROR: {:?}", e),
        }
    };

    Ok(())
}

Answer 1

If you enable feature json from the reqwest library in your Cargo.toml. Then you could simplify the code to this:

let json = client
    .get("https://...")
    .send()
    .await?
    .json::<T>() //T needs to implement serde::de::DeserializeOwned
    .await?;

This will send the request and automatically convert it to JSON. Since the json method internally uses serde_json::from_reader, it's also deserializing the byte stream to a struct.

This will fail if the if the response body is not in JSON format or it can't be properly deserialized to T.