Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to determine which size buffer to use when using parBuffer in Haskell

The documentaion for parBuffer says

parBuffer :: Int -> Strategy a -> Strategy [a]
Like evalBuffer but evaluates the list elements in parallel when pushing them into the buffer."

evalBuffer :: Int -> Strategy a -> Strategy [a]
evalBuffer is a rolling buffer strategy combinator for (lazy) lists. evalBuffer is not as compositional as the type suggests. In fact, it evaluates list elements at least to weak head normal form, disregarding a strategy argument r0."

The first part of my question is does this mean that parBuffer also disregard its strategy argument? Also why would it even provide a Strategy argument if it just disregards it?

The second part of my question is how does one determine which size of buffer to use with parBuffer? If parBuffer always maintains a buffer size of n, what is the difference between n=1 and n=10 other than more sparks must be kept in memory? I was thinking it might be wise to pick n=<number of threads>" but I don't know how that would improve anything because parBuffer will still create a spark whenever a spark is consumed, regardless of whether n=1 or n=4.

like image 882
Justin Raymond Avatar asked Dec 05 '25 07:12

Justin Raymond


1 Answers

First answer: parBuffer takes a Strategy for evaluating one element of a list, and gives you back a Strategy for evaluating the entire list. So yes, it evaluates the list in parallel. But it still needs to know how to evaluate each item. (Did you want normal form or weak head normal form or...?) So it doesn't ignore the first argument.

Second... Benchmark it? I suspect the answer depends on exactly how much work each spark does, how many data dependencies you have, etc.

like image 85
MathematicalOrchid Avatar answered Dec 07 '25 22:12

MathematicalOrchid



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!