Machine learning serving using either kserve, seldon, or BentoML [closed]

Question

What is better, kserve, seldon core or BentoML? And what are the advantages /disavantages and feature of each one?

I did a lot of research and couldn't find a clear answer.

Answer 1

I'm in a similar position where lately I've been looking around the model serving landscape to choose what stack/tech go for. Currently, we're using FastAPI to wrap models into microservices, but we want to split I/O-bound and network-bound consumption (usually from business logic) from compute/memory-bound consumption (models), and also better orchestration (scaling, traffic distribution for A/B tests, etc.).

Generally you have two kinds of tools:

1. Inference servers, which deal with wrapping the model into a microservice
2. Servers orchestrators, which add orchestrating features for scaling, deploying and generally managing the server fleet

BentoML is a model server, and the direct comparison wouldn't be to Seldon Core or KServe, but rather to Seldon Core's MLServer/Python Client and KServe's KFModel (which in turn uses Ray). I feel like their feature set is very similar, so which one is best depends on experience/trial and error. Personally, I went for BentoML at this moment, because it seemed the easiest to iterate on, but I wouldn't exclude switching to the others if BentoML doesn't work as well.

Seldon Core and KServe are more orchestration tools, meaning that their feature set, while including inference servers, also extends beyond that. BentoML also has an orchestration tool, Yatai, but I feel like it's still lacking in features compared to the above two. The good news is that I believe Seldon Core and KServe should work with most inference servers tech (namely BentoML), although some features might be degraded compared to using their own solutions.

I don't have a clear cut answer to which one is best and from my research, it seems people seem to use all of them in some form or another, like:

• BentoML + Helm for deployment
• BentoML + Seldon Core
• Seldon's prepackaged inference servers/custom + Seldon Core
• BentoML + KServe

My personal suggestion is to try out the quickstart tutorials of each and see what best fits your needs, generally going for the path of least resistance - the MLOps landscape changes a lot and quickly, some tools are more mature than others, so not investing too much in a hard tool makes most sense to me.