Usually (With OpenAI, I haven't checked Mistral yet) it means an async api rather than a sync api.
e.g. you submit multiple requests (pdfs) in one call, and get back an id for the batch. You then can check on the status of that batch and get the results for everything when done.
It lets them use their available hardware to it's full capacity much better.
e.g. you submit multiple requests (pdfs) in one call, and get back an id for the batch. You then can check on the status of that batch and get the results for everything when done.
It lets them use their available hardware to it's full capacity much better.