Is it faster than webgpu? If so, what makes it so?

lxe · on April 21, 2023

It is not. It's much slower. But it doesn't require you to use a canary browser with a command line flag, so that's that.

EDIT: I guess I should give credit here... 100-200ms/token with a 1.7B model is not slow. Would love to do a benchmark with webgpu and see how it compares.

terrycody · on April 22, 2023

Sorry for a layman like me, what does this differ with ChatGPT web interface on openAI website? I can't get it what it does.

3np · on April 22, 2023

It runs offline locally on your machine, as opposed to on remote servers.