Hi everyone! A fun project I have been testing and optimizing on NVIDIA RTX Pro 6000 single. See current results above 
. Have few items to optimize in next 1-2 days. The end point will be available on Hugging Face. Anyone who needs to test with API key - message me and will generate one for you to test. Economics are great, ultra cheap. 
What kind of interesting 
aha moments have you had while working on this?
Hi @maxchisto what was interesting about it that I compared economics like cost of server per minute vs speed and efficiency on midsize models - the economics on larger gpu didn’t make sense vs current setup. Also I compared pricing $/speed vs other current providers on Hugging Face.
Basically the more expensive gpu doesn’t always make something more efficient and faster - but the proper fine tuning is the key! Thanks to @clattner and @Modular this is possible today!