Split public LLM provider code from internal

Split public LLM provider code from internal #804