Turn an LLM into a Good Problem Solver