Mr Trump also named Gina Haspel to become the CIA's first female director.
How did Tillerson react?
Speaking to reporters at the Department of State, Mr Tillerson said good work had been done to have better relations with China, and rein in North Korea's nuclear weapons programme.
But he added: "Much work remains to respond to the troubling behaviour and actions on the part of the Russian government.
"Russia must assess carefully as to how its actions are in the best interest of the Russian people and of the world more broadly. Continuing on their current trajectory is likely to lead to greater isolation on their part, a situation which is not in anyone's interest."
Sounding breathless and looking shaken, Mr Tillerson said he had spoken to the president to ensure clarity in the days ahead.
Although Mr Tillerson will not officially leave until 31 March, his deputy John Sullivan will take over running the state department pending the confirmation of Mr Pompeo.
"What is most important is to ensure an orderly and smooth transition during a time that the country continues to face significant policy and national security challenges," Mr Tillerson said.
Mike Pompeo, Director of the CIA, will become our new Secretary of State. He will do a fantastic job! Thank you to Rex Tillerson for his service! Gina Haspel will become the new Director of the CIA, and the first woman so chosen. Congratulations to all!