He told CNN that they used to also pray with Obama