He was arrested by the security upon his suspicious moves.
The National Security Secretariat is investigating the issue, a statement from the Communications Ministry said.
“The man was arrested after his fidgety and suspicious behaviour was noticed during the morning service. He was escorted out by security personnel and following a search, the weapon was found.
“President John Dramani Mahama and the First Family are members of the Church but were absent when the incident occurred,” the statement signed by Dr Edward Omane Boamah said.