This has deep roots in early modern secularizing thinkers like Bacon and Locke. In A Letter concerning Toleration (1689), Locke argued that the state should tolerate religious differences (even including Jews, and I think Muslims), with two exceptions - atheists (you can't trust them to be good citizens because they don't believe God will punish them for treachery) and Catholics (you can't trust them to be good citizens because their first loyalty is to a foreign potentate, i.e. the pope).
My philosophy surrounding religious difference is that in order to gain the right to toleration you must give some form of loyalty test. If the Vatican assures loyalty somehow, you may tolerate Catholics.
Secularism to me is not no religion whatsoever, it is no established religion.