You'd have to go all the way back to 300 AD or so, to a time when there was probably no such phrase as "going to church". As I read the history it seems that back then the saints thought of themselves as the church, not as some place you go. The New Testament word for church is "
eccelsia" which refers to being called out. We are called out of a life of adherence to the dominance of our biological and neurological impulses, to a life where the Holy Ghost leads us into making those decisions that agree with the Lord's will for our lives. The church then was intended to be an organic community of believers who came together everyday to help each other grow and to reinforce their ability to participate in what was happening to them. Yes something was happening to them and should be happening to us-every waking moment the "calling out" process of sanctification claims another area of our psyche and decision making. We willingly relinquish control of those areas because we know the Lord has a more fulfilling option for us.
So when did we start to go to church? I don't really want to malign Constantine, but a lot of stuff sorta' began to go awry when he came on the scene in the early 300s. The beautiful cathedrals were originally designed as an expression of the glory of the Lord. (I have seen many of them in Europe and they are amazingly beautiful) Over time though, people began to conceive that the building was where the Lord was. The more a person entrusts their knowledge of spiritual things solely to others, the more they lose the reality of the Lord's active presence. And so, you have to go somewhere else to find him.
We love coming together to worship. The scriptures encourage us to do so. But we try not to "go to church" We try to "be" the church.