However, Huguenot immigration was so great that after a few years the congregation became too large for the building. In 1704 a new and larger church was built at the corner of Pine and Nassau Streets and was called, for the first time, "Le Temple du Saint-Esprit." It was to serve the parish for the next 130 years. The church was a simple rectangular building, 50 by 75 feet. Beside it was a graveyard. There was a wooden fence on the sides which bordered the streets. It possessed a small tower which was surmounted by a cupola. By all accounts, it looked like a small country church.

Mr. Peiret died in 1704, before the new church could be completed. His successor, Jacques Laborie, had a brief ministry at Saint-Esprit. Mr Laborie had received teleological training in Zurich before studying medicine in London. He was sent to America by an Anglican missionary society. After assuming his duties as pastor he began to pressure the parish to adhere to the Church of England. When it would not, he resigned, two years after his arrival, and moved to Connecticut where, for the rest of his life, he had a rather distinguished career as a physician.

Louis Rou, who succeeded Dr. Laborie, was pastor of Saint-Esprit for the next 40 years until his death in 1750. He was an excellent scholar, he was widely respected, and he possessed a forceful personality. In addition to numerous volumes of .learned sermons, all being of a length and complexity typical of the period, he also wrote poetry, both religious and secular. He is credited with having introduced the game of chess into the colonies. Thanks to his able leadership and the large number of Huguenots in New York, the parish thrived during much of his long ministry.

However, beginning around the 1730s, the membership of Saint-Esprit steadily declined for several reasons. Huguenot immigration to the American colonies, after the great influx during the period following the Revocation of the Edict of Nantes, dwindled to a small number. Louis XIV died in 1715 and his successors, while still not recognizing the practice of the Reformed religion, had no great zeal for persecuting it. French immigrants for reasons of conscience were few in the 18th century. Those Huguenots who came to America during the great immigration followed the familiar pattern of working hard, prospering, and assimilating rapidly. Many of them became distinguished in their communities and as they did so, other, larger churches eagerly sought their support and leadership. The lists of English and Dutch congregations in the 18th century are filled with Huguenot names. The children of the immigrants were no longer comfortable worshipping in the French language. Against these counter trends, Mr. Rou struggled with diminishing success. Revenues fell and were not enough to cover expenses. As the size of the congregation diminished, petty issues began to split the remaining elders and members. By the time of Mr. Rou's death in 1750 Saint-Esprit was in a very weakened state from what it had been.

There followed the most difficult period in the long history of the parish. Elders and parishioners quarreled constantly and often split into opposing factions. These squabbles became well-known in the general community and, even worse, in the Reformed churches of Europe. Because of this, and the precarious state of the parish finances, no qualified candidate would agree to come to New York as pastor from France or Switzerland. In the absence of a full-time pastor, a succession of lay readers tried to hold things together with little success. The strength of the congregation had so declined that by the time of the Revolutionary War, it had virtually disappeared. When the British invaded New York, they requisitioned the church building as a storehouse for arms and ammunition. Regular worship services ceased for almost 20 years.

The revival of Saint-Esprit was the product of a fortunate occurrence in 1795. A Swiss clergyman, J. Louis Duby, passed through New York, became interested in the plight of the French Church, and decided to do what he could. The church building was in a state of dilapidation and could not be used. The French Protestant community was almost non-existent. The Reformed religion had been officially recognized in France by the Edict of Toleration of January 1787 and the refugees during the French Revolution were mostly of the Roman Catholic nobility. Only one elder from former days was still alive. However, Mr. Duby was able to contact a few of the former congregation, enough to form a small group. A notice was put in a local paper asking all interested people to assemble for the purpose of re-establishing the parish. A meeting was held on January 26, and from that small number present a new board of elders was elected. The new elders then applied for incorporation under the laws of the State of New York. Incorporation had never been granted during the colonial era because Saint-Esprit did not belong either to the officially established Church of England or to one of the churches established before English rule, such as the Dutch Reformed Church. There turned out to be more interest in the French Church than anyone had suspected and, encouraged by this modest success, the elders asked Mr. Duby to be their pastor. Planning to return to Switzerland, he declined the offer, but he promised to look for a candidate upon his return. True to his word, he found a young man of 30 named Pierre Antoine Samuel Albert whom he recommended highly. The elders issued a call; Mr. Albert accepted and arrived in New York from Switzerland in 1797. He was the last pastor and the first rector of Saint-Esprit.