I can see how Christian books could be useful; however, there are so many Christian books out there; Christian books may just be too broad. You might still find yourself having to explain and search for more specific adjectives that won't imply what you want to avoid.
Let me draw on a little seminary training to provide you with some further suggestions that don't concede to "Testament" or "covenant":
The 27 Books
Christian Writ
Primary Apostolic Writings
Primary Christian Writings
A bit more eccentric:
The Christian Rule of Faith
The Christian Anthology
It might be helpful to refer to the collection by its components: Gospels, Acts, Letters and Revelation, either list them, or refer to the particular part that is pertinent to conversation. Monica may have mentioned that.
More specific concepts may be used of the original Greek documents, like the Greek Canon, Christian Greek Canon, or any of the above, replacing writings with documents, manuscripts, parchments, or corpus.
A bit too obscure:
Early Christian writings
Greek Christian writings
Apostolic writings
One might think using earliest instead of just early would remedy this; however, earliest has connotations concerning (carbon) dating of the manuscripts and if you merely intend to imply foundation, then, primary would be the word of choice, as foundational would be a mouthful.
I hope this equips you some to communicate more effectively about such matters.