Matthew 5:37 records that Jesus said something similar during his 'Sermon on the Mount' (which predates the Epistle of James): "All you need to say is simply “Yes,” or “No”; anything beyond this comes from the evil one."

I also read a 20th century book which made the interpretation that "thou shalt not take the name of the Lord thy God in vain" of the Ten Commandments is an instruction not to swear oaths on the Bible, since everyone is fallible and so can't guarantee them anyway. I can't remember off-hand which book it was though.

mmm... has I remember you can... only not in vain...