By itself cave ab homine unius libri is not grammatical. Possibly it is genuine and something was left out: "Beware (someone) of one book (written) by man." Yes, that is a stretch.
Your Cave hominem unius libri is grammatical and means "beware the man of one book."
Ab homine is correct as well; as Laurentius said, cavere can also be construed with ab + abl. - and it already could well before Thomas Aquinas. Perhaps it's just a bit less frequent than with an accusative object, I don't know.