Michael Weiss in his Outline of the Historical and Comparative Grammar of Latin notes that the adverbial suffix -ē (which is not an abl. sg. in Latin synchronically, as Bitmap said) has an unclear etymology. It is either an ablaut variant of the masculine/neuter abl. sg. -ōd (with the classic -ē-/-ō- alternation), or the Proto-Indo-European instrumental case ending -eh1 (after receiving an analogical -d as influence from the ablative, the -d is relevant because in Old Latin it is attested as <-ED>).Rather more to the instant theme, though, I have been able to discern that adverbs have been formed in Latin from masculine and feminine in both the nominative (Latin satis, versus) and the accusative (Latin perperam, prōmiscam), as well as from the ablative singular (Latin -ē), in addition to the "petrified" neuter accusative which was the subject hereof.
It may also interest you to hear that bonus and malus (Archaic Latin duenos/malos) are believed to have originally derived the adverbs *duenē and *malē, which then underwent the final-vowel shortening characteristic of two-syllable words with a short vowel in the stem ("iambic shortening" is the technical term), becoming Classical Latin bene and male (with short -e).
I think only adverbs of manner and frequency can be productively formed, with -ē/-ius/-(i)ter (manner) and -iē(n)s (frequency: five times, ten times... note the -ter in quater). Latin exhibits morphological comparative adverbs formed with manner endings. So for these, your question should rather be the origin of these very few specific suffixes.The question about this which occurs to me, is: adverbs have been formed in Latin out of such "petrified" nominatives, accusatives, and ablatives. Is there any rationale for which petrified case form is and has been properly used for the production of the various types of adverb: adverbs of manner, adverbs of time, adverbs of place, adverbs of frequency, and adverbs of degree?
The rest of adverbs are basically very lexical and should be probably studied one by one, and I don't think there is any pattern to be identified in the cases they descend from.
For example, multō and multum 'very' even appears in two different cases with hardly any difference in use. In terms of adverbs of time, herī 'yesterday' descends from a Proto-Indo-European locative *dʰǵʰyés-i (> Proto-Italic *xes-i > the Classical Latin variant here) with the more common locative ending -ī attached (cf. hum-ī 'on the ground', dom-ī 'at home'), but crās 'tomorrow' probably descends from an Indo-European genitive (PIE *kreh2-es), and hodiē 'today' is pretty transparently 'this day' in a locative use of the Latin ablative (cf. hōc diē 'on this day', the noun part -diē of hodiē likely continues an ablative cf. Faliscan foied, but note the ho- part may possibly descend from a Proto-Italic locative, rather than ablative *hōd).
It may interest you to read about the regular syntactic uses of the accusative, ablative and dative cases to make adverbial phrases of time/space/degree out of noun phrases. But this is syntax, not adverb derivations. For this, consult the various sections on the uses of cases in Allen and Greenough's grammar, including §425.b on the accusative to express extent of space (which you might miss). It may interest you to read on the ablative of specification (which is a little bit weird from the point of view of general linguistics, or at least it's not a syntax role linguists often talk about: compare the first example Virtūte praecēdunt 'They excel in courage' with the use of syntactic topics for the same thing in topic-prominent languages like Mandarin Chinese: 勇氣他們是最好的 yǒngqǐ tāmen shi zuì-hǎo de, fairly literally "courage, they be best").