Historical modes
The following list details how historical theorists within the Western tradition categorized the scales (also known as modes) in use in their time.
Lucian of Samosata
The smallest mode system that Zarlino mentions was the four-mode system written by Lucian of Samosata:
exalted Phrygian, joyous Lydian, majestic Dorian, voluptuous Ionic — all these I have mastered with your assistance.
No elaboration is given by Lucian as to the nature of these modes aside from these vague descriptions of their character; however, he treats them as if they were a comprehensive list.
Because Lucian lived in the second century, it could be inferred that he's talking about Greek tone species as described by Aristoxenus; thus, the four modes could represent the four rotations of the tetrachord.
This would give us:
Phrygian: MsLLMsL / LsMLLsM
Lydian: LMsLLMs / MLsLMLs
Dorian: sLMLsLM / sMLLsML
Ionian: LLsMLsM / LMsLMsL
Where L is 9/8, M is any one of [10/9, 9/8, 8/7] depending on the desired tuning, and s is the amount by which LM falls short of 4/3.
Note that Ionian acts as effectively the 0th rotation, since it starts on the 9/8 step between the two tetrachords rather than utilizing some rotation of the tetrachord itself.
Aristoxenus and the Five-Mode System
The next smallest system is the five-mode system and its superset systems; this was apparently a pretty popular one among theorists before and during Zarlino's era, being supported by theorists such as Cassiodorus, Apuleius, and Martanius Capella; Zarlino and Capella also attribute this system to Aristoxenus, though in Stoecheia Harmonica Aristoxenus appears to be of the opinion that the "modes" are just the rotations of the (MV3) diatonic scale.
Cassiodorus
Cassiodorus in particular defines the five base modes with respect to their root note rather than a scale pattern, each separated from one another by a semitone: Dorius, Jastius, Phrygius, Aeolius, and Lydius. These five modes represent five equal divisions of 4/3.
This 4/3 separator is treated as a formal equave: notes separated by 4/3 are given the same base name, with "hyper" added for notes 4/3 above the main range and "hypo" added for notes below that range.
Apuleius
Apuleius attests the five base modes, without regard for their "hypo" and "hyper" forms:
He had skill, too, to make music in every mode, choose which you would, the simple Aeolian or the complex Ionian, the mournful Lydian, the solemn Phrygian, or the warlike Dorian.
Just like Lucian, it's unclear what the actual content of these modes entails. And unlike Lucian, five is a particularly unclear number to work with for a system such as this, as it's difficult to arrange five modes with respect to the Greek tetrachords without leaving huge holes in the categorization.
One guess is that they could be the rotations of diatonic, not considering the two modes which exclude of the perfect fourth and perfect fifth respectively; this is substantiated by the enumeration of the modes by Aristoxenus as (from brightest to darkest w/ respect to mosdiatonic) Hypolydian, Lydian, Ionian, Phrygian, Aeolian, Dorian, Hyperdorian.
Note that "hypo" and "hyper" similarly refer to alteration by 4/3 in this system; in other words, "hyper" would be one step darker than the base mode, and "hypo" one step brighter. Thus, the exclusion of the outer two modes is likely simply because they are considered altered forms of the existing modes, and not considered in their own right due to the missing perfect consonances.
Plato
Plato's conception of the modes was significantly less favorable.
He believed that there were six (Mixolydian, Syntonolydian, Lydian, Ionian, Dorian, Phrygian), and that the former four of them should be banned; Mixo and Syntono because they were too sad, and Lydian and Ionian because they were too happy.
Again, no indication of what these modes are.
Julius Pollux
Chapter IV of Julius Pollux' Onomasticum labels six modes, split into two trios of "masculine" and "feminine" modes. The masculine modes are called Doris, Ias, and Aiolis; the feminine are called Phrygius, Lydius, and Lochriche.
It seems that these modes are somehow meant to be paired together (Doris/Phrygius, Ias/Lydius, and Aiolis/Lochriche), but it's unclear what these pairings represent musically, nor what exactly the "gender" of a mode entails.
To make matters worse, these modes seemingly do not correspond to Aristoxenus' mode names, because Aristoxenus describes Aeolian and Locrian as two potential names for the same mode.
Cleonides
Cleonides' description of the modes was rather conventional for its time, considering them to be rotations of some immutable pattern; but the patterns that he defined for the starting point were the interesting part.
Cleonides described 9/8 being cloven in twelve formally-equal parts, with thirty of them reaching 4/3; this gives us 72edo as our base tuning system.
Additionally, he defined six distinct patterns, rather than the three typical genera. He gives each scale in their Mixolydian mode to start, which I've notated here in edosteps:
Enharmonic: {3, 6, 30, 42, 45, 48, 72}
Chromatic: {4, 8, 30, 42, 46, 50, 72}
Hemiolic: {4, 9, 30, 42, 46, 51, 72}
Tonic: {6, 12, 30, 42, 48, 54, 72}
Diatonic: {6, 15, 30, 42, 48, 57, 72}
Syntonic: {6, 18, 30, 42, 48, 60, 72}
The seven rotations are respectively called Mixolydian, Lydian, Phrygian, Dorian, Hypolydian, Hypophrygian, and Locrian, but Cleonides counts them from top to bottom.
Vicentino
Vicentino's list of the modes is perhaps one of the most liberal.
He describes the modes as belonging to three categories: diatonic, chromatic, and enharmonic. These categories describe the number of pitches per octave, as well as the size of the intervals. Each category contains four scaleforms, and two rotations are considered for each scale; that's a grand total of 12 scales, or 24 recognized modes.
Diatonic Modes
The diatonic category of modes was the simplest. Each diatonic mode has seven notes per octave.
The first three modes (Dorian, Phrygian, and Lydian) are defined as the same modes that carry those names today. Three more modes (Hypodorian, Hypophrygian, and Hypolydian) could be found by starting on the fifth degree of those respective scales.
Vicentino adds Mixolydian to account for the seventh rotation of diatonic, and Hypomixolydian to reach eight total modes, even though Vicentino himself notes the superfluidity of this mode; it's literally the exact same step pattern as Dorian.
Chromatic Modes
The chromatic category of modes has eight notes per octave, with permitted step sizes of 8\31, 3\31, and 2\31.
Eight chromatic modes are described, organized into four pairs, each pair consisting of a mode and its rotation.
Enharmonic Modes
The enharmonic category of modes was the strangest and most complex.
Each mode in this category has around twelve notes per octave, give or take, with little restriction on available step sizes; steps of 1\31 are the most frequent.
Eight enharmonic modes are described, organized as four base forms and their rotations.
For each of the enharmonic patterns, L = 10 edosteps, m = 2, and s = 1.
Franchinus Gafurius
Franchinus Gafurius considers eight modes, which descend from complex alterations against three "chief modes," Dorian, Phrygian, and Lydian.
These modes were formed using patterns of tetrachords and pentachords.
Each of these modes is constructed with the pentachord on the bottom and tetrachord on top; Gafurius allows these to be switched in order to reach the "hypo-" forms, which he considers unique modes.
Hypomixolydian is once again equivalent to Dorian.
The usage of the Dorian tetrachord with both Protian and Tetrardian pentachords implies that the association between a given tetrachord and its corresponding pentachord is not immutable.
These combinations would have made for a grand total of twenty-four distinct mode possibilities.
