Mostly, the badges are sorted in order of percentage complete, as far as I can see. But per the title, Epic (with 2/50, or 4%) sorts ahead of Illuminator (with 24/500, or 4.8%). What's up with that?
As per Oded's answer: (emphasis mine)
status-bydesign - These are sorted by the combined completion %ages of both bars (I don't think there is a reasonable secondary sort to this), given that both parts are required in order for the badge to be earned. In that respect the three highlighted badges are all 5% completed (2/20 == 10% - the other bar is 0% - so together 5% complete, 5/100 == 5% + 1/20 == 5% - so altogether 5% complete).
Please see Patrick Hofman's "Select your next badge dialog doesn't sort the tags as expected" post for more details and his next-badge screenshot. Also note that Oded's answer was given on the initial release of new profile where they were collecting feedback and bugs. The current next-badge UI looks little different.
Here a screenshot of my 'Epic' & 'Illuminator' next-badge section:
Notice, my 'Illuminator' badge (1/500 ) is placed after the 'Epic' badge (0/50).
To obtain the bar percentage you will have to right-click on the next-badge label / title. For example,
Here is what my 'Epic' badge shows:
<div class="bar"> <div class="label">Epic - 0/50</div> <span class="badge2"></span> <div class="percent" style="width: 0%;"></div> </div>
Here is what my 'Illuminator' badge shows:
<div class="bar"> <div class="label">Illuminator - 1/500</div> <span class="badge1"></span> <div class="percent" style="width: 0%;"></div> </div>
So, if I apply Oded's formula i.e. 'Epic': 0/50 == 0% + the bar is 0% - so together 0% complete, and in case of 'Illuminator': 1/500 == 0% (only consider integer part on the left of the decimal point) - bar is %0 - so altogether 0% complete. So, both stood at 0% level and the badges get the default alphabetical sort order.
Now, lets apply this formula to Nathan's badges numbers. 'Epic' with 2/50 == 4% - the bar is 4% - so together 4% complete and in case of 'Illuminator': 24/500 == 4% (only consider integer part on the left of the decimal point) - the bar is 4% - so altogether 4% complete. So, both stood at 4% level and the badges get the default alphabetical sort order.