In Recent feature changes to Stack Exchange is shown the list of the recent feature changes to Stack Exchange.
Is there a list with the features SE sites started with in Private Beta? I mean, before adding all these features.
The list of recent changes is valid not only for Stack Overflow, but also for other Stack Exchange sites, including the ones in beta phase.
There can be some differences between Stack Overflow, and the other SE sites, but (as far as I can see) those differences are not reported in the page you linked. For example, on Stack Overflow the number of closing votes you have (if you have the privilege of voting to close questions) are 50, while in other SE sites you have 24 votes to close questions.
There is not a list of the original features Stack Overflow implemented.
The original stackoverflow site:
I have probably forgotten some, but as you can see, there have been a lot of changes from the big bang.