Update: as of May-June 2016, this appears to be fixed. Clicking reputation tab at network profile appears to load the graph (screen shot). For the sake of completeness, I typically observe a noticeable delay in loading, last time it was about a minute.
This issue seems to be related to my voting habits. I have many (I think few thousands) -1
reputation change events related to downvoting answers. I also have similar amount of rep change events related to rep refunds when such answers are deleted. I am not trying to make it a zero sum game but it seems that more than half of the answers I vote down are eventually deleted, resulting in +1
rep changes (I wouldn't be surprised if it's 80 or maybe even 90%).
Most of these rep change events are at Programmers and it's hardly a coincidence that I experience noticeable delay when loading my profile over there (FWIW these delays don't seem to happen when I load it when logged out). Prior to recent profile changes, I experienced such delays only when loading my reputation tab over there but now whole profile loading slows down, as if the system tries to prefetch my rep changes even though rep tab is not displayed.
At Workplace, my downvoting activity is lower but for various reasons -1
rep changes are less... "amortized" over there, allowing to easily see the trend at my profile activity tab:
Worth noting that these issues appear to be related to the kind of rep change events, but not to their amount.
In more typical cases, system doesn't seem to have difficulties accounting for thousands / tens thousands of reputation change events. As an example, top rep users at Programmers certainly have several thousands rep changes (100K rep means there were about 10K upvotes and accepts) but their reputation graphs load fine and without delays.
Top rep users at Stack Overflow make even more prominent example - they've got 5-10x more rep change events than those at Programmers but still, system loads their reputation graphs pretty fast.
It is as if system generally "knows" how to account for multiple rep change events but this "knowledge" is not being used to optimise events related to downvoting other's answers or rep refunds when such answers are deleted.