Part of the problem is that Gravatar is sending a bad Last-Modified header. It seems to be when you signed up for Gravatar, not when you last changed your gravatar image. This makes Chrome, in particular, think that the image has not changed, even on a Ctrl+F5 refresh. Even though Gravatar sends a max-age of 300 seconds (5 minutes), the browser sees that Last-Modified hasn't changed and figures it doesn't need to re-download the file. The only way I've gotten Chrome to update Gravatars is by clearing my cache.
This could be resolved if Gravatar would send a proper Last-Modified header. Tell them about it here. (Note: that page is technically for reporting abuse, but it is the only way I could find to send a message to an actual person. If there is a page to report problems please let me know.)
Update: Looks like Gravatar has fixed this problem and sends a proper Last-Modified header.
The issue would also be partly resolved in Chrome if Ctrl+F5 refresh actually ignored cache like it is supposed to. (I opened a Chrome bug report about this, which was closed as a dup of this bug report.)
Update: Looks like Chrome has fixed this bug now! Not sure what release will first include the fix though.