Oops, totally misread your question. The only way to remove the automated avatar image without putting a new one there would be to sign up for Gravatar with the same email you use for your SE profile and use a blank image as your Gravatar.
No, there's no way to not use Gravatar, but you have several options:
Change your email address to be of the form
firstname.lastname@example.org. Anything between the
+ symbol and the
@ sign usually gets ignored, so email from SE will still be routed to you, while the Gravatar hash will be changed. (On some services, this will be the
- sign, and on some others, all symbols will be important, so check with your email provider.
Use an alternate email address and setup forwarding to your regular email for SE emails. (Most free online providers like Gmail provide this).
Remove your email from your profile.