Is there an RSS feed like https://serverfault.com/feeds/user/2561 but for answers only, for a specified user?
There is now. I have created an application called Stack2RSS that converts API requests into RSS feeds.
A feed that fetches all of a user's answers looks something like this:
Example that fetches all of the answers posted by user #1 (Jeff Atwood) on SuperUser:
Parse their answer-source:
That will show you their most recent 10. Change pagesize to increase count.