Is there an RSS feed like https://serverfault.com/feeds/user/2561 but for answers only, for a specified user?
There is now. I have created an application called Stack2RSS that converts API requests into RSS feeds.
A feed that fetches all of a user's answers looks something like this:
Example that fetches all of the answers posted by user #1 (Jeff Atwood) on SuperUser:
Parse their answer-source:
That will show you their most recent 10. Change pagesize to increase count.
You can take the regular user feed, and then filter it using one of the rss filtering websites, for example, https://siftrss.com/