@dummzeuch Can't test it right now, as it need more code to make it work, but
Don't ever use select in that way, in general all looks fine and correct, but without socket errors not the ones with API calls return, but the ones from "exceptfds", without checking for socket level errors you are driving in the dark, if error was signaled on the socket then read will hang for ever !
Add exceptfds, and then check for errors before read or write, as in many cases both error and read/write might be triggered and reported at the same time, in your case most likely this what did happen, an acknowledgment received to close and both readfds and exceptfds had been signaled.