Definitely save everything you can, and also document, separately, a timeline of events (even though emails are usually timestamped). Start doing this now, just in case. Keep track of every interaction you have with the prof (about anything) and every interaction you have with anyone else (student societies etc.) related to this issue.
How does this make you feel? Are you still comfortable even talking/interacting with this professor anymore? At most schools, there is a set protocol/procedure for cases like this. This could be a case of sexual harassment. If you choose to "let it slide" and try to get a reference letter from him still, if the harassment gets worse later and you want to do something about it, then it could be harder to make your case because you left it for so long. And people may question, fairly or not, whether you are just doing this for revenge because of a bad reference letter (whether that happens or not).
If you want to pursue the usual protocol, then in most schools, the usual protocol for issues between prof and student is for the two to discuss the issue informally and try to work it out (either party can bring in an ombudsman if necessary). But in cases of harassment (sexual or otherwise), if the student does not feel comfortable talking about it with the professor, the student can go directly to "one level up", usually the department head (if the prof is the head, then to the Dean). Check your school's protocols though. Usually your (grad- or undergrad-) student society has an officer responsible for this kind of thing -- they can help you through the steps and/or be a third party to any meetings between you and the prof, or you and the department head etc. If you are a TA or RA and count as an employee, and the prof is an employment related supervisor, then there may be a separate group/rules to follow (check with your University's HR department and/or labour union if you are unionized).
If you don't feel like you need to do the above, and if you feel comfortable enough pursuing a reference letter from this prof and then cutting off contact, you should still document every single interaction with the prof from now on, just in case the situation changes later on. But also remember that it's unlikely you will only need to ask him for reference letters once -- depending on his role (thesis supervisor? 3rd LOR?), you might need more letter from him, especially if you go to a new school and don't know anyone yet (if you are talking about a grad school application LOR in your original post)
Without the full details (which I'm NOT asking you to share online), I don't really think anyone can advise you on what to do in your specific situation. But hopefully there was some useful thoughts to consider here that will help you decide what you'd want to do!