A diet high in cholesterol, smoking, chronic diseases and stress-related conditions can affect one’s ability to “perform” in bed. 

Also, some men can only get off one time during a sexual session, while others may be able to go several times, this is both totally normal.  

The most important thing for your sex life is not the quantity or number of times but the quality and duration. You can sit down with your partner and talk about which of these things could be affecting him.  If you are still worried, see a doctor.