In the context of UX design, a mental model is what the user believes about how a user experience works. Mental models are built in a user's brain and are based on what they know from past interactions with websites, mobile phones, and other interactive products.
Here is a link to a presentation (partial) that I used to explain mental models to junior designers, uxers and product managers:
