The fame system is set in a way that if you are inactive for a longer period of time, you will slowly lose some of your fame as your fans will slowly start forgetting about you. Out of sight, out of mind, as they say. However, when you go back to the game, you get a hidden buff that will give double the amount of fame you get in missions until you reach your previous level of fame.
