Don’t be pushed by people to alter your training to suit their desires. If you have said you are going for a fast ride, and they just want a social, don’t hang around. You can always meet them at the half way cafe (just you will have worked harder for your treat).
Similarly, if you are on a recovery ride, and they want to up the pace – let them. Put yourself first, and don’t pander to the wishes of your riding friends.
There are no yellow jerseys for the club ride champion.