Record Sets

The filter settings available within the Record sets, thus far, offers everything I need to create a practical record set for specific admin users. However, I have come to discover that upon assigning a saved/custom record set to specific users, said users are able to remove/unassign the custom record set from their own user profile, thus allowing them to gain full view/access to all records. I also understand that the Roles section is where user restrictions can be applied to specific users, however, the roles section doesn’t offer the same level of customization/flexibility that is available in the Record Sets. Is there a way of preventing an admin user from removing their assigned Record Set? Ideally, I would like to use the Record set as a control measure, and not just as a filter. Alternatively, if the Role settings could mirror or include some of the options available from the Record Sets, that would be idea.