There are VERY specific scenarios where you might want to remove a management server from the All Management Servers Resource Pool (AMSRP)

However, there are some challenges you might experience if this is done.  Russ Slaten walks through the challenges and workarounds to make this happen with success.

http://blogs.msdn.com/b/rslaten/archive/2013/05/30/management-servers-turn-gray-when-they-are-removed-from-the-amsrp.aspx