I discussed this very topic with several people whilst @ LUGRadioLive 2007 earlier this year - I'm glad to say that the testing is complete and you can save time when securing a mixed LINUX and Microsoft environment by reading the details for yourself.
Q)WHY use IPSec?
A) IPsec provides the means to isolate your network from machines outside your control - you can force machines to automatically identify themselves and block access to systems outside your management scope. You can optionally encrypt traffic between the systems too though you need to consider the trade off between inspection and confidentiality. IPsec network security is natively supported by most modern operating systems and if correctly configured takes place "behind the scenes" meaning that no user interaction is required.
The team at Port25 (Microsoft's Open Source Software Lab) have released the results of their interoperability testing for IPsec between a range of LINUX systems and Windows Vista - if you have a mixed environment it's well worth a read.
There's a good description of how IPsec can be used to improve network security on DarkReading though their link to the afore mentioned interop guide was broken when I checked it - use the one listed above.
Thanks to Roger Halbheer for his post which inspired this one.
Note: Thanks to Alun for correcting my capitalisation of IPsec!