Time drift when W32Time service enableed

We're experience significant time drift on a collection of HP DL585s running
windows 2003 Enterprise Server. Over six machines, all appear to be drifting
by approximately four seconds every half an hour (in different directions).

The machines are set to sync with a local time server every week (standard
W32Time service), and typically their time corrects by a jump of 5 mins or so
back to the correct time, and then beings to wander off again.

However, on disabling the W32Time service, the time drift stops, and they
keep time correctly. I've seem some posts stating this same issue, but none
with hard evidence as to why it occurs.

Has anybody seen such behaviour before? We're now running XNTP on some of
the machines and they are keeping time/synchronising fine.

Thanks,

Simon