All A/D-converters in audio interfaces use crystal clocks to determine the sample rates. But: due to tolerances, these are not always in sync. 44,1k is not always EXACTLY 44,1k. Therefore, it's possible that one of your friend's DAW recorded e.g. 44007 samples per second and someone else's DAW 44201. Your DAW uses the same sample rate for playback of all files - therefore all tracks are slightly different in length. That's why there's drift.
And that's the reason why "master clock generators" exist to sync different digital devices to the exact same sample rate...