Does this code calculate the length with single-precision arithmetic? I mean after subtract() the three relative coordinates dx, dy, dz should be multiplied added together as 32-bit floats using dx*dx+dy*dy+dz*dz. This may yield some precision loss if one of the components very large compared to the others. I don't think this could explain the in-game value being rounded up, though.
the length() function is a simple wrapper doing exactly what you describe.
To with
Code:
length: function () {
return Math.sqrt(this.dot(this));
},
dot: function (v) {
return this.x * v.x + this.y * v.y + this.z * v.z;
},