In my "Notes in Linear Algebra" it says:
##### We will use the cross (also called vector) product of two vectors to determine the coefficients A,B,C for the plane of a polygon. If P1, P2, P3 are three points in clockwise order on a Polygon:
P2 * * * * * * P1* *P3
Then consider the vectors u=P2P1 and v=P2P3. Clearly both lie in the plane of the polygon.
Their cross product w = u x v is perpendicular to u and v and is therefore perpendicular to the plane of the polygon - i.e. is parallel to the normal to the polygon. Also u, v, w form a right-hand system so that w must point out of the page, not into it. If the points were in anti-clockwise order then w would point into the page. #####
1. First I should point out that I have usually assumed that we adopted the convention that polygon vertices are given anticlockwise on the visible face. So the section above would be used as described in its last sentence. Better however to start from scratch and rewrite the section for anti-clockwise order as:
##### If P1, P2, P3 are three points in anti-clockwise order on a Polygon. Since they are anticlockwise it follows that the interior of the polygon is where I placed the letter I.
P2 * * * * * I * P3* *P1
Then consider the vectors u=P2P1 and v=P2P3. Clearly both lie in the plane of the polygon.
Their cross product w = v x u (note the order!) is perpendicular to u and v and is therefore perpendicular to the plane of the polygon - i.e. is parallel to the normal to the polygon. Also v, u, w in that order form a right-hand system so that w must point out of the page, not into it. If the points were in clockwise order then w would point into the page. #####
However, the section above is not quite correct, although it may appear correct. While v,u,w are mutually orthogonal, there is no reason for v,u,w to be a right-handed system and therefore no reason for w to have to point OUT OF the page. As mentioned elsewhere in my notes, considerable care is needed in computing normal direction!. The above statement is true only if the angle (measured counterclockwise as angles always are, i.e, the interior angle) from v=P2P3 to u=P2P1 is under 180. If it is over 180 then we get the opposite direction. So the algorithm for a normal needs to be special cased. In a convex polygon all angles are under 180 so w = v x u always points out of the polygon.
Another way to understand this is to remember that the size of the cross product of u and v is ||u||||v||sin(theta) and the sin of the angle theta changes from positive to negative as the angle goes above 180.