Ok, here is the analysis of the old R_SetLightProject code before @anonreclaimer 's commit broke it in rev 6684.
Some notes:
The inputs are the same as in the previous analysis: R, U, T are unit vectors from spawnargs, and W, H, D (aka width, height, and depth) are the lengths of spawnargs.
It looks like "z" is the perspective divisor here, but it is applied only to "x" and "y". The falloff coordinate "w" should not undergo perspective division.
You can see decomposition of matrix into three transforms, which produce homogeneous coordinates for "x" and "y" in range [0..1] both. Computation of the falloff coordinate "w" is totally independent of this transformation.
Falloff parameter at a point is determined by its projection to "start"-"end" line segment: it is 0 at start, 1 at end, 0.5 in the middle, etc. In fact, falloff parameter changes linearly from 0 to 1 while going from S to E. Note that it does not depend on the frustum at all: you can e.g. have frustum direction and falloff direction different.
Frustum half-angles are same as in BFG case: length(right) / length(target) and length(up) / length(target).
If R/U/T triple is not orthogonal, then things go equally uncertain as in BFG case. Except that here T is made orthogonal to R/U plane, and becomes N. At the same time, size D is adjusted too: we take length of "target"'s projection along N instead of full length of "target". This perhaps gives some sense to the whole stuff when R and U are orthogonal but T is not.
According to p.6, this code supports T not being orthogonal to R/U plane in some sense. In fact, only the component of "target" orthogonal to R/U plane matters, the component along R/U plane is dropped away. Note that you cannot create a non-orthogonal light frustum, so this "support" does not offer any additional flexibility.
The case of non-orthogonal R and U still looks like unsupported/unintended.
Here is how this light can be set up:
Specify X/Y coordinates on frustum "screen" by choosing orthogonal "right" and "up" vectors.
Specify direction of light frustum by setting "target" vector. Only the orientation (which side of the plane "target" looks into) matters. If you set non-orthogonal "target", then it is automatically replaced by its projection onto X/Y plane normal in the engine.
Set lengths of "right" and "up" in such way that (target +/- up +/- right) vectors look through frustum corners. Here "target" is the orthogonal vector already "fixed" by the engine if you set it non-orthogonal.
Choose "start" and "end" points completely independent of all the previous parameters. Falloff parameter ranges from 0.0 to 1.0 as you move point from "start" to "end". Gradient of "falloff" parameter goes along "end" - "start" everywhere.
DeriveLightData_Spot_D3.pdf