Pain Inverter triggers for each damage "packet" dealt by the foe, and the damage cap of 80 is applied to each packet separately after the damage multiplier.
For example, if a foe deals 200 damage to a single target in one hit, i.e. a single damage packet, Pain Inverter will trigger once and deal the maximum of 80 damage to the foe.
However, if a foe deals 60 damage each to 4 targets, then Pain Inverter (at Asura rank 10) will trigger 4 times, each packet multiplied to 60 × 140% = 84, which is capped at 80, then dealt 4 times for a total of 4 × 80 = 320 damage.
Pain Inverter's damage is based on the original damage deal by the foe, before it is modified by skills like Protective Spirit.