Ben79 wrote:The transformer says it puts out 11.8v
Please forgive my insistence on the pedantic.
Does this mean there is printing on the transformer indicating a nominal design voltage across the secondary (if so: ignore that) or does this mean you have measured 11.8 volts AC across the secondary while the primary is connected to mains?
Typically when you damage a transformer windings will either open (rendering the object a paper weight) or short to adjacent turn on the same winding (essentially changing your turns ratio). I suppose it is possible that you could have shorted some of the primary winding and now you have essentially increased the turns ratio of the secondary - but I sort of doubt this. Burning windings is a smelly business, and requires either a fair bit of heat or exposure to moisture.
Rectified DC is going to be above measured RMS by a noticeable percentage (is it 30%? I forget off hand and am too lazy to look it up). Anyway, it does not sound like you're out of reasonable bounds on what you're measuring, unless that 17 is on the ass end of a 12 volt regulator.
If your regulator is running hot you can series string more diodes at your rectifier (in, before, or past), losing a half volt for each unit used, but this is a total bodge and can look ugly (not recommended for anything beyond personal use/experimentation). I would just rate the parts in the circuit up to meet a higher power supply rail (if possible) and enjoy life.
...unless you are correct in the concern that your PT has been stressed, as those may continue to degrade if compromised and you should have a backup handy.