This is a good cautionary story of why random parts-swapping can be a waste of time and money. Getting out the DMM and measuring voltages is something fewer and fewer people know how to do when troubleshooting electronics, but it certainly saved the OP here; I'd go a little further and figure out why the monitor seems to be leaking power into its HDMI input when switched off --- possibly an ESD-damaged MOSFET or similar?
The issue does not occur when the monitor is connected via DisplayPort.
Not all DMM have probe small enough to connect to the lane. If it's even possible.
What's more, you need to know where to put it, which can be daunting without the proper knowledge. Switching hardware is easier, faster and often the best solution in those case.
Finding hardware fault is hard. Tracing it is even harder.
On the other hand, I recently fried a motherboard while trying to probe it with a multimeter. My fat fingers shorted out two adjacent pins, causing a loud spark and magic smoke.
The issue does not occur when the monitor is connected via DisplayPort.
https://en.wikipedia.org/wiki/DisplayPort#DP_PWR_(pin_20)
Standard DisplayPort cable connections do not use the DP_PWR pin.
There's also an interesting paragraph there, about some nonstandard cables connecting that pin through.