There were some attempts to explore this territory already. You can find some ideas and thoughts here: House Rules for Insanity
I will later on reply in a more detailed fashion, but I’m also working on a Horror-Fantasy campaign, so at some point I’ll have to come up with system and I have some ideas, but I need to formulate them concisely and clearly to spot weaknesses and holes.
But you can hit me up on Discord if you want and we can exchange ideas at some point.