I know what you are going through. I ended up leaving the denomination that I grew up in over my ministry. It took several years to finally understand that other people just don't have the same passion as I do. Oh, sure. It started off with a bang, but as soon as it started getting real and the newness wore off, people started dropping like flies. Several people would come for what my wife and I would refer to "field trips" just to see "those people" and then disappear. I don't think people having any interest at all was the worst part, the worst part is when it seemed people who were warming the pews were actively trying to interfere with my ministry. Some were people I had known my whole life.
While your pastor should be behind you, that does not mean that he is the only one you can or should talk to. I cannot tell you how many times I have called Vhayes or RabbiKnife and just simply cried or blew off steam. Ministry is tuff. It is even harder when it feels that you are going it alone. Find someone mature and firm in the Lord and talk to them. DO NOT keep it to yourself and isolate yourself, that is exactly what Satan wants.
If you do not mind me asking, what kind of ministry are you in?
I cannot volunteer Vhayes or RabbiKnife, but I can and will extend you an invitation to talk. If you need anything, including blowing off steam, you know where to find me. You will be in my prayers.
For what mortal has ever heard the voice of the living God speaking out of fire, as we have, and survived? ~ Deuteronomy 5:26
If you're not prepared to risk your very life for your "enemy" you have no right to speak to him of love. ~ Daughter
Many say they are called... but I am pretty convinced that with many of them it was the wrong number. ~ Project Peter