my pup is pissing

My puppy who's a 4 month old male Vizsla is showing great signs of his intelligence and his natural hunting instinct. We were able to train him to go outside to go to the bathroom real quick, it took about a couple of weeks. But at the end of the day he's still a puppy and he makes mistakes. But recently in the last week he's gone awry it seems. When he grabs things to chew on, such as papers, magazines, socks and what not we(my wife and I) will go to take said items away from him and once we try and take it away from him he starts pissing. Not a lot but he still does. I'm assuming that he's just nervous or excited and can't control it. Sometimes he'll do it when somebody comes in the house. I think that too is due to his excitement. Overall he is a wonderful smart puppy. We've never punished him physically so I ruled that out as a cause. Or maybe I'm expecting too much of such a young dog just because he seemed to catch on real quick about not going potty in the house and he seems to be regressing and not progressing.

Is this something that is normal for a 4 month old. Should I be concerned or am I just wasting energy worrying about this. I'm not really sure. I'm almost 29 years old and the last time I dealt with the raising of a puppy I was about 6 years old so I don't really have any prior experience to look back at.

Anyone have any thoughts or suggestions.