The visiting area isn't situated in an area in which prisoners come and go without strict supervision.
Your son will not be exposed to prison life or other prisoners by visiting his father.
It seems highly unlikely that by visiting your son's father you will be targeted by the prisoners with which he has fought. Seeing your faces isn't going to make such retalliation automatically possible or inevitable.
If these prisoners want to punish your husband by harming his family they don't need to see your faces to do so.
Unless your husband is the kingpin of a powerful prison gang, their desire for revenge, to the extent it exists, will be expressed directly upon your husband. If this conflict has a scope far beyond what your post suggests, his enemies will have the ability to find you and your son regardless of whether or not you visit.
I don't, for one second, mean to suggest that you should be concerned for your safety and the safety of your son because your husband was involved in a prison brawl. It's seems pretty clear that your husband is not a crime overlord, and if he was he would have the resources and ability to protect you and your son.
My point is that the brawl in which he was involved should not be a deciding factor as to whether or not you allow his son to visit him. It is very unlikely that a visit will put you and your son in jeopardy.
This is not to say you should, necessarily, allow the visit.
You husband is, obviously, a habitual criminal facing a life sentence. It would not be unreasonable for you to decide that nothing positive, for your son, would come from a visit with his father.
Does your son want to visit his father?
If he doesn't, then I would say don't bring him to see your husband. If he truly cared for your son he would not have continued his criminal activites. He is not owed the sight of your son.
If he does, I would say you should.
The only reasons you have offered for not allowing the visit don't seem to carry much weight.
Your son will not be exposed to pedophiles and murderers by visiting his father, and the visit is extremely unlikely to put you and him in danger.
If you have another and better reason(s) not to allow the visit, then don't.