Absent father wants visitation of 3 year old in another state (halfway across the country)

The father of my child and I had been together for only a few months before I became pregnant. Needless to say, I was too young to be making adult decisions at the time, and I happen to fall into the arms of a convicted fellon. He had previously spent over a year in prison due to heavy drugs, assult, and using a be b b gun out the window of his car one night before we had met. During my pregnancy he had exhibited behavior that was showing his original lifestyle which proceeded into my daughter's life after she was born. About six months of trying to bring him back to ground, I decided enough was enough. I packed up my car overnight along with my daughter and we started our journey from minnesota to north Carolina to be with my mother. Since then I have held a job for over two years, met a wonderful man, and have put us into the makings of a great future. During this time of building for us, he had landed himself in a prison for a second time for drugs again, and also has on his sheet a instance where he had assaulted a younger boy- this time serving a sentence of two years. I have always allowed open contact, and have never induced barriers between noelle and her father. Though now that he is out he is asking for her to have a week or two of visitation in Minnesota with him and his family. Noelle is a very smart, and mature for her age, but I am very confident that she just is not ready for this kind of step. I am genuinely very concerned for how this will affect her if the tables possibly turn in his direction. Last month I had been awarded temporary custody of her, but he does still have the opportunity to combat if he chooses which I think he may. If there is anyone out there with any advice or has any information regarding the pros and cons of the drastic move please help! SOS. Sincerely- a very worried mama!!