He was there to highlight the "Take our Jobs" challenge from the migrant farm workers. Conservatives scoffed at the idea that these were jobs that no American wants to do. So the migrant workers called their bluff, offering their jobs to any American who wanted them. Only Sixteen Americans applied for these jobs. Colbert was one of them.
Colbert's appearance to Congress made a valid point. That is why he was invited.
The people who objected were people who are opposed to immigration reform.