The actor appeared on Tuesday's episode of "The Late Show with Stephen Colbert" and talked about the fallout surrounding the character he voices, Apu Nahasapeemapetilon.Comedian Hari Kondabolu's documentary "The Problem with Apu" debuted last November and looked at the character as a negative, stereotypical representation of South Asians.Nahasapeemapetilon, a Indian-American character with a thick accent, operates the Kwik-E-Mart convenience store in the fictional town of Springfield. The show recently aired a response to complaints about him.Related: 'The Simpsons' addresses Apu stereotype criticismWriters had the character of Lisa Simpson respond to a question from her mother Marge by saying, "Something that started decades ago and was applauded and inoffensive is now politically incorrect. What can you do?" before Lisa glanced at a framed photograph of Apu on her nightstand, which was inscribed with the message "Don't have a cow."The scene generated further complaints from viewers.Azaria voices multiple characters on the show, including Moe Szyslak and Chief Wiggum. He told Colbert he just wanted to be funny."The idea that anybody, young or old, past or present was bullied or teased based on the character of Apu, it just really makes me sad," Azaria said. "It was certainly not my intention. I wanted to spread laughter and joy with this character and the idea that it's brought pain and suffering in any way, that it was used to marginalize people, it's upsetting."The actor told Colbert he had nothing to do with the show's response, given that Apu doesn't speak during the episode. Azaria said he did not agree with the show's response to the controversy.Azaria said he's given a great deal of thought to how to proceed and believes "the most important thing is to listen to South Asian people, Indian people in this country when they talk about what they feel and how they think about this character."He also called for more inclusion in "The Simpson's" writing room — even if that means he no longer voices Apu."I really want to see Indian, South Asian writer, writers in the room, not in a token way but genuinely informing whatever new direction this character may take, including how it is voiced or not voiced," Azaria said. "I'm perfectly willing and happy to step aside or help transition it into something new. I really hope that's what 'The Simpsons' does and it not only makes sense, but it just feels like the right thing to do to me."CNN has reached out to "The Simpsons" for comment.