A Georgia man is responsible for a frivolous filing tax penalty after questioning his employment status, the U.S. Tax Court affirmed Thursday.