"He's leaving, gracelessly and in infamy," the Oscar-nominated actor tweeted