Democratic Rep. Barney Frank of Massachusetts announced his retirement Monday effective at the end of next year, closing out a congressional career of more than three decades capped by passage of legislation imposing new regulations on Wall Street.