Specifically Dominicans in his district. Dominicans are not more or less conservative than Black folk but even when I was a young guy in NYC Dominicans were having problems with discrimination by other New Yorkers. Rangel has long used anti-Dominican (among other races) sentiment as a rallying point for much of his power.
And yet….those “evil” Republicans are so racist!