Georgetown used to be a slave port. After the Civil War, "a skilled black working class started to emerge alongside a handful of black professionals." Things got worse after that. The C&O Canal flooded; the Depression came; the New Dealers came in and gentrified the town; the "slums" were cleaned up.
Blacks were thus becoming invisible by the time the likes of Democratic doyenne Pamela Harriman started creating Georgetown's all-white "social salons" of such ludicrous legend. Indeed, racism was so entrenched in the nation's capital that even the glamorous young Sen. John F. Kennedy voluntarily signed a deed containing a "restrictive covenant" when he bought his house on N Street NW in 1957, agreeing that the home should not "ever be used or occupied or sold, conveyed, leased, rented, or given to Negroes or any person or persons of the Negro race or blood."
Capt. Penny of "Penny Lane" was a slave trader and vocal anti-abolitionist. This one gets complicated: there's a movement afoot in Liverpool to change the name of the streets that are named for slave traders; but considering the tourist trade, the City Council left that one well enough alone.