Always worth wondering where Madison got his aversion to Factions as an obvious means to organize multiple and disparate ideas into constructive, useful, gentlemanly debates. And the answer might have to do with Gibbon's Decline and Fall. At the end of the work Gibbon makes an attempt to tell his reader why the Roman Empire Fell. Like all honest men he explained that he couldn't really think of a hard and fast reason, rather he understood it as a series of failures. One of those failures, and I'll have to paraphrase because it's alluded to throughout the book, was the capacity of powerful people in their single minded pursuit of their own interests to lose touch with the source of their power, which is a vibrant, cooperating society that daily welcomes the future as a wealth of possibilities.
The result of self interest for Rome was a corruption of a shared idea of the
Empire, and indeed factions within the empire became so besotted by the
possibilities of winning points for their own side they totally forgot that
beyond their borders other societies were way more cohesive, a great deal more
enthusiastic and not so convinced of their own society's invulnerability. An
Eastern Emperor, for example, became so enraged and threatened by the incredible
successes and popularity of his much feted admiral who'd won several important
battles at sea that instead of rewarding the man for enhancing the Empire,
securing it against foreign enemies, the Emperor had the man disposed of. Don't
know about you, but I hear echoes of this kind of blinkered selfishness that
becomes outright destruction.