Why Twitter’s struggling
Twitter has scaling issues because it was built as a content management system for microblogging, but it became a killer app for distributed messaging. Dare Obasanjo explains how Twitter has the wrong architecture for the application it’s become.
Not only does Isreal’s post [Twitter-proxy: Any Interest?] accurately describes the problem with the logical model for Twitter’s “followers” feature, it looks like he may have also nailed the details of their implementation which would explain the significant issues they’ve had scaling the site. The problem is that if you naively implement a design that simply reflects the problem statement then you will be in disk I/O hell. It won’t matter if you are using Ruby on Rails, Cobol on Cogs, C++ or hand coded assembly, the read & write load will kill you.
Obasanjo goes on to say that
they not only have to deal with supporting users with thousands of followers but also users that follow thousands of users [both of which would be optimized in completely different ways]. Clearly a lot of feature decisions have been made on that product without thought to how they impact the scalability of the service.
