Only two days ago the contact messaging application Twitter suffered another bout of downtime, leaving some users frustrated and others asking why the platform continues to suffer problems.

I have recently spoken to an individual who is familiar with the technical problems at Twitter as well as the challenges that lay ahead for the startup. He re-iterated his belief that the problems lay not with Blaine Cook (the former head of engineering who was shown the door), nor with Joyent NTT (their host) but with the early lack of understanding of how complex their problems would be.

The issue is that group messaging is very difficult to achieve at a grand scale. Other large sites such as Wordpress and Digg are mostly dealing with known problems, such as how to serve a large number of pages or a large number of images. Twitter is unique in that it needs to parse a large number of messages and deliver them to multiple recipients, with each user having unique connections to other users.

Social networks have similar complexity issues, but they only usually need to route a message to a single user (or at the most to a defined group). Even so, social networks like Friendster struggled for years with technical and scaling issues. Twitter is specifically dealing with text messages, and in most cases with active users those messages are very frequent and go out to hundreds of contacts (or followers, as they are referred to in Twitter). Every new Twitter user and every new connection results in an exponentially greater computational requirement.

Some of the best web applications are able to efficiently solve very complex problems to produce simple results for users (Eg. Google). The success of these applications is due to the innovative efforts by developers to solve large technical challenges, where they have often had to break new ground for solutions. For Twitter to reach a similar point of reliability they too will need a very comprehensive, ground-breaking solution.

The source that I spoke to also commented on how ill-prepared the Twitter team were and are for their current and future challenges. The small team contains a handful of engineers, with only a person or two committed to infrastructure and architecture. He goes on to point out that at Digg the team for network and systems alone is bigger than the total engineering team at Twitter, and that at Digg they are lead by well-known “A-list rockstars�.

The problems at Twitter are often attributed to their use of RubyOnRails, a web development framework. Twitter is almost certainly the largest site running on Rails, so fans of the framework and its developers have been quick to deflect the criticism and point it back at the engineers at Twitter. Utilizing a framework that has never conquered large-scale territory must certainly add to the risk and work required to find a solution. As an out-of-the box framework, Rails certainly doesn’t lend itself to large-scale application development, but was a big part of the reason why Twitter could experiment and release early.

Rails has enabled Twitter to prototype quickly, to quickly launch and then to easily iterate with new features. But the old adage of “Good, Fast, Cheap - pick two� certainly applies; and Rails would do itself no harm by conceding that it isn’t a platform that can compete with Java or C when it comes to intensive tasks. Twitter is at a cross-roads as an application and Rails has served its purpose very well to date, but you are unlikely to see a computational cluster built with Ruby at Apache any time soon.

What we see at Twitter today is a very useful and popular service, but one with very complex underlying technical challenges to overcome. Twitter will require not only a new architecture approach and a big injection of the best minds they can find ($15 million can help), but will also need a little patience from users and those of us observing.

Activity stream aggregator FriendFeed launched a new feature called FriendFeed Rooms this afternoon, which are effectively topic-based accounts that anyone can create or join (depending on privacy settings). Users can then add links and messages to relevant content.

The main difference between Rooms and a normal FriendFeed account is the fact that multiple users can author it, and that you can’t pull third party feeds into the service.

FriendFeed usage continues to grow steadily, and has clearly gained from Twitter’s (a competitor of sorts) constant downtime. I still haven’t gone religious on it, though, as some have. That’s mostly because i don’t like having a third party service centralize all this data about me and then not let that data back out again. See my rant on the Centralized Me for more on that.

According to FriendFeed a room is like a mini FriendFeed for a particular subject or group of people. You can make a room for your family, your work team, or your knitting club. If you’d like to see what a room looks like, check out the FriendFeed News room, a public room where people share and discuss FriendFeed in the press. Everyone in your room can share stuff with each other and leave comments that only other people in your room can see. You decide whether to make your room public, where anyone can join, or private, where you have to invite or approve each member. You can even choose to view everything from your rooms in your feed, instead of just in the rooms themselves.

There’s no shortage of stories about Google’s newly launched App Engine, but almost all of them get it wrong — because they compare the new service to Amazon’s EC2 and S3 services (AMZN).

If the Silicon Valley echo chamber wants to make up a competitor for AppEngine, its proper correlate (by a whisker) is Facebook’s F8 platform. If you must cram this new service into a pigeon hole, think of App Engine as the Facebook Platform for the grown-up web.

Why isn’t App Engine like EC2 & S3? Constraints, constraints, constraints:

* Google is going the run-time environment route, not the scalable, “put anything you want in a box and we’ll scale it” route that EC2 provides. Case in point: We could run the BricaBox Platform on EC2 by tailoring our own environment (the LAMP stack) and booting it up on Amazon’s servers. We could not get BricaBox running on AppEngine without re-writing in Python, ditching functionality which needed outside libraries or languages, or relational databases.

*Google is not trying to provide pure utility here — they are trying to provide utility tethered to their infrastructure. While EC2’s initial investor pitch was “we have this scale and want to monetize unused portions of it,” many smart people called them out for what they were really doing: creating a new business based on the concept of utility computing, which had nothing at all to do with their core infrastructure at Amazon.com (i.e. they weren’t using EC2 or S3 for their needs).

*Google is clearly looking to have as many of these apps tie into existing pieces of Google’s infrastructure: everything from the authentication systems (Google Accounts only!) to code libraries (most open source or API accessible code from Google is written in Python and ready to run in their environment) is based around this infrastructure and will be based around this going forward.

So, why is App Engine more like the Facebook Platform 3.0?

* App Engine is designed for lightweight apps.

* App Engine apps are wrapped around Google’s existing userbase and communication infrastructure (Gmail), creating a more organic Social Ecosystem for Google.

* App Engine is way more interoperable with the rest of the web than Facebook apps, though still built around a proprietary stack.

* The first App Engine apps will be huge! The next ones will have just as much trouble emerging from the murky waters of the web as any other apps.

After a few years of trying to fill Bill Gates’ shoes as Microsoft’s chief software architect, Ray Ozzie is starting to hit his stride. In a remarkable strategy memo to employees (embedded below), Ozzie essentially shifts Microsoft’s mission from one of creating software for the PC and stand-alone servers to creating an interconnecting mesh between devices and people. He is not abandoning Windows or Office, but he is saying that the value of Microsoft’s software will increasingly depend less on what it can do on its own than what it can do with others. It is not about software anymore so much as it is about Web-based services. Ray, welcome to the club.

Excerpt:

Central to this strategy is our embrace of both a world of the web and a world of devices. Over the past ten years, the PC era has given way to an era in which the web is at the center of our experiences – experiences delivered not just through the browser but also through many different devices including PCs, phones, media players, game consoles, set-top boxes and televisions, cars, and more.

Guiding Principles

There are three overarching principles guiding our services strategy – principles informing the design and development of products being implemented across all parts of Microsoft, for both individuals and business.

    1. The Web is the Hub of our social mesh and our device mesh.

    The web is first and foremost a mesh of people. . . . All applications will grow to recognize and utilize the inherent group-forming aspects of their connection to the web, in ways that will become fundamental to our experiences. In scenarios ranging from productivity to media and entertainment, social mesh notions of linking, sharing, ranking and tagging will become as familiar as File, Edit and View. . . . To individuals, the concept of “My Computer” will give way to the concept of a personal mesh of devices – a means by which all of your devices are brought together, managed through the web, as a seamless whole.

    2. The Power of “Choice” as business moves to embrace the cloud.

    Most major enterprises are in the early stages of a significant infrastructural transition – from the use of dedicated and sometimes very expensive application servers, to the use of virtualization and commodity hardware to consolidate those enterprise applications on computing and storage grids constructed within their data center. . . . Driven in large part by the high-scale requirements of consumer services, the value of this utility computing model is most clearly evident in cloud-based internet services.

    Software built explicitly to provide a significant level of server/service symmetry will afford choice and flexibility in developing, operating, migrating and managing such systems in highly varied enterprise deployment environments that are distributed and federated between the enterprise data center and the internet cloud.

    3.Small Pieces Loosely Joined for developers, within the cloud and across a world of devices.

    Application design patterns at both the front- and back-end are transitioning toward being compositions and in some cases loose federations of cooperating systems, where standards and interoperability are essential. . . . At a higher level, myriad options exist for delivering applications to the user: The web browser, unique in its ubiquity; the PC, unique in how it brings together interactivity/experience, mobility and storage; the phone, unique in its extreme mobility. Developers will need to build applications that can be delivered seamlessly across a loosely coupled device mesh by utilizing a common set of tools, languages, runtimes and frameworks – a common toolset that spans from the service in the cloud to enterprise server, and from the PC to the browser to the phone.

Long-Term Evolution (LTE) is one step closer to industry-wide stability. 3GPP LTE technology (LTE is the name given to a project within the Third Generation Partnership Project) offers wireless broadband speeds with downloads around 100 Mbps and upload of 50 Mbps. Seven telecommunication companies have reached an agreement on a framework for licensing intellectual-property rights that relate to LTE. This agreement will make the transition to LTE easier because the fear of lawsuits will be reduced.

Alcatel-Lucent, Ericsson, NEC, NextWave Wireless, Nokia Siemens Networks and Sony Ericsson have agreed to an industry standard being called FRAND, which stands for Fair, Reasonable And Non-Discriminatory licensing terms. Notebook computers that use LTE will pay a combined maximum royalty in the single digits. Handsets will pay a single-digit royalty that is based on a percentage of the sales price of the device.

Ericsson Senior Vice President and CTO Hakan Eriksson said this agreement will “reassure operators of the early widespread adoption of LET technology throughout the consumer electronics industry.”

Industry giant Qualcomm has yet to sign onto the FRAND framework. Other companies like Verizon Wireless, China Mobile, Vodafone and NTT DoCoMo are working on their own versions of LTE.

The future may not be here yet but it could be by next year. Wireless high-speed access will go a long way to promoting services like high-resolution video streaming and innovative online games that can be accessed almost anywhere at any time.