Last.fm is a popular Internet radio site for music. Founded in 2002, it now claims over 30 million active users based in more than 200 countries. In its early stages of development, Last.fm turned to Sun and the Sun Startup Essentials program to help it get up and running. Over the past two or so years, Sun has saved the organization money, reduced its output, and helped Last.fm scale beyond boundaries it never thought feasible, writes Sun's Stewart Townsend, who shares a blog from Last.fm engineering team member Mike Brodbelt on its Xbox LIVE launch.

On November 17, 2009, Last.fm was fully rolled out to all Xbox Live Gold Members, after a 3-month successful trial. The new application may be downloaded from the Xbox 360 Dashboard and is 6.48MB in size. Last.fm on the Xbox 360 is available only to players in the United States, United Kingdom and Germany.

Brodbelt blogs about using SSD's to stream to Xbox Live. He discusses the need, in the beginning, to scale up the IO capacity of Last.fm's MogileFS cluster independently of the storage capacity. (MogileFS is an open source distributed filesystem from Danga Interactive.)

The first SSD-based system tested was an 8 core system with 2 X5570 CPUs and 12 Gb RAM (a Sun Fire X4170). The group placed 2 hard disks for the OS and 4 Intel X25-E SSDs. "The X25-E drives advertise in excess of 35,000 read IOPS, so we were excited to see what it could do, and in testing, we weren’t disappointed," Brodbelt writes. "Each single SSD can support around 7000 concurrent listeners, and the serving capacity of the machine topped out at around 30,000 concurrent connections in its tested configuration..."

With 10GigE cards, the engineering team decided to push the configuration even higher to solve buffering and connection issues. They opted for 64Gb versions, since the group needed to be able to get enough content onto the SSDs to serve a good number of user requests; "otherwise all that IO wasn’t going to do us any good," explains Brodbelt.

The Linux scheduler defaults were tuned and set as follows for each SSD:

echo noop > /sys/block/sda/queue/scheduler

echo 32 > /sys/block/sda/queue/read_ahead_kb

The team then focused on extending its MogileFS and adjusting its systems:

• to understand the concept of “hot” nodes, storage nodes that are treated preferentially when servicing requests for files.

• by implementing a “hot class,” so when a file is put into such a class, MogileFS would replicate it onto the SSD-based nodes.

• by extending the statistics that MogileFS retrieves from iostat to also include the wait time (await) and the service time (svctm) figures, for better information about device performance.

"Once those changes had been made, we were ready to go live," Brodbelt shares. "We used the same hardware as our final test configuration (Sun Fire X4170 with Intel X25-E SSDs), and we are now serving over 50% of our streaming from these machines, which have less than 10% of our total storage capacity."

More Information

Sun for Startups

Last.fm - How do they stream all that data ?

Sun Fire X4170 Server

Intel Xeon Processor 5500 Series

Intel X25-E Extreme SATA Solid-State Drive [...read more...]

		fullsource
Print Email Current Profile Archive Blog Generate newsletter