Submissions/Swift and the Media Storage System

From Wikimania 2012 • Washington, D.C., USA

This is an accepted submission for Wikimania 2012.

Submission no.

625

Title of the submission
Swift and the Media Storage System
Type of submission (workshop, tutorial, panel, presentation)
Presentation
Author of the submission
Ben Hartshorne
E-mail address
bhartshorne@wikimedia.org
Username
Bhartshorne
Country of origin
USA
Affiliation, if any (organization, company etc.)
WMF
Personal homepage or blog
http://wikitech.wikimedia.org
Abstract (at least 300 words to describe your proposal)

Over the past year we've been preparing to replace the system that manages all the media on Wikimedia Wikis (all the sounds, images, and video clips that are on all the wikis including commons). Since January we've been actively putting the new system into production. We are now using Openstack Swift - a distributed fault tolerant object storage system - to store all the media (and scaled versions of that media) for all of the Wikimedia wikis.

I will talk about what system we used before, its shortcomings, and why we wanted something different. Specifically, I will address some of the effects of the limitations imposed by the previous system and the types of activities that are now possible (or even easy!) given that we've removed that bottleneck. To whet your appetite, we have been holding back on accepting more large donations of media (things like the NARA donation) because of disk space limitations. Getting rid of that specific limitation is a good step towards being able to accept much more traffic.

I will cover the alternatives we considered and why we chose to use Swift. I'll describe the hardware and configuration of the Swift storage system, as well as the process we followed to get it live in production. Finally, I'll cover our plans for growth and where we hope to go in the future.

Track
Technology and Infrastructure
Length of presentation/talk
25 Minutes
Will you attend Wikimania if your submission is not accepted?
yes
Slides or further information (optional)
Special request as to time of presentations


Interested attendees

If you are interested in attending this session, please sign with your username below. This will help reviewers to decide which sessions are of high interest. Sign with four tildes. (~~~~).

  1. DJBauch (talk) 18:25, 15 March 2012 (UTC)[reply]
  2. Daniel Mietchen - WiR/OS (talk) 23:26, 18 March 2012 (UTC)[reply]
  3. Edsu (talk) 01:24, 19 March 2012 (UTC)[reply]
  4. Tbayer (WMF) (talk) 08:47, 19 March 2012 (UTC)[reply]
  5. Emw (talk) 20:17, 16 June 2012 (UTC)[reply]
  6. iwilsonjr (talk) 16:34, 10 July 2012 (UTC)[reply]
  7. Add your username here