Submissions/Swift and the Media Storage System
|
This is an accepted submission for Wikimania 2012. |
- Submission no.
625
- Title of the submission
- Swift and the Media Storage System
- Type of submission (workshop, tutorial, panel, presentation)
- Presentation
- Author of the submission
- Ben Hartshorne
- E-mail address
- bhartshorne@wikimedia.org
- Username
- Bhartshorne
- Country of origin
- USA
- Affiliation, if any (organization, company etc.)
- WMF
- Personal homepage or blog
- http://wikitech.wikimedia.org
- Abstract (at least 300 words to describe your proposal)
Over the past year we've been preparing to replace the system that manages all the media on Wikimedia Wikis (all the sounds, images, and video clips that are on all the wikis including commons). Since January we've been actively putting the new system into production. We are now using Openstack Swift - a distributed fault tolerant object storage system - to store all the media (and scaled versions of that media) for all of the Wikimedia wikis.
I will talk about what system we used before, its shortcomings, and why we wanted something different. Specifically, I will address some of the effects of the limitations imposed by the previous system and the types of activities that are now possible (or even easy!) given that we've removed that bottleneck. To whet your appetite, we have been holding back on accepting more large donations of media (things like the NARA donation) because of disk space limitations. Getting rid of that specific limitation is a good step towards being able to accept much more traffic.
I will cover the alternatives we considered and why we chose to use Swift. I'll describe the hardware and configuration of the Swift storage system, as well as the process we followed to get it live in production. Finally, I'll cover our plans for growth and where we hope to go in the future.
- Track
- Technology and Infrastructure
- Length of presentation/talk
- 25 Minutes
- Will you attend Wikimania if your submission is not accepted?
- yes
- Slides or further information (optional)
- Special request as to time of presentations
Interested attendees
If you are interested in attending this session, please sign with your username below. This will help reviewers to decide which sessions are of high interest. Sign with four tildes. (~~~~).