Surge is brought to you by OmniTI, the leading Web Scalability and Performance provider.
Sponsored by Joyent
"It's a Unix System! I know this!" Using Manta to Scale Event-based Data Collection and Analysis
Data aggregation and analysis problems become notoriously thorny as traffic scales up: conventional databases break down at scale, and map/reduce frameworks such as Hadoop have a substantial developer and operational complexity burden. Wanelo, an online community for all the world's shopping bringing together stores, products and 10M users all in one social platform, became frustrated that the aggregation and analysis tools used when data was small (venerable Unix data processing utilities like grep, awk, cut, sed, uniq and sort) couldn't be used when data became large. Upon discovering Manta, a new cloud-based object storage system that enables the storing and processing of data simultaneously, Wanelo had a solution that no longer required the need to move data between storage and compute. Building on Manta, Wanelo has developed a system for data analysis that allows the team to tackle big data analysis using Unix utilities, resulting in a cost-effective and scalable solution. This talk will discuss Wanelo's experiences building their system on Manta, including their motivations and considered alternatives that led to a Manta-based implementation of fully-parallelized cohort retention analysis in four lines of shell.
Konstantin Gredeskoul is the CTO of Wanelo (“wah-nee-lo,” from Want, Need, Love), an online community for all of the world’s shopping. Wanelo is democratizing and transforming the world’s commerce by reorganizing shopping around people. It brings together all stores, products and people into a single social platform. Prior to joining Wanelo, Konstantin was the Principal Engineer at ModCloth. He oversaw the technology and architecture of the company, and managed a team of more than 20 developers globally. Konstantin previously also served as CTO at Infectious.com and Drop In Media. In addition, he led the web development effort at Blurb.com, where he helped design Blurb’s e-commerce infrastructure in Ruby on Rails from scratch. On weekends, Konstantin can be found DJ-ing around the Bay Area, producing original music (www.cdbaby.com/Artist/PolyGroovers), hiking, biking or playing table tennis. Follow Konstantin on Wanelo, Twitter, or his blog tektastic.