Revving Your Database to Keep Up With Shopping Festivals

Date:   Friday , November 06, 2015

Festivals and seasons have always been a major factor in influencing the volume of traffic seen by retailers. The festivals and seasonal observances that drive these shifts in traffic volume vary regionally, have shifted over time, and have been made more complex and intense by the rise of e-commerce. As The Independent reports however, the Black Friday phenomenon is not limited to the day after Thanksgiving in the United States. Boxing Day in the United Kingdom, Singles Day in China, and the run up to Diwali in India all draw a rush of consumers that has, to quote one account of the origin of the term Black Friday, the potential to put retailers back in the black.

Fifteen years ago during the first Web boom and the initial rise of e-commerce, life was simpler for retailers. E-commerce sales were a small proportion of total sales, and while the technology supporting the e-commerce infrastructure was not complex by contemporary standards, there was little need for sophisticated systems that could handle large transaction volumes. The situation for online retailers has changed dramatically since that time. According to U.S. Census data, the share of total retail sales taken by e-commerce sales more than doubled just since 2006, a trend that is both in response to and that has itself fed, the massive expansion of interest in special shopping holidays across the globe.

The rise of Internet access and mobile adoption has chipped away at many of the barriers consumers formerly faced in taking advantage of the shopping holidays in other countries as well, and in recent years major retailers like Amazon and eBay have offered Black Friday deals to their Indian customers. Indian companies as well are keen to capitalize on the growing international trend of creating new shopping holidays, and as the Economic Times reported in 2012 a partnership of Indian retailers including FlipKart and Snapdeal created The Great Indian Shopping Festival. The festival is akin to the kind of homegrown e-commerce-focused shopping holidays found in other countries, like Cyber Monday in the United States. More and more e-commerce merchants are growing by entering new geographic markets, adding to the cross-border festival phenomenon.

The Challenges Facing E-tailers

These forces are converging to create a unique series of challenges to the technology under - lying e-commerce platforms. Increasing overall volumes of online sales, as well as an increase in the magnitude of the spikes in festival-driven traffic, are requiring retailers to be more agile and adopt more flexible databases that can adapt on a seasonal timescale to changes in transaction volume, while still providing a database system that can weather the unique challenges of e-commerce applications.

ACID - Ensuring the Transaction Completes

E-commerce transactions rely on relational databases that are reliable enough to ensure that customers avoid unpleasant experiences. One job of the relational database is to make sure transactions complete before allowing any changes that would affect that transaction (also to allow the transaction to recover if it\'s interrupted). For example, if an item\'s inventory indicates that there is only one item left, and two customers add it to their cart at the same moment, a relational database will allow only one customer to purchase the product. \"ACID\" is the set of standards that ensure transactions occur reliably, and is absolutely essential for e-commerce. The problem is that there has historically been somewhat of a tradeoff between ACID compliance and performance factors that are increasingly important with the rise of high-volume shopping holidays internationally. Clustrix was founded to solve this very problem with a highly scalable ACID compliant relational database.

Seasonal Scaling

In order to adapt to drastic changes in demand for system resources, the type associated with seasonal high-volume shopping holidays, a company must be able to scale in a straightforward way. The traditional approach to database architecture and providing for its underlying hardware is becoming too unwieldy for many applications.

Consider a typical approach to scaling relational databases, referred to as sharding. As additional hardware capacity is needed to support a growing database, the \'tables\' of the database are broken up and moved to new hardware. The e-commerce applications that run on top of the database must then be manually updated and told where the piece of data has been moved.

This cumbersome process and its counterparts in traditional relational database architecture do not lend themselves to adding capacity in advance of Black Friday or the Great Indian Shopping Festival and scaling capacity back when demand returns to normal after the holiday. The typical solution is therefore to maintain capacity year-round that is capable of handling seasonal spikes; an approach that is fundamentally inefficient and leaves large amounts of capacity unused for the majority of the year.

When selecting a database solution for e-commerce, retailers should keep in mind how these buying patterns are increasingly affecting loads on the database over the course of a year. A particular challenge is marrying the ACID compliance demanded by e-commerce transactions with the speed and scalability that can address seasonal changes in transaction volumes, but it is clear now that the assumption that these features are mutually exclusive no longer rings true. The social changes in buying patterns are putting increasingly untenable strains on traditional approaches to e-commerce databases, requiring the industry to address the problem competently.