NoSQL key to scalable proposition, says Mike McCarthy of SkillPages

Mike McCarthy, CTO, SkillPages. (Source:SkillPages)

The need for a scalable and fully distributed data store a key driver in NoSQL approach

Print

Print

Pro

Read More: NoSQL SkillPages Spotlight

Tweet

Founded in 2010, SkillPages is now the world’s trusted network of people with skills, creating new opportunities for everyone, everywhere by transforming the way skilled people connect to those who need them.

In total, over 20 million people from 160-plus countries have joined the platform. Indeed, more than 200,000 people are presented with opportunities on the platform daily.

To create this model, we had to use technologies that are largely open and which can extend or expand capabilities as needed. In terms of choosing NoSQL for this mission, being able to quickly and reliably deploy a scalable and fully distributed data store was a key decision point.

This architecture plays well as we deploy the platform across other regional data centres, and giving us the ability of being able to support dynamic schemas across significant datasets, which has proven to be highly advantageous.

We track and analyse in excess of 40GB additional telemetry data on a daily basis. This data excludes user generated content and profile data. In fact, our primary social graph models in excess of 2.5 billion objects which underpin real-time activity on the platform.

The implementation of the functionality was initially based on traditional relational database management system (RDBMS) and later commercial graph databases. However, in 2011 as we started getting into large datasets (ones with one billion-plus rows), we realised that we needed a different approach to support the scale at which the company was growing.

Everything a user experiences on the platform is enhanced by this network of their real world connections so maintaining performance was crucial.

We committed the functionality 100% to NoSQL, although in addition we have since migrated our URI encoding and mapping repository over to the open source distributed database management system, Cassandra which has been a significant addition to our model.

In terms of the benefits to such an approach you’d have to first look at the impact for the customer. From an operational perspective, our ethos has always been to focus on solving our users’ problems and not necessarily get consumed with building and maintaining infrastructure. Our NoSQL implementation has enabled us to achieve this.

In terms of the impact on our IT infrastructure, NoSQL allows us to consistently deliver the scale and performance that is expected in the business.  A key success factor in our implementation has been the world class support and tooling we receive from DataStax, a leading supplier of enterprise grade Cassandra.

OpsCenter, part of DataStax Enterprise, is particularly powerful at detecting and resolving issues before they become problems as well. We are able to support and, more importantly, evolve the social underlay of our platform without adding additional engineering capacity to the project.

This infrastructure also allows us to build increasing amounts of social context into more of the jobs and opportunities we present to our users on a daily basis. NoSQL and Cassandra allow the SkillPages team to un-blur the lines for employers and perspective employees everywhere.

Read More: NoSQL SkillPages Spotlight

Related Articles

Big Data use cases too compelling to ignore, argues HP’s Wood →

‘One stop shop’ means no missing Lync, says Videnda MD Holohan →