Blogs By this
author
Related blogs
to this blog
What is Big Data?
Big data generally refers to large volumes of data. But at what point does data (or information) cross the threshold and become big data?
It depends on what that data is, how it’s being stored, and what it’s being used for. That could happen at 1GB, or it could be hundreds of terabytes.
But usually, we find that data is referred to as big data when there is so much of it that you start running into challenges, like:
Difficulty storing it – Conventional data storage options may not be suitable for your needs
Difficulty processing it – For example, if you have 500TB of data, a modern computer (or even multiple computers) will have difficulty have trouble aggregating the data and running queries that need a vast amount of data to solve a problem or detect an anomaly
Difficulty leveraging it – The sheer volume of data may mean you require extra steps or systems to gain meaningful insights or practical applications from it
Fortunately, the benefits of big data can make it well worth the trouble for businesses.
How is Big Data Useful for Businesses?
Today businesses are collecting more data than ever due to the increase in new technologies, real time information, streaming, higher speed internet, multiple formats, multiple sources, and rich media.
And of course, there are many of ways to leverage that data. Businesses can use big data to achieve things like:
- Enabling performance analytics and real-time insights
- Streamlining resource management and working more efficiently
- Implementing ML for automating decision making
- Spotting problems or outliers that explain why something’s not working
- Making better decisions with more accurate insights
- Improved risk management
- Identifying trends — and reacting faster to change
- Identifying opportunities to grow, increase revenue, and even develop new products.
What Types of Businesses Use Big Data?
At Comunet, we’ve provided big data solutions for clients across a range of industries, including all levels of government, education, agriculture, banking, medical, manufacturing, aerospace and many more. We find that, regardless of industry most medium sized businesses these days deliver something that requires data or is backed by data.
In fact, for many service businesses in particular, knowledge or IP (in other words, data) is their core business, even if they don’t yet realise it or they’re not utilising it very well.
To help illustrate this, here are two examples of big data in practice:
Big Data in Practice: Predicting Labour Requirements in the Agriculture Industry
Our team was involved in helping an Australian fruit and vegetable producer collect and use their data for more efficient picking processes. The business needed to determine how many pickers were required across each of their fields on any given week throughout the year, based on the volume of product that needed to be picked. Requirements would vary depending on the location, weather, sun, water, season, crop, and other variables. This is a good example of a real business problem you can solve with big data — without data the producer was flying blind. We developed a system that was able to use the data to accurately predict the number of pickers required in the coming weeks. This meant the business could confidently plan resources and maximise yield.
Big Data in Practice: Anticipating Equipment Requirements in Mining and Resources
In the mining industry, any equipment downtime leads to massive business expenses. If the material or parts needed to run a key component aren’t readily available onsite, the team may need to down tools until supplies arrive, potentially costing the business millions in lost productivity. In the past, these systems were largely based on guesswork — someone onsite does a visual check of the equipment or raw material and estimates whether new supplies are needed. But increasingly, mining operators are using big data to accurately track and predict requirements based on things like mining volumes, equipment performance, and expected life of equipment — even automatically scheduling equipment maintenance and ordering new parts.
Is Big Data Just for Big Business?
In the past, big data was primarily a big business activity that required unique skills and dedicated practices. It was a nice-to-have function that wasn’t the primary output of the business.
These days, big data is no longer exclusive to big business. Increasingly, smaller businesses are finding value in data and even making it their core output. That’s because tooling is much cheaper, easier, and more accessible than ever, allowing people from a variety of roles to develop their confidence with managing data. Even microbusinesses and freelancers are tapping into data insights via tools like Mailchimp and Google Analytics.
Comunet is a great example of creating value by leveraging data. After working on an aviation SaaS solution for around 10 years, it’s now a product that is used by around 300 aviation companies around the world to capture flight and safety management information, creating a repository of knowledge for the business. And of course, both knowledge and data form the basis of our cloud, IT, and consulting services.
Where to Start with Big Data in Your Business
If you’re interested in exploring big data for your business, it’s important to start with your objectives.
Too often, clients come to businesses like ours with an objective like ‘I want a dashboard’. But this is a shiny object, not an objective. Perhaps that’s why so many dashboards simply sit there, unused, without delivering any real value.
Before you get too excited about big data (and the technology that enables it) you need to dig a little deeper and get clear on your purpose. Ask yourself:
- What problems are we trying to solve?
- What do we hope the data can inform us on?
- What outcomes do we want to achieve?
- What does success look like?
Only then can you determine what data you need to collect, how you’ll collect it, how you'll store it, how you’ll report on it, what platform you might use, and how you'll use the data in your business. We talk more about this in our article on the value of data.
This process might feel like a step backwards if you already have a system in place that collects data, stores it, and produces reports. That’s why a lot of people simply go back to the data they already have. But it’s important to set your existing systems aside (at least temporarily) and be open to transforming your practices if they don’t support your objectives or add real value.
Best Practices for Implementing a Big Data Strategy
Once you know your objectives, you can audit your data to see what’s missing. Sometimes you’ll be able to work with your current data sets, but sometimes you’ll need to collect more information to fill in the gaps.
Likewise, you may be able to use your existing storage systems and software. But in my experience, it’s often better to source a simple-but-effective solution that will quickly meet your needs — without any unnecessary bells and whistles. I’m a big advocate for cloud technologies like AWS, because you can:
- Pick and choose the tooling you need to solve the problem
- Only pay for what you need — free and tier-based access to services mean it’s cheap to get started, and costs scale up as your data volumes grow
- Get your prototype into production sooner
- Deliver a quicker ROI on your efforts
- Easily find skilled staff and get support because it’s a popular and well-supported platform.
When implementing your big data strategy, it’s also best practice to take an agile approach. This means you can launch sooner with lower costs, while continually tweaking as the technology, available data, and requirements evolve.
Over time, your big data practices will likely evolve, too. Just like we shared in our article on analytics vs BI vs data science vs machine learning — your business can start with the basics and mature your data practices as you go.
Think You Need a Big Data Solution? We Can Help.
Got a business challenge or objective and think big data might be the answer? Our team of Adelaide-based engineers, developers, project managers, and analysts are here to help.
Contact us and let’s start the conversation about what you’d like to achieve in your business.