Bigger Data - Information for Actionable Intelligence
By Michael Shepherd, Regional MD - ANZ, International Services and Solutions BAE Systems Applied Intelligence
Michael Shepherd, Regional MD - ANZ, International Services and Solutions BAE Systems Applied Intelligence
Gaining an ‘Information Edge’
Modern businesses must harness data and intelligence within their business; pooling resources, data and intelligence from previously unconnected areas, to create an even richer, deeper picture of their business, and the threats and opportunities existing within them.
Comparing behaviours and patterns between sets of data rather than relying on a single source reveals previously-hidden insight, leading to greater understanding, improved decision-making, and actionable intelligence.
As big data becomes an intrinsic part of business, the focus is shifting to using advanced information analytics and machine learning to correlate data sources in real-time, enabling speedy positive outcomes and results.
Considerations for Implementing Big Data Solutions
BAE Systems has vast experience in implementing big data platforms. When doing so, we consider a number of factors throughout the design and implementation phases:
1. Harnessing the power of big data: A well-planned, incremental, and well-mitigated approach, which gives data its due respect as an asset, is the key to success.
2. Continuous improvement: A good data management system should use data analytics and machine learning to continually improve performance.
3. Making big data smaller: This means intelligently filtering and managing the vast quantities of raw data as it enters the organisation, weeding out extraneous information before it is stored, and applying techniques to de-duplicate, compress and delete unneeded data.
4. Maintaining data availability: Maintaining data availability and integrity are essential to a fully utilised platform. Data records must be available on demand and accurate for deep-dive and compliance purposes.
5. Creating a single intelligence picture: Bringing information together to create a single intelligence picture, or a ‘single pane of glass’, is important so that organisations are able to effectively investigate areas of interest, conduct deep-dive reconnaissance, and extract meaning from data.
Within many industries, even more is to be gained from sharing information across organisations
6. Data governance and ethics: Data privacy and data security must be considered at every step of the design and implementation process.
7. Solution testing: Often the greatest challenge is the sheer amount of solution testing and discovery required for a successful implementation.
In my experience, successful Big Data solutions contain some combination of five key components:
1. Filter and pre-processing: Where it is not practical to store everything.
2.Real-time analytics platform: Accepting streaming and pre-summarised data, with the purpose of gleaning intelligence from data streams as they occur in real time (or near real time).
3. Batch processing platform: This takes the main load of the bulk data queries and complex analytics –usually the core of an enterprise Big Data solution.
4. Search and query platform: This provides a scalable data store to support fast resolution of point queries, where the type of query is predictable and the indexing requirements are known up front.
5. Big Data specialists: Many Big Data tools require specialist skill sets, soorganisations will require a team of specialists to manage the toolsets, structure the queries and analysis logic, and interpret the results.
Recently, we implemented a data solution for a large Communications Service Provider (CSP) in Australia, to enable it to comply with Australia’s data retention legislation, mirroring similar requirements around the world.
The program, unprecedented in its complexity and scale in Australia, utilised our advanced technology and experience delivering similar large-scale solutions to CSPs and law enforcement around the world, to deliver the core platform and integrate massive volumes of complex information, allowing for timely and simple search and extraction of the data for up to two years as required.
Implementing Large Scale Information Sharing Platforms
Within many industries, even more is to be gained from sharing information across organisations. Clearly there are boundaries to this – a company’s IP and way of doing business are often their differentiator, and competitive advantage lies within this. However, an example of an area where information sharing within an industry will benefit the entire industry as a whole is in sharing intelligence about cyber threats.
Cyber Threat Intelligence (CTI) sharing within industries is becoming increasingly critical to improve the effective cyber defence capabilities of organisations.
In the BAE Systems 2017 Cyber Defence Monitor report, business leaders said knowledge sharing – with peers, law enforcement, governments, and IT security firms – was critical to supplementing their defences against cyber crime, and 83 per cent of Australian execs said greater intelligence about threats in their industry would help them to take action against potential cyber attacks.
It’s clear from our research that effective collaboration, communication, and intelligence sharing are the bedrock on which effective cyber resiliency can be built.
BAE Systems have experience developing and implementing next-generation information sharing systems that enable cross-organisation, and often cross-border, information sharing in a secure environment, including a multiple security level, multi-intelligence analyst environment, whilst maintaining compliance standards with data retention and data privacy requirements.
BAE Systems provides some of the world’s most advanced, technology led defence, aerospace and security solutions operating in over 40 countries, including in the Asia-Pacific; Australia, Singapore, Malaysia, Philippines, New Zealand, Indonesia, Brunei, India and Japan, and with its Head Office in London.
See Also:
