6. Data governance and ethics: Data privacy and data security must be considered at every step of the design and implementation process.

7. Solution testing: Often the greatest challenge is the sheer amount of solution testing and discovery required for a successful implementation.

In my experience, successful Big Data solutions contain some combination of five key components:

1. Filter and pre-processing: Where it is not practical to store everything.

2.Real-time analytics platform: Accepting streaming and pre-summarised data, with the purpose of gleaning intelligence from data streams as they occur in real time (or near real time).

3. Batch processing platform: This takes the main load of the bulk data queries and complex analytics –usually the core of an enterprise Big Data solution.

4. Search and query platform: This provides a scalable data store to support fast resolution of point queries, where the type of query is predictable and the indexing requirements are known up front.

5. Big Data specialists: Many Big Data tools require specialist skill sets, soorganisations will require a team of specialists to manage the toolsets, structure the queries and analysis logic, and interpret the results.

Recently, we implemented a data solution for a large Communications Service Provider (CSP) in Australia, to enable it to comply with Australia’s data retention legislation, mirroring similar requirements around the world.

The program, unprecedented in its complexity and scale in Australia, utilised our advanced technology and experience delivering similar large-scale solutions to CSPs and law enforcement around the world, to deliver the core platform and integrate massive volumes of complex information, allowing for timely and simple search and extraction of the data for up to two years as required.

Implementing Large Scale Information Sharing Platforms

Within many industries, even more is to be gained from sharing information across organisations. Clearly there are boundaries to this – a company’s IP and way of doing business are often their differentiator, and competitive advantage lies within this. However, an example of an area where information sharing within an industry will benefit the entire industry as a whole is in sharing intelligence about cyber threats.

Cyber Threat Intelligence (CTI) sharing within industries is becoming increasingly critical to improve the effective cyber defence capabilities of organisations.

In the BAE Systems 2017 Cyber Defence Monitor report, business leaders said knowledge sharing – with peers, law enforcement, governments, and IT security firms – was critical to supplementing their defences against cyber crime, and 83 per cent of Australian execs said greater intelligence about threats in their industry would help them to take action against potential cyber attacks.

It’s clear from our research that effective collaboration, communication, and intelligence sharing are the bedrock on which effective cyber resiliency can be built.

BAE Systems have experience developing and implementing next-generation information sharing systems that enable cross-organisation, and often cross-border, information sharing in a secure environment, including a multiple security level, multi-intelligence analyst environment, whilst maintaining compliance standards with data retention and data privacy requirements.

BAE Systems provides some of the world’s most advanced, technology led defence, aerospace and security solutions operating in over 40 countries, including in the Asia-Pacific; Australia, Singapore, Malaysia, Philippines, New Zealand, Indonesia, Brunei, India and Japan, and with its Head Office in London.