Public sector organizations generate huge volumes of log data each day from servers, virtualization infrastructure, databases, security systems, applications, and more. And, according to IDC, unstructured data is growing at an annual compound rate of 60%. But due to its unstructured nature, that data, often called machine data, is much harder to analyze than structured data.

Technological innovations in the storage and computing capacity in the world of data has moved exponentially – especially in the healthcare space.  One of the most dramatic use cases is in the field of genomics.  As Andrea Norris, the CIO for NIH mentioned at a public sector Healthcare Summit on IT Modernization last week, technology has greatly increased the speed and lowered the costs associated with gene sequencing.  Originally, when the human genome project completed the task, it took around 13 years with funding around 2.1 billion dollars (the project was from 1990–2003)

2019 is off to a great start for anyone in the business of working with data in the public sector.  President Trump signed the Foundations for Evidence-Based Policymaking Act on January 14th, and with it, the OPEN Government Data Act has finally become law.[1]

Expert Panel: The Challenges and Opportunities for Modernizing Data Protection

As online data has become ubiquitous, managing that data has become as important an endeavor as amassing and storing it. A host of issues surround data management, not the least of which is security. But many others loom as data increases exponentially both in size and in importance.

Law enforcement and intelligence agencies deal with large volumes of disparate data on a daily basis. Analyzing all this data across multiple data silos and structures, each with their own levels of security permissions, is a big challenge.

The Federal Data Strategy principles (https://strategy.data.gov/principles), as currently articulated, are a set of best practices and guidelines, which could be utilized to govern the development and maturity of an organization’s management of data as an asset.  However, without guidance or a framework within which to actualize these principles, these principles may well be rendered a wish-list.

Any federal IT pro can tell you that analyzing log files is something they’ve been doing for years. That said, as applications get more complex, performance becomes more important and security issues increase. Log analytics are fast becoming a critical component of an agency’s monitoring and management infrastructure.

So much data, so little time. Disparate sources such as sensors, machines, geo-location devices, social feeds, server and security system logs, and more, are generating terabytes of data at unfathomable speeds. Getting any kind of real-time insight and, we dare you to dream, acting on that data as it flows in, is not an easy feat for resource-constrained government agencies.

Data is everywhere in government but turning that data into actionable information and insights remains a persistent problem. “We are data rich and information poor,” said Shelley Metzenbaum, a former associate director for performance and personnel management at the Office of Management and Budget (OMB) at a recent IBM Center for The Business of Government session on “Envision Government in 2040”.

At DLT, July is Big Data Month, where we will be highlighting all things big data in the public sector.  To kick off Big Data Month, we sat down with DLT Chief Data Scientist Sherry Bennett to get her insights into what is going on the world of public sector data and analytics:

INTERVIEWER: So Sherry, what's your story and what led you to DLT?