Big Data Month: New eBook Sheds Light on How Government is Overcoming Persistent Big Data Challenges
There are many opportunities in the public sector for data science and data analytics, yet, almost as many challenges. When we kicked off Big Data Month at DLT, we asked our Chief Data Scientist, Sherry Bennett, for her insights. What became clear is that the obstacles to big data success are universal to both the public and private sector: “…everybody…is grappling with the same thing. They are trying to figure out how to take data that is in different systems and integrate it so that you have a full view of your customer and business operations. Everybody is still struggling with that. How to integrate and how to best utilize that integration.”
Why is data integration so hard? Let’s break it down.
Data is Evolving
Technology is moving forward at a rapid pace, bringing with it new data sources – IoT, sensor-enabled equipment, social network feeds, data warehouses, SaaS, etc. and data types – structured, unstructured, raw and processed. Unique applications and/or complex translations are often required to simply view the data, and correlation of the data in these environments is both complicated and costly. In many instances, these systems have no way of communicating. Data is also taken on another attribute – velocity. The speed at which data is being received and acted upon is increasing.
Aging Technology
Legacy data storage systems and architectures are ill-equipped to deal with these characteristics of the modern data enterprise. For example, any agency that processes benefits and grants must deal with a subset of applicants who may try to fraudulently claim benefits to which they are not entitled. Most governments face this risk, and their aging technology infrastructures are vulnerable.
In many instances, claims audit systems still run on decades-old technologies. A typical batch job might take days to process. Furthermore, these systems may only have access to incomplete or stale data. This limits an agency’s ability for data discovery and makes it easier for perpetrators of fraud to avoid detection.
Cold Hard Cash
And then, of course, there’s money. Finding ways to manage all this data insanity on shrinking budgets, which gives users the capability they need to solve problems, glean insights, and make data actionable, means a lower cost data architecture is needed.
The Answer?
How can you overcome these challenges? That’s the topic of our latest eBook, written in collaboration with Hortonworks. A quick read, using use cases and examples the books explains how public sector agencies can build a modern, secure, and open source data architecture that:
• Stores and analyzes data at massive scale
• Connects people and devices across boundaries and roles for better collaboration
• Extracts critical insights from all types of data from any source
• Reduces data storage costs with inexpensive commodity storage
Download the eBook: Hortonworks Data Platform (HDP) for Govies Today.
Related Blog Posts
Big Data & Analytics, Healthcare, Market Intelligence, State & Local Government
Maternal and child health (MCH) has become a top concern nationwide, particularly in disadvantaged, underserved and rural communities. According to the U.S. Department of Health and Human Services (HHS), the United States is experiencing a maternal health crisis nationwide, as it has one of the highest maternal mortality rates among high-income nations; increasing rates of complications from pregnancy or childbirth; and persistent disparities in such outcomes.
Yvonne Maffia
Big Data & Analytics, Market Intelligence, State & Local Government
Since 2023, we have seen an explosion in the number of state data privacy laws, demonstrating a clear focus amongst state governments in data privacy. In fact, according to the National Conference of State Legislatures, “at least 40 states and Puerto Rico introduced or considered at least 350 consumer privacy bills in 2023.” These data privacy laws create protections for a range of several types of data, from general consumer data protections to specific protections for health data and minor data.
Gabriel Zighelboim
Big Data & Analytics, Market Intelligence, State & Local Government
The year 2024 is just around the corner, and with a new year typically comes new priorities. However, some things, like digital government, are here to stay. Improving the Citizen Experience will continue to be a top priority both now and, in the future, as the modern world emphasizes the importance of personalized services that provide agility, accountability, transparency, equity, efficiency, adaptability.
Prioritizing Citizen Experience
Yvonne Maffia
Big Data & Analytics, Market Intelligence, State & Local Government
Data analytics played a pivotal role in state, local, and education (SLED) governments’ response to the COVID-19 pandemic, and the emphasis on the importance of data has only continued to magnify over the last couple of years. Data analytics and related technology tools provide a host of benefits ranging from real-time data analysis, increased transparency and accountability, streamlined citizen facing services, better decision making and savings in both costs and time.
Yvonne Maffia
Big Data & Analytics, Data and Analytics, Federal Government, Geospatial, Market Intelligence
As the demand for trustworthy visual data collection and advanced technologies to support strategic decision-making continues to rise, geospatial intelligence plays a vital role in delivering essential military and environmental insights to the intelligence community. Recently at the GEOINT Conference in St. Louis, leaders from several intelligence agencies shed light on the future of geospatial intelligence (GEOINT) and its relationship with emerging technologies.
Toan Le
Big Data & Analytics, Cybersecurity, Market Intelligence
In a recent webinar produced by Federal News Network, the Director of the Environmental Protection Agency (EPA)’s Office of Information Security and Privacy, Tonya Manning, detailed the state of the agency’s zero trust and data handling postures, as well as its latest priorities. We’ll spotlight several takeaways and look at what’s to likely come down the pike for the EPA in the coming months and years.
Zero Trust Architecture
Susanna Patten
Big Data & Analytics, Federal Government, State & Local Government
In this Q&A discussion, the former Chief of People and Culture shares her insights into employee and customer experience in government, along with tips to improve both areas.
Mary Kearl
Application Lifecycle, Big Data & Analytics, Cloud Computing, Cybersecurity, DevSecOps, IT Infrastructure
For the second year in a row, Gartner named IBM a Leader in Gartner Magic Quadrant for 2021 Cloud Database Management Systems based on its Ability to Execute and Completeness of Vision. With emergence of a single cloud DBMS market, We believe our portfolio of feature-rich, enterprise-tested offerings, bold acquisitions, and partnerships enable our clients to address the unique needs of their business, respond to the growing volume, velocity and variety of today’s data and drive more accurate data driven decisions.
Holly Vatter
Application Lifecycle, Big Data & Analytics, Cloud Computing, Cybersecurity, DevSecOps, IT Infrastructure
This week's roundup of the latest news and insights gathered from IBM's Government Research Institute thought leaders:
Michael J. Keegan
Analytics & Data Science, Big Data & Analytics, Cloud Computing, Cybersecurity, Market Intelligence
The COVID-19 pandemic has spurned greater demand for health information technology (IT) by demonstrating the importance of having robust medical research, health surveillance and healthcare systems capable of rapidly responding to new and developing situations, something which requires strong IT investment in big data, cybersecurity and cloud. In addition, both the pandemic and emerging technologies have led to numerous changes within the healthcare industry, such as telehealth expansion and increased use of wearables, which necessitate robust health IT solutions.
Gabriel Zighelboim
Big Data & Analytics, Cloud Computing
What is the difference between AI and ML?
Artificial Intelligence and Machine Learning are two of the most transformative technologies in the computing space that are changing the world in which we live. However, there is often confusion in in what constitutes Artificial Intelligence, and what constitutes Machine Learning.
Ricardo Dutton
Big Data & Analytics, Cybersecurity, Market Intelligence
President Joe Biden signed the National Defense Authorization Act (NDAA) for fiscal year 2022 (FY22) into law on December 27, 2021. It authorizes $770 billion in defense spending which is a 5% increase over last year. This marks 61 consecutive years that a bill received bipartisan support from congress (a display of agreement that has become increasingly rare for DC politics).
Toan Le
Big Data & Analytics
Customer experience is now an executive order in the United States! With this monumental news, Lee Becker dives deeper into what this means for federal employees and their customers.
This was a momentous and inspiring week for all U.S. citizens and residents, as President Joe Biden signed an executive order on December 13, 2021, that mandates a focus on customer experience.
Lee Becker
Big Data & Analytics, Cloud Computing, Cybersecurity
With another busy year behind us, it’s time to look ahead to fiscal year (FY) 2022. The official information technology (IT) budget request is $97B, a 4% increase over FY21, which would be a new record. Of course, those numbers undercount all the IT spending that goes unreported. Furthermore, remaining provisions in the American Rescue Plan, the Technology Modernization Fund and IT provisions in the Infrastructure Bill will represent additional pockets of opportunity worth billions for channel partners and technology vendors.
Lloyd McCoy
Big Data & Analytics
DLT’s Chief Data Scientist, Sherry Bennett, recently sat down recently with Lora L. Allen, Medallia’s Public Sector Principal CX Advisor and discussed her service in the public sector, Lora’s role at Medallia helping educate and advise government agencies that are tackling CX challenges, and more.
Q: Prior to joining Medallia, you spent your entire career in public service, can you kindly tell me how your service has shaped and informed your current focus on customer experience?
DLT Solutions
Big Data & Analytics
State governments want their software to be secure, compliant and designed for the 21st century. As such, many state governments are evaluating their current ERP platforms to determine how to modernize financials, procurement, human capital management, and payroll applications.
While some have pursued modernization by embarking on multi-tenant cloud journeys, most organizations are still in the planning phase.
DLT Solutions
Big Data & Analytics
In today’s volatile and disruptive economic environment, CFOs and Finance organizations must lead organizational decision-making processes with insight, speed and confidence. Yet many Finance organizations are still bogged down by inefficiencies in routine processes within the period-end financial close and reporting cycle. And that makes it difficult to shift time to value-added analysis and decision support.
DLT Solutions
Big Data & Analytics
The notion of Data Analytics has been in existence for a very long time. Everyone understood data and analytics as essential to “someone” within the organization, but somehow at a distance. In the best case, the technology team would provide a user with the requested data as a means of assessing success and forecasting the future needs of the organization.
DLT Solutions
Big Data & Analytics, Channel Partners, Digital Design
Does your business have a single source of truth when it comes to data sharing across facilities? Ensuring data is accurate, relevant, and structured is something any business can recognize as important, yet it is more difficult to achieve than most realize. Just because you are use an integrated workplace management system (IWMS) or have relevant GIS data doesn’t mean it provides a single source of truth.
Michael Willette
Big Data & Analytics
Exclusive Interview with DLT Chief Data Scientist, Sherry Bennett and Data Coalition CEO, Nick Hart
Sherry: Welcome Nick Hart, CEO of the Data Coalition. Thank you for participating in our interview today to help launch DLT’s inaugural Data Innovations newsletter. Nick, please tell me a little bit about your professional career in government and how you became CEO of the Data Coalition.
Brandon Norris
Big Data & Analytics
On January 1, 2021, the House and Senate overrode President Trump’s veto of H.R.6395, the National Defense Authorization Act (NDAA) for Fiscal Year 2021, rendering the bill law. The FY2021 NDAA includes the National Artificial Intelligence Initiative Act (NAIIA), one of the most ambitious attempts by Congress to date, seeking to establish the United States leadership in artificial intelligence (AI).
Sherry Bennett
Big Data & Analytics
Article by Rick F. van der Lans
DLT Solutions
Big Data & Analytics
The United States spent $554 billion in discretionary contracts for goods and services during FY18, covering the gamut from janitorial services to hand grenades. This number ranks higher than Sweden’s gross domestic product, as well as 165 other countries’ GDP.
William Bunce
Big Data, Big Data & Analytics, Market Intelligence
Since the release of the final Federal Data Strategy Action Plan (henceforth FDSAP) the end of last year, much has been written about the Federal Data Strategy. However, little has been written to connect the important role it can play in enabling the adoption of AI within the federal public sector.
Sherry Bennett
Big Data, Big Data & Analytics
Is data the new bacon? The world’s most valuable resource? That fuel that powers the digital enterprise?
The most successful companies in the world – Google, Facebook, Amazon, Netflix, etc. – use data to drive business strategies. Being insight-driven isn’t just the domain of consumer tech. Government agencies rely on data to make informed and quick decision making, enhance productivity, improve transparency and build trust with citizens, eliminate fraud and abuse, reduce crime and security threats, and more.
DLT Solutions
Big Data & Analytics
Data continues to transform the public sector in meaningful and compelling ways. As a vast creator and consumer of data, government agencies have a unique opportunity to revolutionize their decision-making power by utilizing data to the fullest.
Shaun Bierweiler
Big Data & Analytics
The summer of 2019 is off to a great start for data professionals seeking to make valuable contributions working in the federal public sector. After several solicitations for public comments over the last year, the Office of Management and Budget (OMB) has at last issued the final draft of the Federal Data Strategy. The Federal Data Strategy by design is intended to help the government accelerate the use of data to drive and deliver mission objectives.
Sherry Bennett
Big Data & Analytics
A survey of Department of Defense employees commissioned by DLT partner, Veritas, and conducted by Federal Computer Week found that 46% of respondents agree that data drives all or most of their decisions (58%), yet only 13% would rate their data management capabilities as “Excellent” while 60% rate them as “Satisfactory” or “Poor”.
Challenges Across Each Stage of Data Life Cycle
Sherry Bennett
Big Data & Analytics
The purpose of this workshop is to provide senior-level executives with a framework and set of actionable steps to understand and assess the data maturity of their organization. By leveraging the Federal Government Data Maturity Model (FGDMM), attendees will engage in "hands-on" exercises to articulate a data strategy and roadmap for their agency.
At this workshop, you will hear from a number of industry leaders as they share their experiences with Data Strategy and Management, including:
Sherry Bennett
Big Data & Analytics
The database is at the heart of every application; when applications are having performance issues, there’s a good chance the database is somehow involved. The greater the depth of insight a federal IT pro has into its performance, the more opportunity to enhance application performance.
Every federal IT pro collects (or, should collect) database performance information. Yet, there is a dramatic difference between simply collecting data and correlating and analyzing that data to get actionable results.
DLT Solutions
Big Data & Analytics
Public sector organizations generate huge volumes of log data each day from servers, virtualization infrastructure, databases, security systems, applications, and more. And, according to IDC, unstructured data is growing at an annual compound rate of 60%. But due to its unstructured nature, that data, often called machine data, is much harder to analyze than structured data.
Rachael Hendrickson
Big Data & Analytics
Technological innovations in the storage and computing capacity in the world of data has moved exponentially – especially in the healthcare space. One of the most dramatic use cases is in the field of genomics. As Andrea Norris, the CIO for NIH mentioned at a public sector Healthcare Summit on IT Modernization last week, technology has greatly increased the speed and lowered the costs associated with gene sequencing. Originally, when the human genome project completed the task, it took around 13 years with funding around 2.1 billion dollars (the project was from 1990–2003)
Sherry Bennett
Big Data & Analytics, Cybersecurity
Expert Panel: The Challenges and Opportunities for Modernizing Data Protection
As online data has become ubiquitous, managing that data has become as important an endeavor as amassing and storing it. A host of issues surround data management, not the least of which is security. But many others loom as data increases exponentially both in size and in importance.
DLT Solutions
Big Data & Analytics, Digital Design
The state of our nation’s infrastructure is grim. The most recent report card from the American Society of Civil Engineers (ASCE) gave U.S. infrastructure an overall score of “D” for “poor” or “at risk”.
Caron Beesley
Big Data & Analytics
This time of year brings reflection and blessings to count. The Big Data, Analytics and Data Science team at DLT has a lot to be grateful for, as we’ve grown and expanded at an accelerated pace, finding success along the way. As the team and portfolio grow, so do the strategic partnerships, initiatives, activities, and tactics.
Rachael Hendrickson
Big Data & Analytics
Law enforcement and intelligence agencies deal with large volumes of disparate data on a daily basis. Analyzing all this data across multiple data silos and structures, each with their own levels of security permissions, is a big challenge.
Rachael Hendrickson
Big Data & Analytics
The Federal Data Strategy principles (https://strategy.data.gov/principles), as currently articulated, are a set of best practices and guidelines, which could be utilized to govern the development and maturity of an organization’s management of data as an asset. However, without guidance or a framework within which to actualize these principles, these principles may well be rendered a wish-list.
Sherry Bennett
Big Data & Analytics
Form follows function – and so should your organization’s data!
Sherry Bennett
Big Data & Analytics
Digital twins may sound like the stuff of science fiction. Yet, digital twin technology is one of the top 10 strategic technology trends for 2018, according to research firm Gartner, Inc. Gartner® recently announced 48% of surveyed organizations working with the Internet of Things (IoT) say they are already using, or plan to use, digital twins in 2018. Gartner’s survey also indicated that adoption of digital twin technology is expected to triple by 2022.
Paul Parker
Big Data & Analytics
The use of facial recognition software to support law enforcement efforts is nothing new – in the movies that is. In 1997, the sci-fi classic The Fifth Element equipped its futuristic airborne police vehicles with tools that let officers compare facial scans of the movie’s mysterious alien heroine to a central criminal database in the hope of making an identification.
Twenty years on, scenarios like this are fast becoming normalized.
Rachael Hendrickson
Big Data & Analytics
Data is everywhere in government but turning that data into actionable information and insights remains a persistent problem. “We are data rich and information poor,” said Shelley Metzenbaum, a former associate director for performance and personnel management at the Office of Management and Budget (OMB) at a recent IBM Center for The Business of Government session on “Envision Government in 2040”.
DLT Solutions
Big Data & Analytics
At DLT, July is Big Data Month, where we will be highlighting all things big data in the public sector. To kick off Big Data Month, we sat down with DLT Chief Data Scientist Sherry Bennett to get her insights into what is going on the world of public sector data and analytics:
INTERVIEWER: So Sherry, what's your story and what led you to DLT?
DLT Solutions
Big Data, Big Data & Analytics, Cybersecurity
On Tax Day, 2018, the unthinkable (but not unexpected) happened. The IRS’ IT system crashed after a hardware issue took down a number of apps including those that interface with third-party tax preparation software as well as the IRS Direct Pay application.
IRS IT Systems are Decades Old
DLT Solutions
Big Data, Big Data & Analytics, Data & Storage, Data and Analytics, Healthcare, Technology
Blockchain is no longer just a buzzword or simply a “technology to watch.” This database technology is being explored by agencies across government, from the General Services Administration (GSA) to Centers for Medicare and Medicaid Services (CMS), from the Federal Maritime Commission to military supply chain professionals across the Department of Defense (DoD).
The promise of blockchain is dramatic. It can help enhance agencies’ business processes and provide far greater transparency and efficiency.
Paul Parker
Big Data, Big Data & Analytics
Government shutdowns are a costly business. The 2013 shutdown cost $24 billion in lost economic output while the 1996 shutdown resulted in $2.1 billion in government costs. We are yet to learn the impact, if any, of the three-day 2018 shutdown.
But, what we do know is that shutdowns are not universal. For many critical government employees, the lights never go out. Here’s just a shortlist:
Veterans Affairs (VA) remained operational.
Valerie Rider
Big Data & Analytics
Take your cloud data, database data, HR data, financial data, throw it in a blender and what have you got? Data soup! It may as well be. You have all the right ingredients, but no recipe for pulling it all together. As we accumulate more data, across disparate data sources, the challenge for today’s government data analyst are multiple:
rachael.lynch
Big Data & Analytics
Public sector leaders are facing an uphill battle when it comes to managing data. Their needs are growing while their budgets are often shrinking. How can federal agencies do more with less?
Leveraging enterprise open source solutions is part of the answer. They can help agencies close this gap and meet today’s needs—whether that’s analyzing warfighter data or building Smart Cities—while also planning for tomorrow’s challenges.
DLT Solutions
Big Data & Analytics
The U.S. Government spends trillions of dollars on benefits programs like Social Security, Medicare and Medicaid each year. Unfortunately, billions of those dollars are improperly paid, reducing the benefits to those who most rely upon them. In 2016, the White House estimated these losses at $144B.
Improper payments, fraud, and abuse takes many forms, consider some of these examples of Medicaid fraud and abuse:
david.rubal
Big Data & Analytics
With a constant influx of data, one of the biggest challenges facing government agencies is determining “What is the right data?” or “What data does my agency need for mission success?”
Then, once the data is discovered, how do you make that data actionable? How do you integrate and visualize it for better insights?
david.rubal