Transforming Big Data into Smart Data for Smart Energy: Deriving Value via harnessing Volume, Variety and Velocity
•Download as PPTX, PDF•
7 likes•11,051 views
Keynote at the Workshop on Building Research Collaboration: Electricity Systems. Purdue University, West Lafayette, IN. Aug 28-29, 2013. Abstract: Big Data has captured much interest in research and industry, with anticipation of better decisions, efficient organizations, and many new jobs. Much of the emphasis is on technology that handles volume, including storage and computational techniques to support analysis (Hadoop, NoSQL, MapReduce, etc), and the challenges of the four Vs of Big Data: Volume, Variety, Velocity, and Veracity. However, the most important feature of data, the raison d'etre, is neither volume, variety, velocity, nor veracity -- but value. In this talk, I will emphasize the significance of Smart Data, and discuss how it is can be realized by extracting value from Big Data. Accomplishing this task requires organized ways to harness and overcome the original four V-challenges; and while the technologies currently touted may provide some necessary infrastructure-- they are far from sufficient. In particular, we will need to utilize metadata, employ semantics and intelligent processing, and leverage some of the extensive work that predates Big Data. For achieving energy sustainability, Smart Grids are known to transform the way we generate, distribute, and consume power. Unprecedented amount of data is being collected from smart meters, smart devices, and sensors all throughout the power grid. I will discuss the central question of deriving Value from the entire smart grid data deluge by discussing novel algorithms and techniques such as Semantic Perception for dealing with Velocity, use of ontologies and vocabularies for dealing with Variety, and Continuous Semantics for dealing with Velocity. I will discuss scenarios that exemplify the process of deriving Value from Big Data in the context of Smart Grid. Additional background is at: http://wiki.knoesis.org/index.php/Smart_Data A previous version of this talk with more technical details but not focused on energy: http://j.mp/SmatData
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Transforming Big Data into Smart Data for Smart Energy: Deriving Value via harnessing Volume, Variety and Velocity
Similar to Transforming Big Data into Smart Data for Smart Energy: Deriving Value via harnessing Volume, Variety and Velocity (20)
Recently uploaded
Recently uploaded (20)
Transforming Big Data into Smart Data for Smart Energy: Deriving Value via harnessing Volume, Variety and Velocity
- 1. 1
- 2. 2 Power Grids: A Historical Perspective on Complexity Before Alternating Current (AC) After Alternating Current After/During Smart Grid High System Complexity! Moderate System Complexity + Low Data Complexity High System + Data Complexity! Separate power lines for different voltages. AC as a boon for Electric companies. Smart Grid = high volume, variety and velocity http://en.wikipedia.org/wiki/Electric_power_transmission Late 1800’s 1900’s Today
- 3. 3 Big Data in Smart Grid One data point per month 96 million data points / day / million consumers Low instrumentation of the power grid with sensors High instrumentation of the power grid with sensors Low number of energy sources High proliferation of cleaner energy sources like renewable energy http://www.smartgridupdate.com/dataforutilities/pdf/DataManagementWhitePaper.pdf
- 4. 4 Sources of Big Data in Smart Grid Velocity Volume Variety Veracity Original 3Vs: Doug Laney: http://goo.gl/wH3qG From: http://www.smartgridupdate.com/dataforutilities/pdf/DataManagementWhitePaper.pdf
- 5. 5 Big Data Analytics in Smart Grid
- 6. • What if your data volume gets so large and varied you don't know how to deal with it? • Do you store all your data? • Do you analyze it all? • How can you find out which data points are really important? • How can you use it to your best advantage? 6 Questions typically asked on Big Data http://www.sas.com/big-data/
- 7. http://techcrunch.com/2012/10/27/big-data-right-now-five-trendy-open-source-technologies/ Variety of Data Analytics Enablers 7
- 8. • Prediction of the spread of flu in real time during H1N1 2009 – Google tested a mammoth of 450 million different mathematical models to test the search terms, comparing their predictions against the actual flu cases; 45 important parameters were founds – Model was tested when H1N1 crisis struck in 2009 and gave more meaningful and valuable real time information than any public health official system [Big Data, Viktor Mayer-Schonberger and Kenneth Cukier, 2013] • FareCast: predict the direction of air fares over different routes [Big Data, Viktor Mayer-Schonberger and Kenneth Cukier, 2013] • NY city manholes problem [ICML Discussion, 2012] 8 Illustrative Big Data Applications
- 9. • Current focus mainly to serve business intelligence and targeted analytics needs, not to serve complex individual and collective human needs (e.g., empower human in health, fitness and well-being; better disaster coordination, smart energy consumption) that is highly personalized/individualized/contextualized – Incorporate real-world complexity: multi-modal and multi-sensory nature of real- world and human perception – Need deeper understanding of data and its role to information (e.g., skew, coverage) – Beyond correlation -> causation :: actionable info, decisions grounded on insights • Human involvement and guidance: Leading to actionable information, understanding and insight right in the context of human activities – Bottom-up & Top-down processing: Infusion of models and background knowledge (data + knowledge + reasoning) 9 What is missing?
- 10. Makes Sense Actionable or help decision support/making 10
- 11. Smart Data Smart data makes sense out of Big data It provides value from harnessing the challenges posed by volume, velocity, variety and veracity of big data, in-turn providing actionable information and improve decision making. 11
- 12. “OF human, BY human and FOR human” Smart data is focused on the actionable value achieved by human involvement in data creation, processing and consumption phases for improving the human experience. Another perspective on Smart Data 12
- 13. • Focus on verticals: advertising‚ social media‚ retail‚ financial services‚ telecom‚ and healthcare – Aggregate data, focused on transactions, limited integration (limited complexity), analytics to find (simple) patterns – Emphasis on technologies to handle volume/scale, and to lesser extent velocity: Hadoop, NoSQL,MPP warehouse …. – Full faith in the power of data (no hypothesis), bottom up analysis 13 Current Focus on Big Data
- 14. Descriptive Exploratory Inferential Predictive Causal Improved Analytics CREATION PROCESSING EXPERIENCE & DECISION MAKING 14 Human Centric Computing
- 15. “OF human, BY human and FOR human” Another perspective on Smart Data 15
- 16. Petabytes of Physical(sensory)-Cyber-Social Data everyday! More on PCS Computing: http://wiki.knoesis.org/index.php/PCS 16 „OF human‟ : Relevant Real-time Data Streams for Human Experience
- 17. “OF human, BY human and FOR human” 17 Another perspective on Smart Data
- 18. Use of Prior Human-created Knowledge Models 18 „BY human‟: Involving Crowd Intelligence in data processing workflows Crowdsourcing and Domain-expert guided Machine Learning Modeling
- 19. “OF human, BY human and FOR human” Another perspective on Smart Data 19
- 20. Electricity usage over a day, device at work, power consumption, cost/kWh, heat index, relative humidity, and public events from social stream Weather Application Power Monitoring Application 20 „FOR human‟ : Improving Human Experience Population Level Observations Personal Level Observations Action in the Physical World Washing and drying has resulted in significant cost since it was done during peak load period. Consider changing this time to night.
- 21. 21 What matters? Personal and Population Level Observations Actionable information for optimized resource utilization “The challenge for utilities in maximizing the benefits from smart grid data analytics is the ability to turn the huge volume of smart grid data into value” - Marianne Hedin, Senior Research Analyst, Navigant Research
- 22. 22 Why do we care about Smart Data rather than Big Data?
- 23. Transforming Big Data into Smart Data for Smart Energy: Deriving Value via harnessing Volume, Variety and Velocity using semantics and Semantic Web Put Knoesis Banner Keynote at Building Research Collaborations: Electricity Systems @ Purdue, August 28-29, 2013 Pavan Kapanipathi Pramod Anantharam Amit Sheth Cory Henson Dr. T.K. Prasad Maryam Panahiazar Contributions by many, but Special Thanks to: Hemant Purohit Special Thanks The Ohio Center of Excellence in Knowledge-enabled Computing (Kno.e.sis), Wright State, USA
- 24. 24 10 Years Ago, August 14, 2003 Blackout! http://www.npr.org/2013/08/14/210620446/10-years-after-the-blackout-how-has-the-power-grid-changed Robert Giroux/Getty Images
- 25. 25 50 Million People without Power in 5 Northeastern States of US http://www.npr.org/2013/08/14/210620446/10-years-after-the-blackout-how-has-the-power-grid-changed Jonathan Fickies/Getty Images
- 26. 26 $6 Billion Lost Revenue http://www.scientificamerican.com/article.cfm?id=2003-blackout-five-years-later http://www.npr.org/2013/08/14/210620446/10-years-after-the-blackout-how-has-the-power-grid-changed Julie Jacobson/AP Julie Jacobson/AP Utilities are hit with millions of dollars of fine when such blackouts happen costing them on an average 1 million dollars a day!
- 27. 27 Cause of the Problem: Informal Investigation Excessive summer heat (31° C or 88° F) caused consumers to draw excess power for running air conditioners. Heating of power lines led to sagged cables touching vegetation creating a fault. FirstEnergy (FE) Corporation’s control room had a failed alarm system further propagating the fault (cascading effect). Lack of situational awareness by the control room is only one aspect of the problem. The problem is deeply rooted in consumer awareness for making informed decisions http://www.npr.org/2013/08/14/210620446/10-years-after-the-blackout-how-has-the-power-grid-changed
- 28. 28 Cause of the Problem: Official Investigation The U.S.-Canada Power System Outage Task Force reported four major causes leading to the blackout: 1) "failed to assess and understand the inadequacies of FE's system, particularly with respect to voltage instability and the vulnerability of the Cleveland-Akron area, and FE did not operate its system with appropriate voltage criteria." 2) "did not recognize or understand the deteriorating condition of its system." 3) "failed to manage adequately tree growth in its transmission rights-of-way." 4) "failure of the interconnected grid's reliability organizations to provide effective real-time diagnostic support." http://en.wikipedia.org/wiki/Northeast_blackout_of_2003
- 29. 29 "We've done some things that will reduce the risks of the blackouts that happened last time, but haven't done things that would prevent the next blackout” -- Paul Hines, University of Vermont Can we Prevent such Blackouts? “we have new sensors installed in the grid, but utilities don't totally understand what to do with all the data” -- Paul Hines, University of Vermont http://epaabuse.com/5159/news/after-coal-plants-close-where-does-america-get-cheap-electricity/
- 30. 30 How could Smart Data help? Value: Utilities Context
- 31. 31 Derive Insights from Smart Grid Data "Big data .. for utility companies.. can turn the information from smart meter and smart grid projects into meaningful operational insights and insights about their customer’s behavior." - Big Data in Action, IBM http://www.ecomagination.com/portfolio/ges-grid-iq-advanced-metering-infrastructureami-point-to-multipoint-p2mp-solution http://gkenergyproject.blogspot.com/2010/07/smart-meter-diagram.html
- 32. 32 Power Grid Control Rooms are Complex! Pacific Gas and Electric Company in California has collected over 70 terabytes of AMI (Advanced Metering Infrastructure) data and this volume is increasing by 3 terabytes a month - Data Management And Analytics for Utilities, Smart Grid Update, 2013 http://www.rugeleypower.com/electricity-generation/producing-electricity.php
- 33. 33 Multimodal, Multisensory, and Real-time Observations Synchrophaso r data Heat index, relative humidity Current Grid Conditions Renewable energy generation forecast What is the overall health of the Grid? What are the vulnerabilities for today? Power consumption by consumers http://www.rugeleypower.com/electricity-generation/producing-electricity.php
- 34. 34 Grid Health Score (diagnostic) Semantic Perception and risk assessment algorithms can transform raw data (hard to comprehend) to abstractions (e.g., Grid Health is 3 on a scale of 5) that is intuitively understandable and valuable for decision makers. Having health score for various parts of a grid will allow efficient utilization of a decision maker’s precious attention Risk assessment model Semantic Perception Synchrophaso r data Heat index, relative humidity Current Grid Conditions Renewable energy generation forecast Power consumption by consumers
- 35. 35 Vulnerability Score (prognostic) Vulnerability score (e.g., Today’s vulnerability score 4 on a scale of 5) is an abstraction that uses current state of the grid (health score), power demand forecast, availability of alternative energy sources, and historical consumer behavior Vulnerability score will alleviate the data deluge problem of decision makers by leveraging prior knowledge of the domain for creating risk assessment models Risk assessment model Semantic Perception Synchrophaso r data Heat index, relative humidity Current Grid Conditions Renewable energy generation forecast Power consumption by consumers
- 36. 36 Value: Consumer Context How could Smart Data help?
- 37. 37 “To make good on the promise of a truly “smart” grid, the industry must continue to implement equipment that employs distributed intelligence, out to the edges of the distribution system.“ -- Layered Intelligence Smart Grid Solutions, S&C Electric Company “Intelligence at the Edges” of a Smart Grid http://www.digikey.com/us/en/techzone/energy-harvesting/resources/articles/zigbees-smart-energy-20-profile.html
- 38. 38 Data Overload for Consumers “They respond well to suggestions to do something.” - Alex Laskey, President and Founder of Opower Personal Schedule Smart Meters Power Consumption Temperature, relative humidity Dynamic pricing information http://www.identika.com/2012/02/every-movie-made/
- 39. 39 Optimizing Cost, Benefit, and Preferences Algorithms on the consumer side of the Smart Grid should should consider cost, benefit, and preference of the user to devise an optimal strategy for power consumption Which devices are contributing to higher power bill? When should I operate the washer/dryer? How much convenience I am willing to forego? Semantic Perception Personalized optimization Personalized recommendation Img: http://marloncarvallovillae.blogspot.com/2011_02_01_archive.html http://www.1800timeclocks.com/icon-time-systems/icon-time-upgrades/icon-time-advanced-pack-upgrade-sb100-pro/ Personal Schedule Smart Meters Power Consumption Temperature, relative humidity Dynamic pricing information
- 40. 41 Big Data to Smart Data: A peek at some domains Healthcare Social Media & Disaster Response http://theshannoncompany.com.au/blog/?p=504
- 41. Sensing is a key enabler of the Internet of Things BUT, how do we make sense of the resulting avalanche of sensor data? 50 Billion Things by 2020 (Cisco) 44
- 42. … and do it efficiently and at scale What if we could automate this sense making ability? 45
- 43. 46 Making sense of sensor data with
- 44. People are good at making sense of sensory input What can we learn from cognitive models of perception? • The key ingredient is prior knowledge 47
- 45. * based on Neisser’s cognitive model of perception Observe Property Perceive Feature Explanation Discrimination 1 2 Perception Cycle* Translating low-level signals into high-level knowledge Focusing attention on those aspects of the environment that provide useful information Prior Knowledge 48
- 46. To enable machine perception, Semantic Web technology is used to integrate sensor data with prior knowledge on the Web 49
- 47. Prior knowledge on the Web W3C Semantic Sensor Network (SSN) Ontology Bi-partite Graph 50
- 48. Prior knowledge on the Web W3C Semantic Sensor Network (SSN) Ontology Bi-partite Graph 51
- 49. Observe Property Perceive Feature Explanation 1 Translating low-level signals into high-level knowledge Explanation Explanation is the act of choosing the objects or events that best account for a set of observations; often referred to as hypothesis building 52
- 50. Explanation Inference to the best explanation • In general, explanation is an abductive problem; and hard to compute Finding the sweet spot between abduction and OWL • Single-feature assumption* enables use of OWL-DL deductive reasoner * An explanation must be a single feature which accounts for all observed properties Explanation is the act of choosing the objects or events that best account for a set of observations; often referred to as hypothesis building 53
- 51. Explanation Explanatory Feature: a feature that explains the set of observed properties ExplanatoryFeature ≡ ∃ssn:isPropertyOf—.{p1} ⊓ … ⊓ ∃ssn:isPropertyOf—.{pn} elevated blood pressure clammy skin palpitations Hypertension Hyperthyroidism Pulmonary Edema Observed Property Explanatory Feature 54
- 52. Discrimination is the act of finding those properties that, if observed, would help distinguish between multiple explanatory features Observe Property Perceive Feature Explanation Discrimination 2 Focusing attention on those aspects of the environment that provide useful information Discrimination 55
- 53. Discrimination Expected Property: would be explained by every explanatory feature ExpectedProperty ≡ ∃ssn:isPropertyOf.{f1} ⊓ … ⊓ ∃ssn:isPropertyOf.{fn} elevated blood pressure clammy skin palpitations Hypertension Hyperthyroidism Pulmonary Edema Expected Property Explanatory Feature 56
- 54. Discrimination Not Applicable Property: would not be explained by any explanatory feature NotApplicableProperty ≡ ¬∃ssn:isPropertyOf.{f1} ⊓ … ⊓ ¬∃ssn:isPropertyOf.{fn} elevated blood pressure clammy skin palpitations Hypertension Hyperthyroidism Pulmonary Edema Not Applicable Property Explanatory Feature 57
- 55. Discrimination Discriminating Property: is neither expected nor not-applicable DiscriminatingProperty ≡ ¬ExpectedProperty ⊓ ¬NotApplicableProperty elevated blood pressure clammy skin palpitations Hypertension Hyperthyroidism Pulmonary Edema Discriminating Property Explanatory Feature 58
- 56. Through physical monitoring and analysis, our cellphones could act as an early warning system to detect serious health conditions, and provide actionable information canary in a coal mine Our Motivation kHealth: knowledge-enabled healthcare 59
- 57. How do we implement machine perception efficiently on a resource-constrained device? Use of OWL reasoner is resource intensive (especially on resource-constrained devices), in terms of both memory and time • Runs out of resources with prior knowledge >> 15 nodes • Asymptotic complexity: O(n3) 60
- 58. intelligence at the edge Approach 1: Send all sensor observations to the cloud for processing Approach 2: downscale semantic processing so that each device is capable of machine perception 61 Henson et al. 'An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-Constrained Devices, ISWC 2012.
- 59. Efficient execution of machine perception Use bit vector encodings and their operations to encode prior knowledge and execute semantic reasoning 010110001101 0011110010101 1000110110110 101100011010 0111100101011 000110101100 0110100111 62
- 60. O(n3) < x < O(n4) O(n) Efficiency Improvement • Problem size increased from 10’s to 1000’s of nodes • Time reduced from minutes to milliseconds • Complexity growth reduced from polynomial to linear Evaluation on a mobile device 63
- 61. 2 Prior knowledge is the key to perception Using SW technologies, machine perception can be formalized and integrated with prior knowledge on the Web 3 Intelligence at the edge By downscaling semantic inference, machine perception can execute efficiently on resource-constrained devices Semantic Perception for smarter analytics: 3 ideas to takeaway 1 Translate low-level data to high-level knowledge Machine perception can be used to convert low-level sensory signals into high-level knowledge useful for decision making 64
- 62. Qualities -High BP -Increased Weight Entities -Hypertension -Hypothyroidism kHealth Machine Sensors Personal Input EMR/PHR Comorbidity risk score e.g., Charlson Index Longitudinal studies of cardiovascular risks - Find correlations - Validation - domain knowledge - domain expert Parameterize the model Risk Assessment Model Current Observations -Physical -Physiological -History Risk Score (Actionable Information) Model CreationValidate correlations Historical observations of each patient Risk Score: from Data to Abstraction and Actionable Information 65
- 63. 66 1 http://www.pdf.org/en/parkinson_statistics 10 million 60,000 $25 billion $100,00 0 1 million People worldwide are living with Parkinson's disease1. Americans are diagnosed with Parkinson's disease each year1. Spent on Parkinson’s alone in a year in the US1 Therapeutic surgery can cost up to $100,000 dollars per patient1. Americans live with Parkinson’s Disease1 Parkinson‟s Disease (PD)
- 64. Parkinson’s disease (PD) data from The Michael J. Fox Foundation for Parkinson’s Research. 67 1https://www.kaggle.com/c/predicting-parkinson-s-disease-progression-with-smartphone-data 8 weeks of data from 5 sensors on a smart phone, collected for 16 patients resulting in ~12 GB (with lot of missing data). Variety Volume VeracityVelocity Value Can we detect the onset of Parkinson’s disease? Can we characterize the disease progression? Can we provide actionable information to the patient? semantics Representing prior knowledge of PD led to a focused exploration of this massive dataset WHY Big Data to Smart Data: Healthcare example
- 65. 68 Big Data to Smart Data Using a Knowledge Based Approach ParkinsonMild(person) = Tremor(person) ∧ PoorBalance(person) ParkinsonModerate(person) = MoveSlow(person) ∧ PoorSleep(person) ∧ MonotoneSpeech(person) ParkinsonAdvanced(person) = Fall(person) Control Group PD Patients Movements of an active person has a good distribution over X, Y, and Z axis Restricted movements by a PD patient can be seen in the acceleration readings Audio is well modulated with good variations in the energy of the voice Audio is not well modulated represented a monotone speech Declarative Knowledge of Parkinson’s Disease used to focus our attention on symptom manifestations in sensor observations
- 66. 69 1http://www.nhlbi.nih.gov/health/health-topics/topics/asthma/ 2http://www.lung.org/lung-disease/asthma/resources/facts-and-figures/asthma-in-adults.html 3Akinbami et al. (2009). Status of childhood asthma in the United States, 1980–2007. Pediatrics,123(Supplement 3), S131-S145. 25 million 300 million $50 billion 155,000 593,000 People in the U.S. are diagnosed with asthma (7 million are children)1. People suffering from asthma worldwide2. Spent on asthma alone in a year2 Hospital admissions in 20063 Emergency department visits in 20063 Asthma
- 67. Asthma is a multifactorial disease with health signals spanning personal, public health, and population levels. 70 Real-time health signals from personal level (e.g., Wheezometer, NO in breath, accelerometer, microphone), public health (e.g., CDC, Hospital EMR), and population level (e.g., pollen level, CO2) arriving continuously in fine grained samples potentially with missing information and uneven sampling frequencies. Variety Volume VeracityVelocity Value Can we detect the asthma severity level? Can we characterize asthma control level? What risk factors influence asthma control? What is the contribution of each risk factor?semantics Understanding relationships between health signals and asthma attacks for providing actionable information WHY Big Data to Smart Data: Healthcare example
- 68. 71 Population Level Personal Public Health Variety: Health signals span heterogeneous sources Volume: Health signals are fine grained Velocity: Real-time change in situations Veracity: Reliability of health signals may be compromised Value: Can I reduce my asthma attacks at night? Decision support to doctors by providing them with deeper insights into patient asthma care Asthma: Demonstration of Value
- 69. 72 Sensordrone – for monitoring environmental air quality Wheezometer – for monitoring wheezing sounds Can I reduce my asthma attacks at night? What are the triggers? What is the wheezing level? What is the propensity toward asthma? What is the exposure level over a day? What is the air quality indoors? Commute to Work Personal Public Health Population Level Closing the window at home in the morning and taking an alternate route to office may lead to reduced asthma attacks Actionable Information Asthma: Actionable Information for Asthma Patients
- 70. Personal, Public Health, and Population Level Signals for Monitoring Asthma Asthma Control => Daily Medication Choices for starting therapy Not Well Controlled Poor Controlled Severity Level of Asthma (Recommended Action) (Recommended Action) (Recommended Action) Intermittent Asthma SABA prn - - Mild Persistent Asthma Low dose ICS Medium ICS Medium ICS Moderate Persistent Asthma Medium dose ICS alone Or with LABA/montelukast Medium ICS + LABA/Montelukast Or High dose ICS Medium ICS + LABA/Montelukast Or High dose ICS* Severe Persistent Asthma High dose ICS with LABA/montelukast Needs specialist care Needs specialist care ICS= inhaled corticosteroid, LABA = inhaled long-acting beta2-agonist, SABA= inhaled short-acting beta2-agonist ; *consider referral to specialist Asthma Control and Actionable Information Sensors and their observations for understanding asthma 73
- 71. 74 Personal Level Signals Societal Level Signals (Personal Level Signals) (Personalized Societal Level Signal) (Societal Level Signals) Societal Level Signals Relevant to the Personal Level Personal Level Sensors (kHealth**) (EventShop*) Qualify Quantify Action Recommendation What are the features influencing my asthma? What is the contribution of each of these features? How controlled is my asthma? (risk score) What will be my action plan to manage asthma? Storage Societal Level Sensors Asthma Early Warning Model (AEWM) Query AEWM Verify & augment domain knowledge Recommended Action Action Justification Asthma Early Warning Model *http://www.slideshare.net/jain49/eventshop-120721, ** http://www.youtube.com/watch?v=btnRi64hJp4
- 72. 75 Population Level Personal Wheeze – Yes Do you have tightness of chest? –Yes ObservationsPhysical-Cyber-Social System Health Signal Extraction Health Signal Understanding <Wheezing=Yes, time, location> <ChectTightness=Yes, time, location> <PollenLevel=Medium, time, location> <Pollution=Yes, time, location> <Activity=High, time, location> Wheezing ChectTightness PollenLevel Pollution Activity Wheezing ChectTightness PollenLevel Pollution Activity RiskCategory <PollenLevel, ChectTightness, Pollution, Activity, Wheezing, RiskCategory> <2, 1, 1,3, 1, RiskCategory> <2, 1, 1,3, 1, RiskCategory> <2, 1, 1,3, 1, RiskCategory> <2, 1, 1,3, 1, RiskCategory> . . . Expert Knowledge Background Knowledge tweet reporting pollution level and asthma attacks Acceleration readings from on-phone sensors Sensor and personal observations Signals from personal, personal spaces, and community spaces Risk Category assigned by doctors Qualify Quantify Enrich Outdoor pollen and pollution Public Health Health Signal Extraction to Understanding Well Controlled - continue Not Well Controlled – contact nurse Poor Controlled – contact doctor
- 73. • Real Time Feature Streams: http://www.youtube.com/watch?v=_ews4w_eCpg • kHealth: http://www.youtube.com/watch?v=btnRi64hJp4 76 Demos
- 74. 77 Smart Data in Social Media & Disaster Response To Understand critical information dynamics in real world events
- 75. Twitris‟ Dimensions of Integrated Semantic Analysis 78Sheth et al. Twitris- a System for Collective Social Intelligence, ESNAM-2013
- 76. What is Smart Data in the context of Disaster Management ACTIONABLE: Timely delivery of right resources and information to the right people at right location! 79
- 77. Join us for the Social Good! http://twitris.knoesis.org RT @OpOKRelief: Southgate Baptist Church on 4th Street in Moore has food, water, clothes, diapers, toys, and more. If you can't go,call 794 Text "FOOD" to 32333, REDCROSS to 90999, or STORM to 80888 to donate $10 in storm relief. #moore #oklahoma #disasterrelief #donate Want to help animals in #Oklahoma? @ASPCA tells how you can help: http://t.co/mt8l9PwzmO CITIZEN SENSORS RESPONSE TEAMS (including humanitarian org. and ‘pseudo’ responders) VICTIM SITE Coordination of emerging needs after a disaster Does anyone know where to send a check to donate to the tornado victims? Where do I go to help out for volunteer work around Moore? Anyone know? Anyone know where to donate to help the animals from the Oklahoma disaster? #oklah oma #dogs Matched Matched Matched Serving the need! If you would like to volunteer today, help is desperately needed in Shawnee. Call 273-5331 for more info http://www.slideshare.net/hemant_knoesis/cscw-2012-hemantpurohit-11531612 80 Purohit et al. Framework to Analyze Coordination in Crisis Response, 2012. Int’l Collaboration in-progress:
- 78. Smart Data from Twitris system for Disaster Response Coordination Which are the primary locations of power failure? Who are all the people to engage with for better information diffusion?Where are the charging stations to sustain communication? Smart data provides actionable information and improve decision making through semantic analysis of Big Data. Who are the resource seekers and suppliers? 81
- 79. Disaster Response Coordination: Twitris Summary for Actionable Nuggets 83 Important tags to summarize Big Data flow Related to Oklahoma tornado Images and Videos Related to Oklahoma tornado
- 80. 84 Disaster Response Coordination: Twitris Real-time information for needs Incoming Tweets with need types to give quick idea of what is needed and where currently #OKC Legends for Different needs #OKC (It is real-time widget for monitoring of needs, so will not be active after the event has passed) http://twitris.knoesis.org/oklahomatornado
- 81. 85 Disaster Response Coordination: Influencers to engage with for specific needs Influential users are respective needs and their interaction network on the right.
- 82. Really sparse Signal to Noise: • 2M tweets during the first week after #Oklahoma-tornado-2013 - 1.3% as the highly precise donation requests to help - 0.02% as the highly precise donation offers to help 86 • Anyone know how to get involved to help the tornado victims in Oklahoma??#tornado #oklahomacity (OFFER) • I want to donate to the Oklahoma cause shoes clothes even food if I can (OFFER) Disaster Response Coordination: Finding Actionable Nuggets for Responders to act • Text REDCROSS to 909-99 to donate to those impacted by the Moore tornado! http://t.co/oQMljkicPs (REQUEST) • Please donate to Oklahoma disaster relief efforts.: http://t.co/crRvLAaHtk (REQUEST) For responders, most important information is the scarcity and availability of resources, can we mine it via Social Media?
- 83. 87 Disaster Response Coordination: Engagement Interface for responders What-Where-How-Who-Why Coordination Influential users to engage with and resources for seekers/supplies at a location, at a timestamp Contextual Information for a chosen topical tags
- 84. • Illustrious scenario: #Oklahoma-tornado 2013 88 Disaster Response Coordination: Anecdote for the value of Smart Data FEMA asked us to quickly filter out gas-leak related data Mining the data for smart nuggets to inform FEMA (Timely needs) Engaged with the author of this information to confirm (Veracity) e.g., All gas leaks in #moore were capped and stopped by 11:30 last night (at 5/22/2013 1:41:37) Lot of tweets for ‘how to/where to’ assist (‘pseudo’ responders) e.g., I want to go to Oklahoma this weekend & do what i can to help those people with food,cloths & supplies,im in the feel of wanting to help ! :)
- 85. 89 Current Grid Conditions Renewable energy generation forecast Synchrophasor data Heat index, relative humidity Power consumption by consumers Big Data from Smart Grid Smart Data from Smart Grid What is the overall health of the Grid? What are the vulnerabilities for today? Red, yellow, and green indicate high, medium, and low risk allowing decision makers to focus on red & yellow lines Big Data vs. Smart Data in Smart Grids (Utilities perspective)
- 86. 90 Personal Schedule Big Data from Smart Grid & Smart Meters Smart Data from Smart Grid & Smart Meters Smart Meters Power Consumption Temperature, relative humidity Dynamic pricing information http://www.digikey.com/us/en/techzone/energy-harvesting/resources/articles/zigbees-smart-energy-20-profile.html Which devices are contributing to higher power bill? When should I operate the washer/dryer? Red, yellow, and green indicating high, medium, and low power consumption Recommendation algorithms will analyze these abstractions with domain knowledge Actions to optimize power bill will be recommended Big Data vs. Smart Data in Smart Grids (Consumer perspective)
- 87. Take Away • Data processing for Smart Grids/Utilities and Consumers is lot more than a Big Data processing problem • It is all about the human – not computing, not device: help them make better decisions, give actionable information – Computing for human experience • Whatever we do in Smart Data, focus on human-in-the-loop (empowering machine computing!): – Of Human, By Human, For Human – But in serving human needs, there is a lot more than what current big data analytics handle – variety, contextual, personalized, subjective, spanning data and knowledge across P- C-S dimensions 91
- 88. Acknowledgements • Kno.e.sis team • Funds: NSF, NIH, AFRL, Industry… • Note: • For images and sources, if not on slides, please see slide notes • Some images were taken from the Web Search results and all such images belong to their respective owners, we are grateful to the owners for usefulness of these images in our context. 92
- 89. • OpenSource: http://knoesis.org/opensource • Showcase: http://knoesis.org/showcase • Vision: http://knoesis.org/node/266 • Publications: http://knoesis.org/library 93 References and Further Readings
- 90. Amit Sheth’s PHD students Ashutosh Jadhav Hemant Purohit Vinh Nguyen Lu Chen Pavan Kapanipathi Pramod Anantharam Sujan Perera Alan Smith Pramod Koneru Maryam Panahiazar Sarasi Lalithsena Cory Henson Kalpa Gunaratna Delroy Cameron Sanjaya Wijeratne Wenbo Wang Kno.e.sis in 2012 = ~100 researchers (15 faculty, ~50 PhD students)
- 91. 95 thank you, and please visit us at http://knoesis.org Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled Computing Wright State University, Dayton, Ohio, USA Smart Data
Editor's Notes
- Major historical events are outlined here.Before Alternating Current (AC) was discovered, each device operating at different voltage required a separate power line! Also, the power generation has to be very close to the load. The power grid was visualized more like a distributed generators all throughout the grid.After AC, the power could be transmitted over long distances and a single voltage could be adapted to various devices operating at different voltages.The power grids are getting complex with many alternate sources of energy, varying consumer demand, increasing area of coverage, and increased loads due to electric vehicles. Smart Grid will generate lot of data and interpretation of this data is a challenging.
- AMI – Advanced Metering Infrastructure
- Source: http://techcrunch.com/2012/10/27/big-data-right-now-five-trendy-open-source-technologies
- http://radhakrishna.typepad.com/rks_musings/2013/04/big-data-review.htmlGoogle predicted the spread of flu in real time - after analyzing two datasets, a.) 50 million most common terms that Americans type, b.) data on the spread of seasonal flu from public health agency- tested a mammoth of 450 million different mathematical models to test the search terms, comparing their predictions against the actual flu cases- model was tested when H1N1 crisis struck in 2009 and gave more meaningful and valuable real time information than any public health official system (Big Data, Viktor Mayer-Schonberger and Kenneth Cukier, 2013)
- Better Algorithms Beat More Data — And Here’s Whyhttp://allthingsd.com/20121128/better-algorithms-beat-more-data-and-heres-why/Big Data Cannot Replace Human Judgmenthttp://www.matchcite.com/blog/blog/2012/july/big-data-cannot-replace-human-judgment.aspx**Comments about the articles
- Smart data makes sense out of big data – it provides value from harnessing the challenges posed by volume, velocity, variety and veracity of big data, to provide actionable information and improve decision making.
- - HUMAN CENTRIC!!
- Information is CREATED by human with the Machinery available – Wikipedia tool, sensors and social networksInformation is STORED in Man+Machine readable format, LODInformation is PROCESSED using the LOD and Human assisted Knowledge-basedHigher level abstraction on info is now consumed in many mechanistic ways (including GIS) to provide EXPERIENCE for humans
- All the data related to human activity, existence and experiencesMore on PCS Computing: http://wiki.knoesis.org/index.php/PCS
- Information is CREATED by human with the Machinery available – Wikipedia tool, sensors and social networksInformation is STORED in Man+Machine readable format, LODInformation is PROCESSED using the LOD and Human assisted Knowledge-basedHigher level abstraction on info is now consumed in many mechanistic ways (including GIS) to provide EXPERIENCE for humans Example of a human guided modeling and improved performancehttp://research.microsoft.com/en-us/um/people/akapoor/papers/IJCAI%202011a.pdf
- Also, we have weather application which performs abstraction on weather sensory observations to identify blizzard conditions (food for actions!!) :--20,000 weather stations (with ~5 sensors per station)-- Real-Time Feature Streams - live demo: http://knoesis1.wright.edu/EventStreams/ - video demo: https://skydrive.live.com/?cid=77950e284187e848&sc=photos&id=77950E284187E848%21276
- Lets find it..
- Starting slide Various Big data problems – Traditional examples vs what we are doing examples. Variety and Velocity than Volume. kHealth problem. People will be interested in Smart Data.Traditional ML techniques, High Performance Computing, Statistics. Human level of Abstraction is Smart data.
- http://www.nytimes.com/2003/09/13/national/13POWE.html?scp=2&sq=midwest%20iso&st=cse&pagewanted=1
- Such a blackout would cause billions of dollars in lost revenue. This particular blackout resulted in 6 billion dollar loss.Not only the consumers lose revenue even the power utility companies are fined almost a million dollars a day.
- Two of the problems are problems in “understanding” the system and the available data/observations. Lack of experience/training in deciphering this data had serious implications.The items 1 and 2 related to understanding real-world complexity by incorporating multi-modal and multi-sensory observations. Algorithms that can provide abstractions to decision makers for better comprehension of the situation.Providing abstractions in the context of the grid state continuously would lead to actionable information to decision makers.
- “Syncrophasors are like traffic cameras on a road traffic monitoring system”It provides high frequency (30 samples per second) updates on voltage, power flow in an electric network, phase difference, and many more electric quantities.https://www.selinc.com/SELUniversity/Courses/SYS/310/ http://green.blogs.nytimes.com/2010/04/01/for-the-smart-grid-a-synchophasor/?_r=0
- All the dollar amount here is per year.On an average, each household can save $369 / year which is $4 billion / year of reduced energy bills in the US=> Not all this is by doing nothing but just monitoring the usage and provide near real-time power consumption and electricity bill using smart meters
- Larry Smarr is a professor at the University of California, San DiegoAnd he was diagnosed with Crones DiseaseWhat’s interesting about this case is that Larry diagnosed himselfHe is a pioneer in the area of Quantified-Self, which uses sensors to monitor physiological symptomsThrough this process he discovered inflammation, which led him to discovery of Crones DiseaseThis type of self-tracking is becoming more and more common
- - what if we could automate this sense making ability?- and what if we could do this at scale?
- sense making based on human cognitive models
- perception cycle contains two primary phasesexplanationtranslating low-level signals into high-level abstractions inference to the best explanationdiscriminationfocusing attention on those properties that will help distinguish between multiple possible explanationsused to intelligently task sensors and collect additional observations (rather than brute force approach of blindly collecting all observations)
- perception cycle contains two primary phasesexplanationtranslating low-level signals into high-level abstractions inference to the best explanationdiscriminationfocusing attention on those properties that will help distinguish between multiple possible explanationsused to intelligently task sensors and collect additional observations (rather than brute force approach of blindly collecting all observations)
- A single-feature (disease) assumption means that all the observed properties (symptoms) must be explained by a single feature.i.e., this framework is not expressive enough to model comorbidity where there may be more than one feature (disease) co-existing For example, if there are two diseases causing disjoint symptoms, and all the symptoms of both the diseases are observed, then this framework will not be able to find the coverage and returns no diseases.
- perception cycle contains two primary phasesexplanationtranslating low-level signals into high-level abstractions inference to the best explanationdiscriminationfocusing attention on those properties that will help distinguish between multiple possible explanationsused to intelligently task sensors and collect additional observations (rather than brute force approach of blindly collecting all observations)
- - With this ability,many problems could be solved- For example: we could help solve health problems (before they become serious health problems) through monitoring symptoms and real-time sense making, acting as an early warning system to detect problematic health conditions
- Intelligence distributed at the edge of the networkRequires resource-constrained devices (mobile phones, gateway notes, etc.) to be able to utilize SW technologies
- Intelligence distributed at the edge of the networkRequires resource-constrained devices (mobile phones, gateway notes, etc.) to be able to utilize SW technologiesHenson et al. 'An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-Constrained Devices, ISWC 2012.
- compute machine perception inferences -- i.e., explanation and discrimination -- of high-complexity on a resource-constrained devices in milisecondsDifference between the other systems and what this system provides
- Intelligence at the age. Shipping computation and domain models to the edge (Distributed)
- Massive amount of data will be collected by sensors and mobile devices yet patients and doctors care about “actionable” information.This data has all the four Vs of big data and we used knowledge enabled techniques to transform it into valueIn the context of PD, we analyzed massive amount of sensor data collected by sensors on a smartphones to understand detection and characterization of PD severity.
- Main idea: Prior knowledge of PD was used to facilitate its detection from massive sensor data by reducing the search spaceDetails:Declarative knowledge of PD includes PD severity and their symptoms as shown in the logical rule aboveEach PD severity level is a conjunction of a set of PD symptomsEach symptom was mapped to its manifestation in sensor observationsThe availability of declarative knowledge significantly improved the analytics by aiding feature selection processThe graphs above contrasts the physical movements and voice of two control group members and two PD patients
- kHealth:http://www.youtube.com/watch?v=btnRi64hJp4EventShop:*http://www.slideshare.net/jain49/eventshop-120721, http://dl.acm.org/citation.cfm?id=2488175
- Much of the early work in Big data is being done with focusing on uni-directional among XYZ.
- Categorization of severity based on weather conditions. Actionable information is contextually dependent.
- Power Grid Context:Power blackout will result in critical needs such as food, water, charging stations, fuel, etc. which need to be addressed by the on-ground responders and remote responders. - 1 (+half) minuteAlright, so let’s motivate by this situation during emergency - Various actors: resource seekers, responder teams, resource providers at remote siteAnd - each of these actor groups have questions --- - needs - providers - responders: wondering!Here we have social network to connect these actors and bridge the gap for communication platformBut it’s potential use is yet to be realized for effective helpBecause.. (next slide)
- Talk about what kind of smart data we provide that helps the actions of crisis response coordination.
- Source: Purohit et. al 2013 (https://docs.google.com/a/knoesis.org/document/d/1aBJ2egHICUwaWxR8jOoTIUfEYj1QAnUt0q7haIKoYGY/edit# , http://www.knoesis.org/library/resource.php?id=1865)
- http://twitris.knoesis.org/oklahomatornado
- (It is real-time widget for monitoring of needs, so will not be active after the event has passed) http://twitris.knoesis.org/oklahomatornado
- Highly rich interface for response team
- Imagine if this is a power failure, we can get actionable insights in a timely manner
- More at: http://wiki.knoesis.org/index.php/PCSAnd http://knoesis.org/projects/ssw/