Curr Psychiatry Rep (2014) 16:523 DOI 10.1007/s11920-014-0523-3

PSYCHIATRY IN THE DIGITAL AGE (JS LUO, SECTION EDITOR)

New Measures of Mental State and Behavior Based on Data Collected From Sensors, Smartphones, and the Internet Tasha Glenn & Scott Monteith

# Springer Science+Business Media New York 2014

Abstract With the rapid and ubiquitous acceptance of new technologies, algorithms will be used to estimate new measures of mental state and behavior based on digital data. The algorithms will analyze data collected from sensors in smartphones and wearable technology, and data collected from Internet and smartphone usage and activities. In the future, new medical measures that assist with the screening, diagnosis, and monitoring of psychiatric disorders will be available despite unresolved reliability, usability, and privacy issues. At the same time, similar non-medical commercial measures of mental state are being developed primarily for targeted advertising. There are societal and ethical implications related to the use of these measures of mental state and behavior for both medical and non-medical purposes. Keywords Remote monitoring . E-mental health . Smartphone . Behavioral targeting . Emotion recognition

Introduction Today, psychiatrists rely on patient history and observation, objective data such as laboratory tests, and clinical judgment to diagnose mental illness and assess treatment. There is renewed emphasis on finding additional objective measures

This article is part of the Topical Collection on Psychiatry in the Digital Age T. Glenn ChronoRecord Association, Inc., Fullerton, CA 92834, USA S. Monteith (*) Michigan State University College of Human Medicine Traverse City Campus, 1400 Medical Campus Drive, Traverse City, MI 49684, USA e-mail: [email protected]

to assist with these challenges [1, 2]. A biomarker is an objective measure of a normal biological process, pathogenic processes, or response to a therapeutic intervention [3, 4]. Categories of potential psychiatric biomarkers include genetic, proteins or other molecules, or neuroimaging findings [1, 5, 6]. With the rapid emergence and acceptance of digital technologies, alternative measures of mental state and behavior are being developed for screening, diagnosis, and monitoring. In contrast to symptoms reported by patients directly to psychiatrists, these measures will be based on data collected from diverse sensors within smartphones, devices, and wearable technology, as well as data collected from smartphone and Internet usage and activities. Algorithms will be used to analyze the collected data to estimate mental health and behavior. In the future, the new measures of mental state and behavior will be available not only from regulated products with scientifically demonstrated clinical utility, but also from unregulated applications aimed at consumers and businesses. Regardless of whether these alternative measures are considered biomarkers, the new measures are coming and will impact psychiatry.

The Coming Age of Pervasive Healthcare Ubiquitous computing and ubiquitous communications are behind the upcoming measures of mental state and behavior. In 1991, Mark Weiser of Xerox PARC noted that “the most profound technologies are those that disappear” and predicted that computing elements would become so ubiquitous that no one would notice their presence [7]. We are quickly moving toward that vision [8] as about 5 % of man-made objects now contain embedded microprocessors [9]. Cell phones are viewed as such an essential part of life that many feel uncomfortable or inadequate without them [10, 11]. Ubiquitous communications, or the ability to communicate anytime and

523, Page 2 of 10

anywhere, was enabled by the recent expansion of wireless communications and ad hoc networking [12]. By 2020, Cisco Systems estimates that there will be 50 billion devices and objects connected to the Internet [13] for a world population estimated to be 7.5 billion [14]. One predicted result of this extraordinary growth in computing is pervasive healthcare or the access to healthcare applications for anyone, anytime [12, 15, 16]. A major component of pervasive healthcare will be remote disease monitoring, including mental state and behavior [15, 16], with the US market for remote disease monitoring projected to reach $22.5 billion by 2015 [17].

Potential Benefits for Psychiatry Although the majority of remote disease monitoring efforts are focused upon somatic illnesses, research into new measures and monitoring for psychiatry is increasing. New measures may help to prevent the frequent long time lag between symptom onset and diagnosis, which is of particular importance as earlier intervention may lead to a better outcome in schizophrenia and bipolar disorder [18–20]. There is a need for unobtrusive, easy-to-use, and inexpensive technologies to provide objective information about the symptoms that patients frequently do not report accurately. For example, about 40–70 % of patients do not adhere with treatment regimens for psychiatric medications [21, 22]. Psychiatric medications containing ingestible sensors will provide data on exactly when the medications were taken [23•]. There is only moderate correlation between self-reported and objective measures of sleep [24], and technologies will provide an understanding of sleep behaviors and patterns. Some objective measures will assist with diagnostic challenges due to overlapping symptoms and similar clinical presentations of many psychiatric disorders [25]. Other objective measures will help track the progression of gradually changing symptoms like cognition, and technology may help some patients with cognitive impairment to live independently [26, 27].

Curr Psychiatry Rep (2014) 16:523

Data Collected From Sensors The embedded computing element most critical to the expansion of pervasive healthcare computing is sensors. A wide range of intelligent, lightweight, ultra-low-power, and lowcost sensors are now available for medical monitoring [28, 29]. These sensors are embedded in smartphones; are wearable (embedded in clothing), implantable, and ingestible; and are placed in the home including in walls and floors [30–32]. A typical smartphone contains multiple sensors such as internal motion (accelerometer), ambient light, gyroscopic, gesture, magnetometer, temperature and humidity, and barometer. The communication interfaces commonly found on a smartphone are Wi-Fi, GPS, near field communications (NFC), Bluetooth, and infrared (IR) LED. Many additional sensors are available for physiological measurements [33]. Research into the use of physiologic sensors to measure emotion is increasing, including skin conductivity, heart rate, respiration, blood pressure, ECG, EEG, and EMG [34]. Data Collected From Internet and Smartphone Activities Measures of smartphone activities that are input into medical algorithms include details on incoming and outgoing call frequency, duration, and contacts [35]. Measures of Internet activities include details on search queries, pages visited, website type, advertising selected, and e-commerce history [36]. The content of user-created data such as emails, SMS (text messages), social media, or blogs can be analyzed [37•]. Additionally, metadata (information about information) are created for transactions from a smartphone or the Internet, which includes account numbers, login IDs, passwords, browser types, IP addresses, web pages visited, date, time, email sender and recipient, cookies, and device fingerprints [38].

Potential New Measures for Psychiatry

Basis for New Measures for Psychiatry Many measures are currently being investigated by medical researchers for use in psychiatry, including analysis of speech, facial expression, gesture, posture, movement, eye tracking, as well as Internet and smartphone behavior and activity. Two general classes of data are fundamental to these measures of mental state and behavior: (1) data collected from sensors and (2) data collected from Internet and smartphone activities. Algorithms are used to estimate the measures based on the collected data.

Table 1 provides examples of medical research for psychiatry. These measures have the potential to provide similar benefits to psychiatry that are associated with traditional biomarkers in relation to screening, diagnosis, and monitoring. Many of these systems are prototypes, and measures of emotion and behavior discovered in controlled medical or research conditions may not be suitable for natural settings. Future measures that are able to successfully integrate multiple modalities, such as speech and facial expression, are expected to be more precise at identifying emotion and behavior [39]. Additionally, combining the results of multiple measurements may increase diagnostic accuracy [40].

Curr Psychiatry Rep (2014) 16:523

Page 3 of 10, 523

Table 1 Examples of medical projects in development to measure mental health for psychiatry Study

Year Technology

Item measured

Chang [110] Dickerson [111]

2011 Cell phone or smartphone 2011 Multiple sensors and devices in home 2012 Headband system with embedded electrodes 2012 Wearable eye trackersa

Speech analysis Mental health monitor of emotion and stress Multiple, including speech, activity, Continuous monitoring for depression, PTSD sleep, weight, movement EEG and heart rate variability Monitor mental stress level

2010 Wearable vest with integrated sensorsb 2014 Smartphone acceleration and GPS sensors 2013 Ingestible sensor in tabletsb

Motor activity levels, using accelerometers and ECG Activity and mobility

Roh [112] Vidal [113] Minassian [114] Gruenerbl [115] Kane [23•] Kappeler-Setz [116]

2013 Wearable sock with sensor of skin conductance Jashinsky [117] 2014 Twitter De Choudhury [37•] 2013 Twitter Kotikalapudi [36] 2012 Internet usage Alvarez-Lozano [35] McIntyre [118] Matic [119] Miskelly [120]

2014 2009 2012 2005

Smartphone Video images RFID in clothing, video Smartphone with GPSa

a

Using commercially available products

b

Using commercially available, FDA-approved products

Eye tracking and eye movement

Medication ingestion Electrodermal activity (sweat secretion) Keywords and phrases in tweets Usage patterns, language Traffic volume, type of activities and sites Usage patterns, app selection Facial activity and expression Dressing ability Location

Aim

Mental health monitoring; discriminate among disorders Define activity patterns in mania and schizophrenia Mood state recognition (mania and depression in bipolar disorder) Monitor medication adherence in schizophrenia or bipolar disorder Monitor patients with bipolar disorder Surveillance for suicide risk Predict onset of depression Passive monitoring for depression Monitor mood in bipolar disorder Identify depression and anxiety Monitor cognitive skills Track wandering in dementia

Establish Clinical Utility

Patient Issues

As with any candidate biomarker, all new measures of mental state and behavior need to be thoroughly evaluated for use in routine clinical practice. To have clinical utility, the measure should add new information or clinical value to the information that is already available, and should be cost-effective [41]. Depending on the planned role of the measure such as screening, diagnosis, or monitoring, evaluation may include tests of reliability (repeatability in the same and different settings), validity (sensitivity, specificity, and predictive values), discrimination (receiver operator curves), association (relative risk), and cost-effectiveness analysis [41]. The process to establish clinical utility for biomarkers can be long and difficult [41, 42]. Initial studies of new measures often use small, highly selected samples, and results may not be generalizable to other clinical samples [42]. These small studies may also overrepresent the prevalence of the disease as compared with the general population, and results of validity analyses vary with the prevalence [43]. Confounders that may alter the results of the measurement need to be identified for each planned role of the measure [44]. Additionally, new diagnostic or predictive measures that are correlated with currently available measures may not provide additional clinical benefit [40, 45].

Patient Technical Skill Many people with mental illness may not have regular access to smartphones or the Internet or may lack the technical skills to use these effectively [46]. Usability of the monitoring device is critical, and the patient should receive detailed training. The product should be simple to operate and maintain, and designed for the technical skill level of an average patient who suffers from the targeted mental illness [12]. All home monitoring systems require some effort and discipline, and patient errors are a problem in general medical monitoring systems, such as inadequately recharging smartphone systems, wearing sensors incorrectly, and turning connections off [47•, 48, 49]. Patient Interest Level The successful use of remote monitoring devices is completely dependent on the cooperation of the patient. While some patients find that remote monitoring allows them to feel more control over their illness, others may resist even if they can easily use the technology. Some patients with chronic illness feel overwhelmed by their treatment burdens, do not want the

523, Page 4 of 10

added responsibility of self-monitoring [50], or prefer not to have a constant reminder that they are ill [51]. Some patients will become annoyed by the effort involved or lose interest and stop monitoring. Other patients may be unable to cope with the routine technological issues when experiencing symptoms [52].

Technology Issues Sensor Issues There are reliability problems with remote disease monitoring related to data quality and transmission delays [53]. The limiting characteristics of sensors include battery life, memory capacity, and difficulties with the patient-sensor interface [54, 55]. With wearable sensor systems, there are problems related to user comfort, sensor location in garments, quality of skin contact, and power consumption [53, 55]. Many wearable sensor systems were designed for use by the elderly, and the preferences of those with mental illnesses need to be considered. Unlike in a healthcare setting, remote measurements will not be repeated by a technician if a reading is invalid. In addition to sensor malfunctions and communication problems, real-world user activities can interfere with interpretation of collected sensor data. For example, motion artifacts can impact the quality of heart rate and respiration measurements [47•], sweat from exercise can require recalibration of body worn sensors [31], and ambient noise can impact smartphone voice analysis [56]. Walls and other obstructions in homes can interfere with wireless communications. Data collected from remote monitoring systems can be obtained at irregular time intervals, often in asynchronous bursts and in different contexts. The software must be able to differentiate hardware and software malfunctions from medical abnormalities. Current algorithms to analyze data from remote physiological sensors focus on anomaly detection, prediction, and decision making, especially when used for continuous monitoring, and many issues remain [57]. Internet and Smartphone Issues There are also concerns about the algorithms used to estimate mental state based on Internet and smartphone data and activities. People vary greatly in their mobile habits, and assessment algorithms based primarily on smartphone usage may not work well for light smartphone users [58]. Analysis of data from social media such as Twitter, Facebook, and blogs must find a few relevant messages among the hundreds of millions of messages sent daily [59]. The content of these messages and other social media is noisy unstructured text that varies tremendously in quality, and automatic text analysis

Curr Psychiatry Rep (2014) 16:523

algorithms are used to separate the high-quality content [60]. Since different methodologies are used to filter, remove noise, parse and identify emotions, and create user profiles from social media [37•, 59, 61, 62•], inconsistent results would be expected among vendors. Additionally, algorithms combine various additional data elements with the social media to improve results. Many accounts on social media are fake, including an estimated 67 million accounts on Facebook [63] and 20 million accounts on Twitter in 2013 [64]. Finally, not all demographic groups use social media with the same frequency. In a national US survey in 2013, 67 % of all adult Internet users accessed any social networking site, with those aged 18–29 the most likely to do so (83 %) [65]. Context Awareness and Data Interpretation A critical issue in the interpretation of data for monitoring of mental health is the user’s context. The implicit situational information found in human-human interactions is not automatically present in human-computer interactions. This contextual awareness can lead to a change in interpretation of data, such that the relevancy and importance of the information depends on the situation, and should be included in all algorithms [66–68]. As an example, one software framework to identify context establishes three entities (people, places, and things) that are described by four categories (identity, location, status or activity, and time) [68]. Both indoor and outdoor location sensings are now possible using smartphones, wristwatches, and sensors [69]. With contextaware software, the presentation of information, execution of services, and tagging of data elements will vary with the context [68]. While context awareness should improve the quality and clinical usefulness of mental health monitoring, the required convergence of simultaneous measurements such as of activity and vital signs will also increase both the volume of data collected and the processing requirements [66].

Security and Privacy The security and privacy aspects of systems for monitoring mental state and behavior are critical for public acceptance and an area of considerable concern [70, 71]. The medical data must be accurate and reliable, accessible only by those authorized to do so, and comply with all state and federal privacy regulations including HIPAA. In addition to the complex security challenges associated with medical data, there are many inherent security risks with mobile users, wireless networks, shared resources, and shared control of monitoring systems [47•, 53, 70, 72]. Most breaches of medical data involve mobile devices [73], and there is a very high risk for medical data on smartphones or laptops to reach unauthorized

Curr Psychiatry Rep (2014) 16:523

users. In 2013, 3.1 million smartphones were stolen, and only 54 % of smartphone users either set a PIN code to lock the screen or used encryption [74]. In a 2008 study of 106 US airports, 12,255 laptops were lost per week, and 55 % of travelers used no security protections [75]. Monitoring systems must also be designed and operated to address the threat of intentional attacks [47•].

Page 5 of 10, 523

Expression Recognition Toolkit (CERT) for classifying emotions from facial expressions [91]. Some products to analyze human emotion are currently on the marketplace. Emotient and Affdex based on facial expression analysis, and Moodies based on voice analysis, are currently being sold to businesses to analyze customer emotional response [92–94]. NeuroSky and Emotiv, based on EEG, are currently being sold to consumers to optimize brain fitness [95, 96].

Large Influx of New Data The information flowing from the sensors has the potential to create a massive volume of data [71]. The scalability of the monitoring systems in terms of both processing power and volume of data to be stored and analyzed will become a critical issue as large numbers of patients start to use these systems [31, 53]. Systems that function in clinical environments will require an IT organization to receive and process the patient data and transfer the results to the psychiatrists.

Measures for Non-medical Commercial Uses There is widespread non-medical commercial research into computer recognition of human emotion and behavior. The future of human-computer interaction envisions flexible computers that are capable of identifying and tracking people and then adapting and responding to user moods, preferences, and intentions [76–79]. Commercial research to recognize emotion and behavior is focused on developing subtle, continuous, real-time, and context-specific interpretations of human affective displays and on combining multiple modalities to improve results [79, 80, 81•, 82]. A range of commercial applications for computer recognition of human emotion and behavior are envisioned. These include improving the effectiveness of targeted advertising [83–85], improving the humancomputer interface for standard devices such as home appliances [77, 86], developing robotic devices that can have human-like interactions [79, 87], expanding the use of computers in tedious but attention-requiring tasks such as surveillance work [76, 82, 87], recognizing human emotions to improve safety such as when driving [88], for intelligent tutors in education [89] and for entertainment. Table 2 provides examples of non-medical commercial emotion- and behavior-related research, although much of this is proprietary. Other technologies being developed include EmotionML, a proposed standard for an emotion markup language to allow software to respond to the detected emotional state of the user from the World Wide Web consortium (W3C) [90], and the Computer

Ethical and Societal Issues Concerns With Medical Measures for Psychiatry There are many ethical and societal questions associated with the medical measurements of mental state and behavior. Psychiatric profiling will identify some individuals as being at high risk and requiring special intervention. This is especially troubling when children are involved, as a high-risk label may shape individual self-perception and societal attitudes about a child [1]. A label of being at high risk for mental illness could also lead to discrimination by employers, lenders, insurance companies, and even volunteer organizations [97]. Patients using remote monitoring should have a clear understanding of how to obtain help in emergency situations, and of the standard turnaround time for a response to incoming messages or data [98]. Some patients with a remote monitor may incorrectly expect an immediate response to their digital transmissions whenever they are in trouble [98]. If a patient’s daily activities are being monitored, the patients should agree to which specific activities are included [99]. There are additional ethical considerations regarding the use of surveillance technologies for individuals with cognitive impairment [100]. Concerns With Measures for Non-medical Commercial Use There are many areas of concern related to the non-medical commercial estimates of mental state and behavior. Physicians are obligated to act in the best interests of their patients, but commercial application vendors and data brokers are not involved in patient care and have no such obligation. The vendors of most applications and devices sold to consumers do not need to demonstrate efficacy or safety [101], as these products are not regulated by the FDA [102]. Many consumers may not be aware that data from applications not directly accessed by medical providers are also outside of the scope of HIPAA protections and may be sold to third parties [103]. Users may also not understand the sensitivity of some data, such as from the commercial EEG products, which can be used as a unique personal identifier [104].

523, Page 6 of 10

Curr Psychiatry Rep (2014) 16:523

Table 2 Examples of non-medical commercial projects in development to measure mood, emotion, and behavior Study (patent application)

Year Technology

LiKamWa [121] (Microsoft, etc.) 2011 Smartphone

Item measured

Aim Measure mood

Chittaranjan [122] (Nokia, etc.)

2011 Smartphone

Lu [123•] (Intel, etc.) Konopnicki [62] (IBM) Pandey [124] (Yahoo!, Google, etc.) Aly [125] (Yahoo!)

2012 Smartphone 2013 Social media 2011 Internet 2012 Internet

SMS (text messaging), calls, email, application usage, web browsing, location Call logs, SMS, Bluetooth scans, application usage Stress in voice Social media data Online queries, browsing, ad clicks, history Internet activities

Gallager [126] (Raytheon)

2013 Internet

Social networking and smartphones

Taigman [127•] (Facebook, etc.) 2014 Internet Sahami Shirazi [128] 2013 Smartphone (Google, etc.) Samsung [129] 2012 Smartphone Apple [130]

Microsoft [131] Microsoft [85] Yahoo [132] Yahoo [133] Dell [134]

Facial recognition Interaction with mobile phone app

Classify social network user emotional states 2014 Hardware/software Voice, facial expression, physiologic, devices Internet activity, compared to a user baseline profile 2014 Microcontroller Biometric data on emotional state such as heart rate, skin conductance 2012 Xbox Scan email, messages, Kinect movement sensors, facial expressions 2013 Internet Generates user profile with mood gradient, friends preferences 2014 Mobile devices, Voice analysis, speech, tone computers 2014 Computer with EEG, heart rate, other physiologic sensors sensors

Commercial organizations that monitor people’s everyday health-related habits, along with their daily activities, can combine this with other data obtained from data brokers to estimate mental state and behavior. Initially, the most frequent use of the estimates of emotional state and behaviors may be to profile individuals for sales purposes [105]. However, there are many possible implications if commercial organizations define and detect profiles associated with mental illnesses. Without their knowledge or active participation, people could be branded and blacklisted by an algorithm. Targeted advertising for health-related products may influence medical choices [106] and contain incorrect content that may mislead, offer false hope of cure [107], or result in delays in seeking established treatment. Furthermore, the algorithms used by the commercial companies for health analyses are not published, may be incorrect, and cannot be duplicated [108]. If these algorithms are considered trade secrets, legal protections will suppress this information from the public, even though building on the published findings of

Classify personality traits Classify stress in real-time conversation Define user profiles for targeted advertising Improve user profiles for behavioral targeting Update profiles daily for behavioral targeting Track movement and predict future behavior Human accuracy in facial recognition Track sleep behavior Control interactions between users related to emotional states Infer mood and deliver mood-based content

Mood-actuated device reacts to user mood Target ads based on emotional state Target items (song playlist, movies) to mood Stream content using voice-based mood analysis Determine mood and emotion for use in education, gaming

others is fundamental to scientific inquiry [109]. Public discussions of commercial use of mood, emotion, and behavioral data are required.

Conclusions New measures of mental state and behavior that are generated using technology and analytics are coming to psychiatry. In the future, clinically useful measures will help with the screening, diagnosis, and treatment of psychiatric disorders, and there will be continued advances in this emerging field. There are many technical issues to resolve, primarily relating to reliability, usability, privacy, and clinical utility of the new measures. At the same time, the use of non-medical commercial products based on similar measures will become pervasive. Society must address the ethical issues associated with measures of mental state and behavior for medical and nonmedical use.

Curr Psychiatry Rep (2014) 16:523 Compliance With Ethics Guidelines Conflict of Interest Scott Monteith declares no conflict of interest. Tasha Glenn shares a patent for ChronoRecord software but does not receive any financial compensation from the ChronoRecord Association, a 501(c)(3) nonprofit organization.

Page 7 of 10, 523 16.

17.

18. Human and Animal Rights and Informed Consent This article does not contain any studies with human or animal subjects performed by any authors.

19.

20.

References

21.

Papers of particular interest, published recently, have been highlighted as: • Of importance

22.

1. 2.

3.

4.

5.

6.

7. 8. 9.

10.

11. 12.

13. 14.

15.

Singh I, Rose N. Biomarkers in psychiatry. Nature. 2009;460: 202–7. Kapur S, Phillips AG, Insel TR. Why has it taken so long for biological psychiatry to develop clinical tests and what to do about it? Mol Psychiatry. 2012;17:1174–9. Biomarkers Definitions Working Group. Biomarkers and surrogate endpoints: preferred definitions and conceptual framework. Clin Pharmacol Ther. 2001;69:89–95. Lesko LJ, Atkinson Jr AJ. Use of biomarkers and surrogate endpoints in drug development and regulatory decision making: criteria, validation, strategies. Annu Rev Pharmacol Toxicol. 2001;41:347– 66. Lakhan SE, Kramer A. Schizophrenia genomics and proteomics: are we any closer to biomarker discovery? Behav Brain Funct. 2009;5:2. McIntyre RS, Cha DS, Jerrell JM, et al. Advancing biomarker research: utilizing ‘Big Data’ approaches for the characterization and prevention of bipolar disorder. Bipolar Disord. 2014;16(5):531–47. Weiser M. The computer for the 21st century. Sci Am. 1991;265: 94–104. Gershenfeld N, Krikorian R, Cohen D. The Internet of things. Sci Am. 2004;291:76. Vinge V. Who’s afraid of first movers? Singularity Summit 2012. https://docs.google.com/document/d/1W7Vs8e3MNwjU_ 1ZZZqmq0O-dFKCwD4VRyww-qOvDdRU/pub. Accessed 9 Jul 2014. Venta L, Isomursu M, Ahtinen A et al. “My phone is a part of my soul”—how people bond with their mobile phones. In Mobile Ubiquitous Computing, Systems, Services and Technologies, 2008. UBICOMM ’08. The second international conference on IEEE, 2008; 311–17 Srivastava L. Mobile phones and the evolution of social behaviour. Behav Inf Technol. 2005;24:111–29. Korhonen I, Bardram JE. Guest editorial: introduction to the special section on pervasive healthcare. IEEE Trans Inf Technol Biomed. 2004;8:229–34. Cisco Systems. Internet of things. http://www.cisco.com/web/ solutions/trends/iot/indepth.html. Accessed 9 Jul 2014. US Census. International data base. World population: 1950– 2050. 2011 http://www.census.gov/population/international/data/ idb/worldpopgraph.php. Accessed 9 Jul 2014. Varshney U. Pervasive healthcare and wireless health monitoring. Mob Netw Appl. 2007;12:113–27.

23.•

24.

25. 26.

27. 28.

29.

30. 31. 32.

33.

34.

35.

36.

37.•

Orwat C, Graefe A, Faulwasser T. Towards pervasive computing in health care—a literature review. BMC Med Inform Decis Mak. 2008;8:26. Kalorama Information. Remote and wireless patient monitoring markets. 2011. http://www.kaloramainformation.com/RemoteWireless-Patient-6487095/. Accessed 9 Jul 2014. Marshall M, Rathbone J. Early intervention for psychosis. Schizophr Bull. 2011;37:1111–4. Berk M, Brnabic A, Dodd S, et al. Does stage of illness impact treatment response in bipolar disorder? Empirical treatment data and their implication for the staging model and early intervention. Bipolar Disord. 2011;13:87–98. Killackey E, Yung AR. Effectiveness of early intervention in psychosis. Curr Opin Psychiatry. 2007;20:121–5. Bulloch AG, Patten SB. Non-adherence with psychotropic medications in the general population. Soc Psychiatry Psychiatr Epidemiol. 2010;45:47–56. Cramer JA, Rosenheck R. Compliance with medication regimens for mental and physical disorders. Psychiatr Serv. 1998;49:196–201. Kane JM, Perlis RH, DiCarlo LA, et al. First experience with a wireless system incorporating physiologic assessments and direct confirmation of digital tablet ingestions in ambulatory patients with schizophrenia or bipolar disorder. J Clin Psychiatry. 2013;74:e533–40. Pilot study using sensors embedded in placebo tablets. Lauderdale DS, Knutson KL, Yan LL, et al. Self-reported and measured sleep duration: how similar are they? Epidemiology. 2008;19:838–45. Cook IA. Biomarkers in psychiatry: potentials, pitfalls, and pragmatics. Prim Psychiatry. 2008;15:54. Wood A, Stankovic JA, Virone G, et al. Context-aware wireless sensor networks for assisted living and residential monitoring. IEEE Netw. 2008;22:26–33. Braddock D, Rizzolo MC, Thompson M, et al. Emerging technologies and cognitive disability. J Spec Educ Technol. 2004;19:49–56. Milenković A, Otto C, Jovanov E. Wireless sensor networks for personal health monitoring: issues and an implementation. Comput Commun. 2006;29:2521–33. Darwish A, Hassanien AE. Wearable and implantable wireless sensor network solutions for healthcare monitoring. Sensors. 2011;11:5561–95. Pentland A. Healthwear: medical technology becomes wearable. Computer. 2004;37:42–9. Alemdar H, Ersoy C. Wireless sensor networks for healthcare: a survey. Comput Netw. 2010;54:2688–710. Yoo H-J. Your heart on your sleeve: advances in textile-based electronics are weaving computers right into the clothes we wear. IEEE Solid-State Circ Mag. 2013;5:59–70. Lowe SA, Ólaighin G. Monitoring human health behaviour in one’s living environment: a technological review. Med Eng Phys. 2014;36:147–68. Jerritta S, Murugappan M, Nagarajan R, et al. Physiological signals based human emotion recognition: a review. In Signal Processing and its Applications (CSPA), 2011 I.E. 7th International Colloquium on. IEEE. 2011:410–15 Alvarez-Lozano J, Frost M, Osmani V, et al. Tell me your apps and I will tell you your mood: correlation of apps usage with bipolar disorder state. ACM proceedings of 7th international conference on pervasive technologies related to assistive environments. 2014. Kotikalapudi R, Chellappan S, Montgomery F, et al. Associating Internet usage with depressive behavior among college students. IEEE Technol Soc Mag. 2012;31:73–80. De Choudhury M, Counts S, Horvitz E, et al. Predicting depression via social media. 2013. Microsoft Research. http://research. microsoft.com/apps/pubs/default.aspx?id=192721. Accessed 9 Jul 2014. Example of analyzing social media to predict mood.

523, Page 8 of 10 38.

39. 40. 41.

42. 43.

44. 45. 46.

47.• 48.

49.

50. 51. 52.

53. 54. 55.

56.

57.

58.

59.

60.

61.

Guardian. A guardian guide to your metadata. 2013 http://www. theguardian.com/technology/interactive/2013/jun/12/what-ismetadata-nsa-surveillance%23meta=0000000. Accessed 9 Jul 2014. Picard RW. Affective computing. Cambridge: MIT; 1997. Pepe MS, Thompson ML. Combining diagnostic test results to increase accuracy. Biostatistics. 2000;1:123–40. Pletcher MJ, Pignone M. Evaluating the clinical utility of a biomarker: a review of methods for estimating health impact. Circulation. 2011;123:1116–24. Perlis RH. Translating biomarkers to clinical practice. Mol Psychiatry. 2011;16:1076–87. Brenner H, Gefeller O. Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence. Stat Med. 1997;16:981–91. Mayeux R. Biomarkers: potential uses and limitations. NeuroRx. 2004;1:182–8. Gerszten RE, Wang TJ. The search for new cardiovascular biomarkers. Nature. 2008;451:949–52. Monteith S, Glenn T, Bauer M. Searching the internet for health information about bipolar disorder: some cautionary issues. Int J Bipolar Disord. 2013;1:22. Ko JG, Lu C, Srivastava MB, et al. Wireless sensor networks for healthcare. Proc IEEE. 2010;98:1947–60. Clear overview article. Chan M, Estève D, Fourniols JY, et al. Smart wearable systems: current status and future challenges. Artif Intell Med. 2012;56: 137–56. Healey J, Nachman L, Subramanian S, et al. Out of the lab and into the fray: towards modeling emotion in everyday life. In Pervasive computing. Springer, Berlin, 2010:156–73. May C, Montori VM, Mair FS. We need minimally disruptive medicine. BMJ. 2009;339:b2803. Mol A. Living with diabetes: care beyond choice and control. Lancet. 2009;373:1756–7. Luxton DD, McCann RA, Bush NE, et al. mHealth for mental health: integrating smartphone technology in behavioral healthcare. Prof Psychol Res Pr. 2011;42:505. Baig MM, Gholamhosseini H. Smart health monitoring systems: an overview of design and modeling. J Med Syst. 2013;37:9898. Jurik AD, Weaver AC. Remote medical monitoring. Computer. 2008;41:96–9. McAdams E, Krupaviciute A, Gehin C, Grenier E, et al. Wearable sensor systems: the challenges. In Engineering in Medicine and Biology Society, EMBC, 2011 Annual international conference of the IEEE. IEEE, 2011:3648–51 Lane ND, Mohammod M, Lin M, et al. Bewell: a smartphone application to monitor, model and promote wellbeing. In 5th international ICST conference on pervasive computing technologies for healthcare. 2011: 23–6. Banaee H, Ahmed MU, Loutfi A. Data mining for wearable sensors in health monitoring systems: a review of recent trends and challenges. Sensors (Basel). 2013;13:17472–500. Ma Y, Xu B, Bai Y, et al. Daily mood assessment based on mobile phone sensing. In Wearable and Implantable Body Sensor Networks (BSN), 2012 Ninth international conference on. IEEE. 2012: 142–7 Alexe B, Hernandez MA, Hildrum KW, et al. Surfacing timecritical insights from social media. In Proceedings of the 2012 ACM SIGMOD international conference on management of data. ACM, 2012: 657–60. Agichtein E, Castillo C, Donato D, et al. Finding high-quality content in social media. In Proceedings of the 2008 International conference on web search and data mining. ACM, 2008:183–194. Wang W, Chen L, Thirunarayan K, et al. Harnessing Twitter ‘big data’ for automatic emotion identification. In Privacy, Security, Risk and Trust (PASSAT), 2012 International conference on and

Curr Psychiatry Rep (2014) 16:523

62.•

63.

64.

65.

66.

67. 68.

69.

70. 71.

72.

73.

74.

75.

76.

77. 78.

79.

80. 81.•

2012 International conference on social computing (SocialCom). IEEE. 2012:587–592 Konopnicki D, Shmueli-Scheuer M, Cohen D, et al. A statistical approach to mining customers’ conversational data from social media. IBM J Res Dev. 2013;57:14–1. Example of creating user profiles from social media. Munson L. Facebook: At least 67 million accounts are fake. 2014 http://nakedsecurity.sophos.com/2014/02/10/facebook-at-least67-million-accounts-are-fake/. Accessed 9 Jul 2014. Perlroth N. Fake Twitter followers become multimillion-dollar business. New York Times. 2013 http://bits.blogs.nytimes.com/ 2013/04/05/fake-twitter-followers-becomes-multimillion-dollarbusiness/?_php=true&_type=blogs&ref=technology&_r=0. Accessed 9 Jul 2014. Duggan M. The demographics of social media users. 2013. http:// www.pewinternet.org/files/old-media/Files/Reports/2013/PIP_ SocialMediaUsers.pdf. Accessed 9 Jul 2014. Viswanathan H, Chen B, Pompili D. Research challenges in computation, communication, and context awareness for ubiquitous healthcare. IEEE Commun Mag. 2012;50:92–9. Baldauf M, Dustdar S, Rosenberg F. A survey on context-aware systems. Int J Ad Hoc Ubiquit Comput. 2007;2:263–77. Dey AK, Abowd GD, Salber D. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Human-computer interaction. 2001;16:97–166. Welbourne E, Lester J, LaMarca A et al. Mobile context inference using low-cost sensors. In Location-and Context-Awareness. Springer, Berlin, 2005:254–63. Li M, Lou W, Ren K. Data security and privacy in wireless body area networks. Wireless Communications, IEEE. 2010;17:51–8. Sarasohn-Kahn J. Making sense of sensors: how new technologies can change patient care. California Healthcare Foundation. 2013. http://www.chcf.org/publications/2013/02/making-sense-sensors. Accessed 9 Jul 2014 Halperin D, Kohno T, Heydt-Benjamin TS, et al. Security and privacy for implantable medical devices. IEEE Pervasive Comput. 2008;7:30–9. Redspin 2014. Breach report 2013: protected health information (PHI) http://www.redspin.com/resources/whitepapers-datasheets/ Request-2013-Breach-Report-Protected-Health-Information-PHIRedspin.php. Accessed 9 Jul 2014. Consumer Reports. Smart phone thefts rose to 3.1 million last y e a r, C o n s u m e r R e p o r t s f i n d s . 2 0 1 4 . h t t p : / / w w w. consumerreports.org/cro/news/2014/04/smart-phone-thefts-roseto-3-1-million-last-year/index.htm#. Accessed 9 Jul 2014. Ponemon Institute. Airport insecurity: the case of missing & lost laptops. 2008. http://www.dell.com/downloads/global/services/ dell_lost_laptop_study.pdf. Accessed 9 Jul 2014. Pantic M, Sebe N, Cohn JF et al. Affective multimodal human-computer interaction. In Proceedings of the 13th annual ACM international conference on multimedia. ACM, 2005; 669–76. Schmidt A, Pfleging B, Alt F, et al. Interacting with 21st-century computers. IEEE Pervasive Comput. 2012;11:22–31. Pentland A. Looking at people: sensing for ubiquitous and wearable computing. IEEE Trans Pattern Anal Mach Intell. 2000;22: 107–19. Sebe N, Cohen I, Gevers T, et al. Multimodal approaches for emotion recognition: a survey. In Electronic Imaging 2005, International Society for Optics and Photonics, 2005: 56–67 Gunes H, Pantic M. Automatic, dimensional and continuous emotion recognition. Int J Synth Emotions (IJSE). 2010;1:68–99. Zeng Z, Pantic M, Roisman GI, et al. A survey of affect recognition methods: audio, visual, and spontaneous expressions. A survey of affect recognition methods: audio, visual, and spontaneous

Curr Psychiatry Rep (2014) 16:523

82. 83.

84.

85.

86.

87. 88.

89. 90.

91.

92. 93. 94. 95. 96.

97. 98.

99.

100.

101.

102.

103.

expressions. IEEE Trans Pattern Anal Mach Intell. 2009;31:39– 58. Clear survey article. Aggarwal JK, Ryoo MS. Human activity analysis: a review. ACM Comput Surv. 2011;43:16. Network Advertising Initiative. Study finds behaviorally-targeted ads more than twice as valuable, twice as effective as non-targeted online ads. 2010 http://www.networkadvertising.org/pdfs/NAI_ Beales_Release.pdf. Accessed 9 Jul 2014 Yan J, Liu N, Wang G, et al. How much can behavioral targeting help online advertising? In: Proceedings of the 18th international conference on world wide web. ACM. 2009:261–270. Microsoft, owner. Chung WJ, Patwa P, Markov MM, inventors. Targeting advertisements based on emotion. United States Patent Application 20120143693 A1. 6/7/12 Meyer S, Rakotonirainy A. A survey of research on context-aware homes. In Proceedings of the Australasian information security workshop conference on ACSW frontiers. Australian Computer Society, Inc., 2003; 21:159–68. Chui M, Löffler M, Roberts R. The internet of things. McKinsey Quarterly 2. 2010: 1–9. Lisetti CL, Nasoz F. Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP J Appl Signal Processing. 2004: 1672–87. D’Mello S, Picard R, Graesser A. Towards an affect-sensitive autotutor. IEEE Intell Syst. 2007;22:53–61. Schröder M, Baggia P, Burkhardt F, et al. EmotionML—an upcoming standard for representing emotions and related states. In Affective Computing and Intelligent Interaction. Springer, Berlin. 2011:316–25. Littlewort G, Whitehill J, Wu T et al. The computer expression recognition toolbox (CERT). In Automatic Face & Gesture Recognition and Workshops (FG 2011). IEEE International conference on. IEEE. 2011:298–305 Emotient. www.emotient.com. Accessed 9 Jul 2014. Affdex. http://www.affdex.com/. Accessed 9 Jul 2014. Moodies. http://www.beyondverbal.com/emotions-analytics/whyis-it-so-special/. Accessed 9 Jul 2014. NeuroSky. http://neurosky.com/. Accessed 9 Jul 2014. Emotiv Insight. https://www.kickstarter.com/projects/tanttle/ emotiv-insight-optimize-your-brain-fitness-and-per. Accessed 9 Jul 2014. Overton SL, Medina SL. The stigma of mental illness. J Couns Dev. 2008;86:143–51. Luxton DD, June JD, Kinn JT. Technology-based suicide prevention: current applications and future directions. Telemed J E Health. 2011;17:50–4. Glascock AP, Kutzik DM. Behavioral telemedicine: a new approach to the continuous nonintrusive monitoring of activities of daily living. Telemed J. 2000;6:33–44. Niemeijer AR, Frederiks BJ, Riphagen II, et al. Ethical and practical concerns of surveillance technologies in residential care for people with dementia or intellectual disabilities: an overview of the literature. Int Psychogeriatr. 2010;22:1129–42. Sharpe R. Many health apps are based on flimsy science at best, and they often do not work. The Washington Post. 2012. http:// www.washingtonpost.com/national/health-science/many-healthapps-are-based-on-flimsy-science-at-best-and-they-often-do-notwork/2012/11/12/11f2eb1e-0e37-11e2-bd1a-b868e65d57eb_ story.html. Accessed 9 Jul 2014. FDA. Mobile medical applications. 2014. http://www.fda.gov/ MedicalDevices/ProductsandMedicalProcedures/ ConnectedHealth/MobileMedicalApplications/default.htm?utm_ source=twitterfeed&utm_medium=twitter. Accessed 9 Jul 2014. Glenn T, Monteith S. Privacy in the digital world: medical and health data outside of HIPAA protections. Curr Psychiatry Rep. 2014;16(11):494.

Page 9 of 10, 523 104.

105.

106.

107.

108. 109.

110.

111.

112.

113. 114.

115.

116.

117. 118.

119.

120.

121.

122.

123.•

Stopczynski A, Greenwood D, Hansen LK. et al. Privacy for personal neuroinformatics. 2014. http://arxiv.org/abs/1403.2745. Accessed 9 Jul 2014. Labrecque LI, Markos E, Milne GR. Online personal branding: processes, challenges, and implications. J Interact Mark. 2011;25: 37–50. Stark M, Fins JJ. Engineering medical decisions: computer algorithms and the manipulation of choice. Camb Q Healthc Ethics. 2013;22:373–81. Petersen A, Seear K. Technologies of hope: techniques of the online advertising of stem cell treatments. New Genet Soc. 2011;30:329–46. Lazer D, Kennedy R, King G, et al. Big data. The parable of Google Flu: traps in big data analysis. Science. 2014;343:1203–5. Simpson MP. The future of innovation-trade secrets, property rights, and protectionism-an age-old tale. Brook L Rev. 2005;70: 1121. Chang KH, Fisher D, Canny J, et al. How’s my mood and stress?: an efficient speech analysis library for unobtrusive monitoring on mobile phones. In Proceedings of the 6th international conference on body area networks. ICST (Institute for Computer Sciences, SocialInformatics and Telecommunications Engineering), 2011; 71–7. Dickerson RF, Gorlin EI, Stankovic JA. Empath: a continuous remote emotional health monitoring system for depressive illness. In Proceedings of the 2nd conference on wireless health. ACM. 2011: 5. Roh T, Bong K, Hong S, et al. Wearable mental-health monitoring platform with independent component analysis and nonlinear chaotic analysis. Conf Proc IEEE Eng Med Biol Soc. 2012;2012:4541–4. Vidal M, Turner J, Bulling A, et al. Wearable eye tracking for mental health monitoring. Comput Commun. 2012;35:1306–11. Minassian A, Henry BL, Geyer MA, et al. The quantitative assessment of motor activity in mania and schizophrenia. J Affect Disord. 2010;120:200–6. Gruenerbl A, Osmani V, Bahle G, et al. Using smart phone mobility traces for the diagnosis of depressive and manic episodes in bipolar patients. In Proceedings of the 5th augmented human international conference. ACM, 2014:38. Kappeler-Setz C, Gravenhorst F, Schumm J, et al. Towards long term monitoring of electrodermal activity in daily life. Pers Ubiquitous Comput. 2013;17:261–71. Jashinsky J, Burton SH, Hanson CL, et al. Tracking suicide risk factors through Twitter in the US. Crisis. 2014;35:51–9. McIntyre G, Gocke R, Hyett M, et al. An approach for automatically measuring facial activity in depressed subjects. In Affective computing and intelligent interaction and workshops, 2009. ACII 2009. 3rd International conference on, IEEE, 2009:1–8 Matic A, Mehta P, Rehg JM, et al. Monitoring dressing activity failures through RFID and video. Methods Inf Med. 2012;51:45– 54. Miskelly F. Electronic tracking of patients with dementia and wandering using mobile phone technology. Age Ageing. 2005;34:497–9. LiKamWa R, Liu Y, Lane ND, et al. Can your smartphone infer your mood?. In Proc. PhoneSense Workshop. 2011. Microsoft Research. http://research.microsoft.com/en-us/projects/ moodsense/. Accessed 9 Jul 2014. Chittaranjan G, Blom J, Gatica-Perez D. Who’s who with big-five: analyzing and classifying personality traits with smartphones. In Wearable Computers (ISWC), 2011 15th annual international symposium on, IEEE. Nokia Research Center. https://research.nokia.com/ publications?search=Gokul+Chittaranjan. Accessed 9 Jul 2014 Lu H, Frauendorfer D, Rabbi M, et al. StressSense: detecting stress in unconstrained acoustic environments using smartphones. In Proceedings of the 2012 ACM conference on ubiquitous

523, Page 10 of 10

124.

125.

126.

127.•

128.

computing. ACM. 2012:351–360. Example of voice analysis using smartphones. Pandey S, Aly M, Bagherjeiran A, et al. Learning to target: what works for behavioral targeting. In Proceedings of the 20th ACM international conference on Information and knowledge management. ACM. 2011:1805–14. Aly M, Hatch A, Josifovski V, et al. Web-scale user modeling for targeting. In Proceedings of the 21st international conference companion on World Wide Web. ACM. 2012:3–12. Gallagher R. Software that tracks people on social media created by defence firm. The Guardian. 2013. http://www.theguardian. com/world/2013/feb/10/software-tracks-social-media-defence? intcmp=239. Accessed 9 Jul 2014 Taigman Y, Yang M, Ranzato M, et al. DeepFace: closing the gap to human-level performance in face verification. 2014. https:// www.facebook.com/publications/546316888800776/. Accessed 9 Jul 2014. Advances in facial recognition. Sahami Shirazi A, Clawson J, Hassanpour Y, et al. Already up? using mobile phones to track & share sleep behavior. Int J Hum Comput Stud. 2013;71:878–88.

Curr Psychiatry Rep (2014) 16:523 129.

130.

131.

132.

133.

134.

Samsung, owner. Lee H, inventor. Apparatus and method for sharing user’s emotion. United States Patent Application 20130144937 A1. 7 Aug 2012. Apple, owner. Greenzeiger MF, Phulari R, Sanghavi M, inventors. Inferring user mood based on user and group characteristic data. United States Patent Application 20140025620 A1. 23 Jan 2014. Microsoft, owner. Roseway A, Davis FA, Carroll EA, et al. inventors. Mood-actuated device. United States Patent Application 20140085181 A1. 27 Mar 2014 Yahoo, owner. White W, Slaney M, inventors. System and method for generating a playlist from a mood gradient. United States Patent Application US8583615 B2. 12 Nov 2013 Yahoo, owner. Kamdar G, inventor. Method and system for voice based mood analysis. United States Patent Application 20140046660 A1. 13 Feb 2014 Hamblem M. Dell’s new research division wants computers to detect your mood. Computerworld. 2014. http://www.computerworld.com/ s/article/9247285/Dell_s_new_research_division_wants_computers_ to_detect_your_mood. Accessed 9 Jul 2014