- Nov 10, 2018
Sandeep's work is awarded 2nd Best Poster at the MHI Research Festival
Sandeep's work titled 'Modeling how couples influence each other’s behavior during therapy interactions' is awarded 2nd Best Poster Award at the 9th Annual Ming Hsieh Department of Electrical Engineering Research Festival.
- October, 2018
One paper was presented at ICMI 2018, Boulder, Colorado, United States
"Honey, I Learned to Talk": Multimodal Fusion for Behavior Analysis
Shao-Yeng Tseng*, Haoqi Li*, Brian Baucom, and Panayiotis Georgiou
- September, 2018
Four papers are presented at Interspeech 2018, Hyderabad, India
Modeling Interpersonal Influence of Verbal Behavior in Couples Therapy Dyadic Interactions
Sandeep Nallan Chakravarthula, B. R. Baucom, and Panayiotis Georgiou
An Unsupervised Neural Prediction Framework for Learning Speaker Embeddings using Recurrent Neural Networks
Arindam Jati and Panayiotis Georgiou
Towards an Unsupervised Entrainment Distance in Conversational Speech using deep Neural Networks
Md Nasir, Brian Baucom, Shrikanth Narayanan, and Panayiotis Georgiou
Multimodal Speaker Segmentation and Diarization using Lexical and Acoustic Cues via Sequence to Sequence Neural Networks
Tae Jin Park and Panayiotis Georgiou
- April, 2018
Three papers are presented at ICASSP 2018, Calgary, Canada
TOWARDS PREDICTING PHYSIOLOGY FROM SPEECH DURING STRESSFUL CONVERSATIONS: HEART RATE AND RESPIRATORY SINUS ARRHYTHMIA
Arindam Jati, Paula Williams, Brian Baucom, and Panayiotis Georgiou
A DEEP REINFORCEMENT LEARNING FRAMEWORK FOR IDENTIFYING FUNNY SCENES IN MOVIES
Haoqi Li, Naveen Kumar, Ruxin Chen, and Panayiotis Georgiou
MULTISTREAM DIARIZATION FUSION USING THE MINIMUM VARIANCE BAYESIAN INFORMATION CRITERION
Tae Jin Park and Panayiotis Georgiou
- Nov 10, 2017
Nasir's work is honorably mentioned at the MHI Research Festival
Nasir's work titled 'Complexity in Speech and its Relation to emotional Bond in Therapist-Patient Interactions During Suicide Risk Assessment Interviews' is honorably mentioned at the 8th Annual Ming Hsieh Department of Electrical Engineering Research Festival.
- Oct 01, 2017
Our Work on Couples Outcomes is Featured in the Media
Original journal :
"Predicting couple therapy outcomes based on speech acoustic features", in PLOS ONE September, 2017.
Md Nasir, Brian Robert Baucom, Panayiotis Georgiou, and Shrikanth Narayanan
The Conversation [PDF version]
International Business Times UK [PDF version]
EJ Insight [PDF version]
- August, 2017
Four papers are presented at Interspeech 2017, Stockholm, Sweden
Speaker2Vec: Unsupervised Learning and Adaptation of a Speaker Manifold using Deep Neural Networks with an Evaluation on Speaker Segmentation
Arindam Jati and Panayiotis Georgiou
Exploiting Intra-annotator Rating Consistency through Copeland's Method for Estimation of Ground Truth Labels in Couples' Therapy
K. Mundnich, M. Nasir, P. Georgiou, and S. Narayanan
Complexity in speech and its relation to emotional bond in therapist-patient interactions during suicide risk assessment interviews
M. Nasir, B. Baucom, C. Bryan, S. Narayanan, and P. Georgiou
Approaching Human Performance in Behavior Estimation in Couples Therapy Using Deep Sentence Embeddings
Shao-Yen Tseng, Brian Baucom, and Panayiotis Georgiou
- Jan, 2017
Paper Accepted to ICASSP 2017 : UNSUPERVISED LATENT BEHAVIOR MANIFOLD LEARNING FROM ACOUSTIC FEATURES: AUDIO2BEHAVIOR
Haoqi Li, Brian Baucom, Panayiotis Georgiou
In this work we exploit the slow varying properties of human behavior. We hypothesize that nearby segments of speech share the same behavioral context and hence share a similar underlying representation in a latent space. Specifically, we propose a Deep Neural Network (DNN) model to connect behavioral context and derive the behavioral manifold in an unsupervised manner. We evaluate the proposed manifold in the couples therapy domain and also provide examples from publicly available data (e.g. stand-up comedy). We further investigate training within the couples’ therapy domain and from movie data. The results are extremely encouraging and promise improved behavioral quantification in an unsupervised manner and warrants further investigation in a range of applications.
Nine papers are presented at Interspeech 2016, San Fransisco, USA
Sparsely Connected and Disjointly Trained Deep Neural Networks for Low Resource Behavioral Annotation: Acoustic Classification in Couples' Therapy
Haoqi Li, Brian Baucom, and Panayiotis Georgiou.
Complexity in Prosody: A Nonlinear Dynamical Systems Approach for Dyadic Conversations; Behavior and Outcomes in Couples Therapy
Md Nasir, Brian Baucom, Shrikanth Narayanan, and Panayiotis Georgiou.
Perception Optimized Deep Denoising AutoEncoders for Speech Enhancement
Prashanth Gurunath Shivakumar and Panayiotis Georgiou.
Multimodal Fusion of Multirate Acoustic, Prosodic, and Lexical Speaker Characteristics for Native Language Identification
Prashanth Gurunath Shivakumar, Sandeep Nallan Chakravarthula, and Panayiotis Georgiou.
Couples Behavior Modeling and Annotation Using Low-Resource LSTM Language Models
Shao-Yen Tseng, Sandeep Nallan Chakravarthula, Brian Baucom, and Panayiotis Georgiou.
A Deep Learning Approach to Modeling Empathy in Addiction Counseling
James Gibson, Dogan Can, Bo Xiao, Zac Imel, David Atkins, Panayiotis Georgiou, and Shrikanth Narayanan.
Laughter Valence Prediction in Motivational Interviewing based on Lexical and Acoustic Cues
Rahul Gupta, Nishant Nath, Taruna Agrawal, Panayiotis Georgiou, David Atkins, and Shrikanth Narayanan.
Robust multi-channel gender classification from speech in movie audio
Naveen Kumar, Md Nasir, Panayiotis Georgiou, and Shrikanth Narayanan.
Behavioral Coding of Therapist Language in Addiction Counseling using Recurrent Neural Networks
Bo Xiao, Dogan Can, James Gibson, Zac Imel, David Atkins, Panayiotis Georgiou, and Shrikanth Narayanan.
- Aug 30, 2016
Our Work on Couples Outcomes is Featured in Media
Our work on couples outcomes has been featured in multple media outlets.
Original paper :
"Still Together?: The Role of Acoustic Features in Predicting Marital Outcome", in Proceedings of Interspeech, Dresden, Germany, September 2015.
Md Nasir, Wei Xia, Bo Xiao, Brian Baucom, Shrikanth Narayanan, and Panayiotis Georgiou.
http://www.scientificamerican.com/podcast/episode/this-algorithm-can-predict-relationship-trouble http://numrush.nl/2015/11/25/all-you-need-is-code-algoritme-nu-al-betere-matchmaker-dan-relatietherapeut/http://chronicledaily.com/2015/11/25/can-your-tone-of-voice-predict-the-success-of-your/ http://phys.org/news/2015-11-words-tone-voice.htmhttp://psychcentral.com/news/2015/11/24/tone-of-voice-more-important-than-word-choice-during-spousal-arguments/95304.htmlhttp://techcrunch.com/2015/11/24/algorithm-predicts-relationship-success-through-couples-tone-of-voice/http://thespokedblog.com/2015/11/25/algorithm-can-predict-if-your-relationship-will-survive/http://www.eurekalert.org/pub_releases/2015-11/uosc-swc112315.phphttp://www.newsroomamerica.com/story/532180/words_can_deceive_but_tone_of_voice_cannot.htmlhttp://www.plymouthherald.co.uk/success-marriage-depend-tone-voice-boffins-claim/story-28239085-detail/story.htmlhttp://www.techinsider.io/usc-university-of-utah-created-a-computer-program-that-can-predict-relationship-success-2015-11http://news.usc.edu/89023/sticks-and-stones-words-can-deceive-tone-of-voice-cannot/https://news.usc.edu/77596/therapist-to-patient-what-i-hear-is/http://www.newkerala.com/news/2015/fullnews-152827.html
- June 07, 2016
NIA grant on Stress Assessment
Development of a Standard Protocol for the Assessment of Stress-Related Vocal Responding,
NIA -- Stress Measurement Network
Brian Baucom, Paula Williams, co-PIs; Panayiotis Georgiou, co-I
Seven Papers Presented at Interspeech 2015, Dresden , Germany
A dynamic model for behavioral analysis of couple interactions using acoustic features
W. Xia, J. Gibson, B. Xiao, B. Baucom, and P. Georgiou
Analysis and modeling of the role of laughter in motivational interviewing based psychotherapy conversations
R. Gupta, T. Chaspari, P. G. Georgiou, D. C. Atkins, and S. Narayanan,
Analyzing speech rate entrainment and its relation to therapist empathy in drug addiction counseling
B. Xiao, Z. Imel, D. Atkins, P. Georgiou, and S. Narayanan
Assessing empathy using static and dynamic behavior models based on therapist's language in addiction counseling
S. N. Chakravarthula, B. Xiao, Z. E. Imel, D. C. Atkins, and P. Georgiou
Automated evaluation of non-native english pronunciation quality: Combining knowledge- and data-driven features at multiple time scales
M. Black, D. Bone, Z. I. Skordilis, R. Gupta, W. Xia, P. Papadopoulos, S. N. Chakravarthula, B. Xiao, M. V. Segbroeck, J. Kim, P. Georgiou, and S. Narayanan
Automatic estimation of parkinson's disease severity from diverse speech tasks
J. Kim, M. Nasir, R. Gupta, M. V. Segbroeck, D. Bone, M. Black, Z. I. Skordilis, Z. Yang, P. Georgiou, and S. Narayanan
- April, 2015
Paper Accepted to ICASSP 2015 : REDUNDANCY ANALYSIS OF BEHAVIORAL CODING FOR COUPLES THERAPY AND IMPROVED ESTIMATION OF BEHAVIOR FROM NOISY ANNOTATIONS
Md Nasir, Brian Baucom, Panayiotis Georgiou, and Shrikanth S. Narayanan
Assessment and quantification of behavior is an important research objective in the recently developed field of behavioral signal processing. This paper focuses on the estimation of behavior from noisy human assessment. It aims to address the redundancy of behavioral descriptors for couples therapy by introducing a lower-dimensional representation of the behavioral space. We present an improved method for estimating the ground truth of behavioral ratings from assessment by multiple experts or annotators. The results show improved estimation performance using the proposed method and provide an insightful analysis of reconstruction error and decorrelation of annotator bias in the reduced behavioral space.
Paper Accepted to ICASSP 2015 : A LANGUAGE-BASED GENERATIVE MODEL FRAMEWORK FOR BEHAVIORAL ANALYSIS OF COUPLES'S THERAPY
Sandeep Nallan Chakravarthula, Rahul Gupta, Brian Baucom, and Panayiotis Georgiou
Observational studies for psychological evaluations rely on careful assessment of multiple behavioral cues. Recent studies have made good progress in automating the psychological evaluation, which often involved tedious manual annotation of a set of behavioral codes. However, the current methods impose strict and often unnatural assumptions for evaluation. In this work, we specifically investigate two goals:
(1) Human behavior changes throughout an interaction and better models of this evolution can improve automated behavioral annotation and
(2) Human perception of this evolution can be quite complex and non-linear and better techniques than averaging need to be investigated.
For this purpose, we propose a Dynamic Behavior Modeling (DBM) scheme, which models a spouse as undergoing changes in behavioral state within a session, and contrast it against a Static Behavior Model (SBM) which allows only a constant session-long behavioral state. We use Negativity in a couples therapy task as our case study. We present results and analysis on both models for capturing the local behavior information and predicting the session level negativity label. April, 2015
Paper Accepted to ICASSP 2015 : QUANTIFYING EDA SYNCHRONY THROUGH JOINT SPARSE REPRESENTATION: A CASE-STUDY OF COUPLES’ INTERACTIONS
Theodora Chaspari, Brian Baucom, Adela Timmons, Andreas Tsiartas, Larissa Borofsky Del Piero, Katherine Baucom, Panayiotis Georgiou, Gayla Margolin, and Shrikanth S. Narayanan
The co-variation degree between individuals in their physiological signals can reveal insights about the quality of their interaction as well as their personal characteristics. In an effort to capture the amount of synchrony between Electrodermal Activity (EDA) streams occurring in parallel during dyadic interactions, we propose Sparse EDA Synchrony Measure (SESM), an index derived from the joint sparse representation of EDA ensembles. Sparse decomposition is performed using Simultaneous Orthogonal Matching Pursuit (SOMP) from a knowledge-driven dictionary of tonic and phasic atoms, capturing the slow-varying trends and high-frequency signal fluctuations, respectively. At each iteration the atom having the maximum average correlation with the residuals is selected. We compute SESM as the negative natural logarithm of the joint reconstruction error and evaluate it with data from interactions of married and young dating couples participating in tasks of varying emotional intensity. Through statistical analysis and multiple linear regression experiments, our results indicate that SESM depicts significant differences across tasks in both datasets considered and can be associated to individuals’ attachment-related characteristics. June, 2014
Paper Accepted to Interspeech 2014 : Predicting Client's inclination towards Target Behavior Change in Motivational
Interviewing and investigating the role of laughter
Rahul Gupta, Panayiotis G. Georgiou, David C. Atkins, and Shrikanth Narayanan
Motivational interviewing (MI) is a goal oriented psychotherapy that facilitates intrinsic motivation within a client in order to change behavior in a dialog setting. The Motivational Interviewing Skills Code (MISC) is a manual observational coding method used to quantify and evaluate the quality of MI sessions using their audio-visual recordings. However, this coding method is both labor intensive and expensive. We present an approach towards automating MISC assignments in MI involving addiction cure. Specifically, we focus on predicting valence for "Client Change Talk" (ChangeTalk) utterances, which indicate a client's attitude towards a "Target Behavior Change" (Target). We further study the effect of incorporating counselor behavior in the model. We observe that our best model achieves an unweighted accuracy of 50.8% in a 3-way classification of positive vs negative valence ChangeTalk vs no ChangeTalk. Furthermore, we study the effect of including non-verbal behavior, specifically laughters, in our model. Information regarding location of laughters improves the unweighted accuracy of our model to 51.4% and our experimental results suggest prosodic differences in laughters belonging to ChangeTalk utterances with different valences. June, 2014
Paper Accepted to Interspeech 2014 : Modeling Therapist Empathy through Prosody in Drug Addiction Counseling
Bo Xiao, Daniel Bone, Maarten Van Segbroeck, Zac E. Imel, David C. Atkins,
Panayiotis G. Georgiou, Shrikanth S. Narayanan
Empathy measures the capacity of the therapist to experience the same cognitive and emotional dispositions as the patient, and is a key quality factor in counseling. In this work we build computational models to infer the empathy of therapist using prosodic cues. We extract pitch, energy, jitter, shimmer and utterance duration from the speech signal, and normalize and quantize these features in order to estimate the distribution of certain prosodic patterns during each interaction. We find significant correlation between empathy and the distribution of prosodic patterns, and achieve 75% accuracy in classifying therapist empathy levels using this distribution. Experiment results suggest high pitch and energy of the therapist are negatively correlated with empathy. These observations agree with domain literature and human intuition.
- June, 2014
Paper Accepted to Interspeech 2014 : Unsupervised Speaker Diarization Using Riemannian Manifold Clustering
Che-Wei Huang, Bo Xiao, Panayiotis G. Georgiou, Shrikanth S. Narayanan
We address the problem of speaker clustering for robust unsupervised speaker diarization. We model each speaker homogeneous segment as one single full multivariate Gaussian probability density function (pdf) and take into consideration the Riemannian property of Gaussian pdfs. By assuming that segments from different speakers lie on different (possibly intersected) sub-manifolds of the manifold of Gaussian pdfs, we formulate the original problem as a Riemannian manifold clustering problem. To apply the computationally simple Riemannian locally linear embedding (LLE) algorithm, we impose a constraint on the length of each segment so as to ensure the fitness of single-Gaussian modeling and to increase the chance that all k-nearest neighbors of a pdf are from the same sub-manifold (speaker).
- September, 2013
Md Nasir, Wei Xia, and Sandeep Nallan Chakravarthula join our lab.
- August, 2013
Panayiotis Georgiou joins Ming Hsieh Electrical Engineering Faculty
Panayiotis Georgiou joins Ming Hsieh Electrical Engineering Faculty