Projects I have been working on.

Scientific Figure Captioning
Generating Figure Captions for Scientific Documents with Automated Text Summarization
Photo of CrowdWriting
Crowd-AI Lab, Penn State University Dr. Kenneth Huang 2022

Our project formulates scientific figure captioning as a summarization task, where paragraphs mentioning the figures are summarized to produce informative captions. Through fine-tuning the Pegasus model, we have achieved significant performance improvements compared to vision-to-language models. Furthermore, token alignment experiments have revealed that roughly 75% of the captions can be found in the figure-mentioning paragraphs, confirming the effectiveness of our summarization approach.

Story Plot Prediction
Conveying the Predicted Future to Users via Story Plot Prediction
Photo of CrowdWriting
Crowd-AI Lab, Penn State University Dr. Kenneth Huang 2022

We have developed a system, Semantic Frame Enhanced GPT-2, which generates short descriptions to support novel writing by providing potential future story plots. To evaluate the system's performance, we conducted human evaluations that assessed the quality of the generated story plots and how they were used in a writing task. Despite GPT-3's impressive performance, the proposed model could also make a positive impact on the writing process.

Guided K-best Selection for Semantic Parsing Annotation
Computer-assisted semantic parsing
Photo of CrowdWriting
Semantic Machines, Microsoft Research Charles Chen, Dr. Jacob Andreas, and Dr. Ben Van Durme. 2021

Our paper proposes an AI-powered human-in-the-loop process to collect conversational semantic parsing data efficiently, even with limited data. Our proposed method, guided k-best selection, generates possible candidates, allows users to filter incorrect parses, and asks them to select the correct parse with minimal modifications. We conducted a user study with five annotators and found that combining keyword searching and suggestion enabled fast and accurate annotation.

Semantic Frame Prediction
Predicting What Would Happen in the Follow-up Story
Photo of CrowdWriting
Crowd-AI Lab, Penn State University Dr. Kenneth Huang 2021

We introduce frame representations to describe story blocks, a story snippet that contains a fixed number of sentences. Using this formulation, we treat a full story as a sequence of story blocks and propose a Semantic Frame Prediction task where the idea is to predict what would happen in the follow-up story using previous information (either in text or in frame representation).

Learner-Like Agent
Assessing Helpfulness of the Learning Materials
Photo of the Learner-Like Agent
Collaborated with NLPSA Lab, Academia Sinica Dr. Lun-Wei Ku 2020

Automatically finding useful learning materials is hard. In this project, we develop a Learner-Like Agent that can mimic learners' behavior. By asking the agent to learn all the materials and test its corresponding performance, we can then find out the good materials.

Helping Creative Writing
Photo of CrowdWriting
Crowd-AI Lab, Penn State University Dr. Kenneth Huang 2019

Writing is a complicated task that needs a complex skills. Supporting writing, therefore, is a difficult task for AI since AI is not capable of understanding. In this project, we try to provide various helps for writer by using the power of crowd.

Geographic Information Prediction on Twitter
Location Prediction on Pure Text
Photo of Geographic Information
Data Lab, Arizona State University Dr. Hanghang Tong 2018

Geographic Information plays an important role on both marketing and event mining, but is usually blocked due to the privacy issues. This project introduces a deep learning architecture taking the attention mechanism, the subword feature, and the location hierarchy structure into account to predict the geographic information for a given post on Twitter.

Response Time Prediction
EmotionPush Dataset and its application
Photo of Response Time Prediction
NLPSA Lab, Academia Sinica Dr. Lun-Wei Ku 2018

This project aims to predict the response time of a given message sending on the instance message system. This task could be viewed as a measurement of the dialog generation system. A deep learning model integrating conversation and some user-specific information is proposed.

A Keyboard for Sentence Suggestion According to Emotions.
Photo of MoodSwipe
NLPSA Lab, Academia Sinica Dr. Lun-Wei Ku 2017

MoodSwipe is a mobile phone keyboard that suggests text messages according to the user-specified emotion. We aim to create a convenient user interface to enjoy the technology of emotion classification and text suggestion, and at the same time to collect labeled data automatically. Two emotion classifier models, CNN and LSTM, and two sentence suggestion models, BM25 and similarity of sentence embedding, are built for MoodSwipe.

Color-Based Emotion Cues for Messaging Applications
Photo of EmotionPush
NLPSA Lab, Academia Sinica Dr. Lun-Wei Ku 2017

EmotionPush provides a machine-learning-powered system that automatically conveys users’ emotions in messages by color-based emotion cues to bridge the limitation of text-based chatting system in expressing rich emotion.

Learning Synonyms by Example Sentences
Photo of GiveMeExample
NLPSA Lab, Academia Sinica Dr. Lun-Wei Ku 2016 - 2017

GiveMeExample aims to suggest critical example sentences for language learner to clarify the confusion of synonym. Three main components, the sentence difficulty assessment built by a regression model, the word-sentence fitness estimator built by GMM and BiLSTM, and the heuristic clarification scoring function are introduced to solve this problem. Several websites are built for collecting data and holding evaluation tests.