I am a Research Scientist at Facebook AI Research (FAIR), Menlo Park. I finished my Ph.D. in the Robotics Institute at Carnegie Mellon University, where I worked with Yaser Sheikh. During my PhD, I interned at Facebook Reality Labs, Pittsburgh (Summer and Fall, 2017) and Disney Research Zurich (Summer, 2015). Before joining CMU, I spent three years as a researcher at ETRI, a government-funded research organization in Korea. I received my M.S. in Electrical Engineering (under the supervision of Prof. In So Kweon), and B.S. in Computer Science, both
from KAIST, Korea. The Samsung Scholarship partly supported my graduate study, and I am a recipient of the CVPR Best Student Paper Award in 2018.
Research
The goal of my research is to endow machines and robots with the ability to perceive and understand human behaviors in 3D. Ultimately, I dream to build an AI system that can behave like humans in new environments and can interact with humans using a broad channel of nonverbal signals (kinesic signals or body languages). I pursue this direction by creating new tools in computer vision, machine learning, and computer graphics.
DIY A Multiview Camera System: Panoptic Studio Teardown Hanbyul Joo, Tomas Simon, Hyun Soo Park, Shohei Nobuhara, and Yaser Sheikh
In Conjunction with CVPR 2017 [Tutorial Page][Video]
Publications
3D Multi-bodies: Fitting Sets of Plausible 3D Human Models to Ambiguous Image Data Benjamin Biggs, Seb Ehrhadt, Hanbyul Joo, Ben Graham, Andrea Vedaldi, David Novotny
In NeurIPS 2020   (Spotlight)
[arxiv]
[Project Page]
FrankMocap: A Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration Yu Rong, Takaaki Shiratori, Hanbyul Joo arXiv 2020 [arxiv][Project Page][Code]
Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation Hanbyul Joo, Natalia Neverova, Andrea Vedaldi
arXiv 2020 [arxiv][Code/Dataset]
Perceiving 3D Human-Object Spatial Arrangements
from a Single Image in the Wild Jason Y. Zhang*, Sam Pepose*, Hanbyul Joo, Deva Ramanan, Jitendra Malik, Angjoo Kanazawa
In ECCV 2020 [arxiv][Project Page]
PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization Shunsuke Saito, Tomas Simon, Jason Saragih, Hanbyul Joo
In CVPR 2020   (Oral) [arxiv][Project Page][Code][Colab Demo]
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions Evonne Ng, Donglai Xiang, Hanbyul Joo, Kristen Grauman
In CVPR 2020   (Oral) [arxiv][Project Page][Code/Data]
Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction Hanbyul Joo, Tomas Simon, Mina Cikara, Yaser Sheikh
In CVPR 2019   (Oral) [arxiv][Code/Dataset][Video]
Sensing, Measuring, and Modeling Social Signals in Nonverbal Communication Hanbyul Joo
PhD Thesis, Robotics Institute, Carnegie Mellon University, 2019 [PDF]
Panoptic Studio: A Massively Multiview System for Social Interaction Capture Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh
In TPAMI 2017 (Extended version of ICCV15) [published version][arXiv version][Dataset]
Panoptic Studio: A Massively Multiview System for Social Motion Capture Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara,
and Yaser Sheikh
In ICCV 2015   (Oral) [Paper(PDF)] [Supplementary Material][Project Page]
Press Coverage:
MAP Visibility Estimation for Large-Scale Dynamic 3D Reconstruction Hanbyul Joo, Hyun Soo Park, and Yaser Sheikh
In CVPR 2014   ((Oral) [Paper(PDF)] [Project Page][Dataset]
Press Coverage:
...
Graph-based Shape Matching for Deformable Objects Hanbyul Joo,
Yekeun Jeong, Olivier Duchenne, and In So Kweon
In ICIP 2011
[Paper(PDF)]
Graph-Based Robust Shape Matching for Robotic Application Hanbyul Joo,
Yekeun Jeong, Olivier Duchenne, Seong-Young Ko, and In So Kweon
In ICRA 2009
[Paper(PDF)]
Statistical Background Subtraction Based on the Exact Per-pixel Distributions
Youngbae Hwang, Hanbyul Joo, Jun-sik Kim, and In So Kweon
In IAPR workshop on Machine Vision Applications (MVA) 2007
[Paper(PDF)]
Talks
"Measuring and Modeling Human Motion"
Facebook AI Video Summit, June 2019.
"Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction"
Motion capture apparatus and method (Patent No.: US 8805024 B2) Hanbyul Joo, Seong-Jae Lim, Ji-Hyung Lee, Bon-Ki Koo
Method for automatic rigging and shape surface transfer of 3D standard mesh model based on muscle and nurbs by using parametric control (Patent No.: US 7171060 B2)
Seong Jae Lim, Ho Won Kim, Hanbyul Joo, Bon Ki Koo
3D model shape transformation method and apparatus (Patent Application No.: US 20120162217 A1) Seong-Jae Lim, Hanbyul Joo, Seung-Uk Yoon, Ji-Hyung Lee, Bon-Ki Koo.