Hanbyul (Han) Joo

Research Scientist
Facebook AI Research (FAIR)
Email: hjoo-at-fb-dot-com
[Google Scholar] [CV]

I am a Research Scientist at Facebook AI Research (FAIR), Menlo Park. I finished my Ph.D. in the Robotics Institute at Carnegie Mellon University, where I worked with Yaser Sheikh. During my PhD, I interned at Facebook Reality Labs, Pittsburgh (Summer and Fall, 2017) and Disney Research Zurich (Summer, 2015). Before joining CMU, I spent three years as a researcher at ETRI, a government-funded research organization in Korea. I received my M.S. in Electrical Engineering (under the supervision of Prof. In So Kweon), and B.S. in Computer Science, both from KAIST, Korea. The Samsung Scholarship partly supported my graduate study, and I am a recipient of the CVPR Best Student Paper Award in 2018.


Research

The goal of my research is to endow machines and robots with the ability to perceive and understand human behaviors in 3D. Ultimately, I dream to build an AI system that can behave like humans in new environments and can interact with humans using a broad channel of nonverbal signals (kinesic signals or body languages). I pursue this direction by creating new tools in computer vision, machine learning, and computer graphics.

   
Markerless Total Motion Capture
   The Panoptic Studio
  PIFuHD: 3D Human from A Single Image

News

  • Feb 2021 Our Body2Hands paper got accepted in CVPR 2021

  • Oct 2020 One paper got accepted in NeurIPS 2020

  • Aug 2020 We have released FrankMocap, an easy-to-use 3D body+hand pose estimator from images/videos: Link

  • July 2020 We have released EFT psuedo 3D human pose dataset: Link

  • July 2020 One paper got accepted in ECCV 2020

  • June 2020 Our PIFuHD went super viral! Check out fun results in twitter with #pifuhd hash tag: Link

  • June 2020 Better late than never, we have released the "processed version" of haggling dataset with annotations: Link

  • June 2020 PIFuHD code has been released: Link

  • Feb 2020 Two papers got accepted as Oral Publications in CVPR 2020.

  • June 2019 Code and dataset of our Monocular Total Capture (MTC) paper has been released: link

  • Mar 2019 Two papers got accepted as Oral Publications in CVPR 2019.

  • Jan 2019 I joined Facebook AI Research (FAIR) as a research scientist from Jan 2019.

  • Dec 2018 I have defended my PhD thesis!!

  • Sep 2018 I talked about our research direction in the interview with "This Week in Machine Learning & AI"

  • Jun 2018 Our Total Capture paper received Best Student Paper Award at CVPR 2018

  • Feb 2018 Our research has been featured on WIRED

  • Feb 2018 I have two papers accepted to CVPR 2018

Dataset/Library

FrankMocap

CMU Panoptic Studio Dataset

OpenPose Library

Tutorial

DIY A Multiview Camera System: Panoptic Studio Teardown
Hanbyul Joo, Tomas Simon, Hyun Soo Park, Shohei Nobuhara, and Yaser Sheikh
In Conjunction with CVPR 2017
[Tutorial Page] [Video]

Publications

Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics
Evonne Ng, Shiry Ginosar, Trevor Darrell, Hanbyul Joo
CVPR 2021
[arxiv] [Project Page]
3D Multi-bodies: Fitting Sets of Plausible 3D Human Models to Ambiguous Image Data
Benjamin Biggs, Seb Ehrhadt, Hanbyul Joo, Ben Graham, Andrea Vedaldi, David Novotny
In NeurIPS 2020   (Spotlight)
[arxiv] [Project Page]
FrankMocap: A Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration
Yu Rong, Takaaki Shiratori, Hanbyul Joo
arXiv 2020
[arxiv] [Project Page] [Code]
Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation
Hanbyul Joo, Natalia Neverova, Andrea Vedaldi
arXiv 2020
[arxiv] [Code/Dataset]
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild
Jason Y. Zhang*, Sam Pepose*, Hanbyul Joo, Deva Ramanan, Jitendra Malik, Angjoo Kanazawa
In ECCV 2020
[arxiv] [Project Page]
PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization
Shunsuke Saito, Tomas Simon, Jason Saragih, Hanbyul Joo
In CVPR 2020   (Oral)
[arxiv] [Project Page] [Code] [Colab Demo]
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions
Evonne Ng, Donglai Xiang, Hanbyul Joo, Kristen Grauman
In CVPR 2020   (Oral)
[arxiv] [Project Page] [Code/Data]
Single-Network Whole-Body Pose Estimation
Gines Hidalgo, Yaadhav Raaj, Haroon Idrees, Donglai Xiang, Hanbyul Joo, Tomas Simon, Yaser Sheikh
In ICCV 2019
[arxiv] [OpenPose Training] [OpenPose]
Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction
Hanbyul Joo, Tomas Simon, Mina Cikara, Yaser Sheikh
In CVPR 2019   (Oral)
[arxiv] [Code/Dataset] [Video]
Monocular Total Capture: Posing Face, Body and Hands in the Wild
Donglai Xiang, Hanbyul Joo, Yaser Sheikh
In CVPR 2019   (Oral)
[arxiv] [Dataset and Code] [Video]
Sensing, Measuring, and Modeling Social Signals in Nonverbal Communication
Hanbyul Joo
PhD Thesis, Robotics Institute, Carnegie Mellon University, 2019
[PDF]
Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies
Hanbyul Joo, Tomas Simon, and Yaser Sheikh
In CVPR 2018   (Oral)  
[CVPR Best Student Paper Award]
[PDF] [Supplementary Material] [video] [Project Page]
Structure from Recurrent Motion: From Rigidity to Recurrency
Xiu Li, Hongdong Li, Hanbyul Joo, Yebin Liu, Yaser Sheikh
In CVPR 2018
[PDF]
Hand Keypoint Detection in Single Images using Multiview Bootstrapping
Tomas Simon, Hanbyul Joo, Iain Mattews, and Yaser Sheikh
In CVPR 2017
[arXiv] [Project Page] [Code] [Dataset]
Panoptic Studio: A Massively Multiview System for Social Interaction Capture
Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh
In TPAMI 2017 (Extended version of ICCV15)
[published version] [arXiv version] [Dataset]
Panoptic Studio: A Massively Multiview System for Social Motion Capture
Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh
In ICCV 2015   (Oral)
[Paper(PDF)] [Supplementary Material] [Project Page]
Press Coverage:

MAP Visibility Estimation for Large-Scale Dynamic 3D Reconstruction
Hanbyul Joo, Hyun Soo Park, and Yaser Sheikh
In CVPR 2014   ((Oral)
[Paper(PDF)] [Project Page] [Dataset]
Press Coverage: ...
Graph-based Shape Matching for Deformable Objects
Hanbyul Joo, Yekeun Jeong, Olivier Duchenne, and In So Kweon
In ICIP 2011
[Paper(PDF)]

Graph-Based Robust Shape Matching for Robotic Application
Hanbyul Joo, Yekeun Jeong, Olivier Duchenne, Seong-Young Ko, and In So Kweon
In ICRA 2009
[Paper(PDF)]

Statistical Background Subtraction Based on the Exact Per-pixel Distributions
Youngbae Hwang, Hanbyul Joo, Jun-sik Kim, and In So Kweon
In IAPR workshop on Machine Vision Applications (MVA) 2007
[Paper(PDF)]

Talks

  • "Measuring and Modeling Human Motion"

    • Facebook AI Video Summit, June 2019.

  • "Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction"

  • "Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies"

  • "Measuring and Modeling Social Signals for Computational Behavioral Understanding"

    • UC Berkeley, BAIR, MAY 2018 (hosted by Prof. Jitendra Malik).
    • UT Austin, School of Computer Science, April 2018.
    • CMU, School of Computer Science, April 2018.
    • MIT, CSAIL, April 2018.
    • MIT, Media Lab, Nov 2017.

  • "The Panoptic Studio: A Massively Multiview System for Social Interaction Capture"

    • UC Berkeley, Computer Vision Group, Dec 2016 (hosted by Prof. Alexei A. Efros).
    • Stanford, Computer Vision and Geometry Lab (CVGL), Dec 2016 (hosted by Prof. Silvio Savarese).
    • Adobe Research, Dec 2016 (hosted by Dr. Joon-Young Lee).
    • ASSP4MI workshop of ICMI, Nov 2016.
    • CMU, Machine Learning Lunch Seminar, Oct 2016.
    • ICCV Oral Talk, Dec 2015 (video link).
    • CMU, VASC Seminar, Dec 2015.
    • ETH Zurich, Computer Vision and Geometry lab, Oct 2015 (hosted by Prof. Marc Pollefeys).
    • Seoul National University, June 2015 (hosted by Prof. Kyoung Mu Lee).
    • ETRI, CG Team, May 2015 (hosted by Dr. Seong-Jae Lim).
    • KAIST, May 2015 (hosted by Prof. In So Kweon).

  • "The Panoptic Studio (CVPR14)"

    • Graduate Seminar, Civil & Environmental Engineering, CMU, Feb 2015 (hosted by Prof. Hae Young Noh).
    • People Image Analysis Consortium, CMU, Nov 2014.
    • Reality Computing Meetup, Autodesk, Nov 2014.

  • "MAP Visibility Estimation for Large-Scale Dynamic 3D Reconstruction"

Press Coverage

Patents

  • Motion capture apparatus and method (Patent No.: US 8805024 B2)
    Hanbyul Joo, Seong-Jae Lim, Ji-Hyung Lee, Bon-Ki Koo

  • Method for automatic rigging and shape surface transfer of 3D standard mesh model based on muscle and nurbs by using parametric control (Patent No.: US 7171060 B2)
    Seong Jae Lim, Ho Won Kim, Hanbyul Joo, Bon Ki Koo

  • 3D model shape transformation method and apparatus (Patent Application No.: US 20120162217 A1)
    Seong-Jae Lim, Hanbyul Joo, Seung-Uk Yoon, Ji-Hyung Lee, Bon-Ki Koo.