Esha Dasgupta
Education
2020-present: Ph.D. in Computer Science, University of Birmingham
2019-2020: M.Sc. in Advanced Computer Science, University of Birmingham
2016-2019: B.A. in Computer Science, University of Cambridge
Projects
CraftFormer: Clustered jitter Reduced human pose with Adversarial Fusion Transformer
Esha Dasgupta, Bo Eun Kim, Sang Hoon Yeo, Hyung Jin Chang: MoCha ECCV 2026 We propose a multi-stream fusion framework, Clustered jitter Reduced human pose with an Adversarial Fusion Transformer, CraftFormer, for regressing a subject’s 3D pose smoothly from sparse, “in-the-wild” video sequences rather than dense frame sequences. Traditional Human Pose Estimation, HPE, suffers from jitter, occlusion and a reliance on structured lab data and can result in unrealistic movement trends. These make it unsuitable for use in applications where the smoothness and naturalness of motion is critical. To overcome these limitations, CraftFormer utilises a token clustering strategy to reduce computational load and redundancy to cluster the input video into a compact set of informative keyframes, process them through separate spatial and temporal streams, then lift the fused embeddings to 3D joint positions and SMPL body parameters via a cross-stream fusion transformer. A linear regression head maps the lifted representations to SMPL pose and shape parameters, providing kinematically valid outputs. We train adversarially against motion capture sequences from AMASS, using the motion capture distribution as a biomechanical prior that penalises physically implausible motion dynamics.
SimForce: SimForce: Force and Surface Electromyography from Full Body Video with Graph Neural Nets
Esha Dasgupta, Bo Eun Kim, Sang Hoon Yeo, Hyung Jin Chang: WACV 2026 We propose a novel framework, named SimForce, for simultaneously estimating skeletal pose, ground reaction force and surface electromyography from an input video. Simforce predicts the more biomechanically accurate 3D human pose and shape of a given subject, along with their proposed muscle activations and resultant ground reaction force which leads to their input motion. Previous research has either focused on estimating these attributes singly and not treated them as a related task by taking into account the inherent shared motion between the three. In contrast, Sim- Force is designed to take advantage of the shared biological structure of the human body and its intrinsic connections to infer these attributes jointly using past, current, and future frames. SimForce features a newly introduced temporal and attention aware GCN-based architecture. To learn the subtle links between the body parts and how it affects the distribution of weight on the muscles over time, we introduce the Spatially Aware Attention Module.
BioPose: Unified Framework for Biometrics and Pose Estimation
Esha Dasgupta, Bo Eun Kim, Sang Hoon Yeo, Hyung Jin Chang: Imageomics-AAAI-2025 We propose a unified framework, named BioPose, for simultaneously estimating biometric attributes and 3D human pose. BioPose predicts five key biometric attributes—age, height, weight, gender, and ethnicity—along with 3D human pose and shape parameters from a single full-body image. Previous research has either focused on estimating multiple attributes from facial features or ttributes from full-body images but has not addressed the simultaneous estimation of these five attributes. In contrast, BioPose is designed to infer all these attributes and human pose jointly from a single input. The motivation behind BioPose lies in the intrinsic correlations among these attributes, which can enhance prediction accuracy when learned through a multi-task learn- ing approach within a unified framework. BioPose features a newly introduced attention-augmented U-Net architecture. To enable accurate attribute estimation by focusing on the most relevant image regions, we integrate attention modules into the U-Net structure. Despite tackling multiple tasks simultaneously, BioPose achieves reasonable performance compared to state-of-the-art methods tailored to each individual task, demonstrating its robustness and efficacy as a baseline.
HS-Pose: Hybrid Scope Feature Extraction for Category-level Object Pose Estimation
Linfang Zheng, Chen Wang, Yinghan Sun, Esha Dasgupta, Hua Chen, Ales Leonardis, Wei Zhang, Hyung Jin Chang: CVPR 2023 In this paper, we focus on the problem of category-level object pose estimation, which is challenging due to the large intra-category shape variation. 3D graph convolution (3D-GC) based methods have been widely used to extract local geometric features, but they have limitations for complex shaped objects and are sensitive to noise. Moreover, the scale and translation invariant properties of 3D-GC restrict the perception of an object’s size and translation information. In this paper, we propose a simple network structure, the HS-layer, which extends 3D-GC to extract hybrid scope latent features from point cloud data for category-level object pose estimation tasks. The proposed HS-layer: 1) is able to perceive local-global geometric structure and global information, 2) is robust to noise, and 3) can encode size and translation information. Our experiments show that the simple replacement of the 3D-GC layer with the proposed HS-layer on the baseline method (GPV-Pose) achieves a significant improvement, with the performance increased by 14.5% on 5d2cm metric and 10.3% on IoU75. Our method outperforms the state-of-the-art methods by a large margin (8.3% on 5d2cm, 6.9% on IoU75) on REAL275 dataset and runs in real-time (50 FPS).
Ph.D. project: Vision-based 3D Human Posture Estimation and Musculoskeletal Regression
A continuation of the Masters Project, outlined below. The Project aims to take in monocular video of a subject’s motion, estimate its personalised body model, biometric attributes and predict their trajectory as a pose estimation problem. In addition, the subject’s resultant ground reaction forces and surface electromyography is calculated from the motion. Typically this task would be computationally expensive and need specialised equipment but the project replaces this with a single camera.
Masters Project: Scientific implementation of machine learning algorithms in motion modelling pipelines
This project tries to estimate the inverse dynamics of a person from a video taken by a RGB camera. The 3D pose is calculated from a 2D image along with the ground forces acting upon the subject. OpenSim is used to customise a personalized body model before its motion in video is converted into motion capture records and used for calculating the inverse kinematics as well as dynamics for the body.
Mini Project: Classifying Facial Emotion to explore a Hybrid Model
This work combined the categorical and continuous models of emotion by treating compound emotions as vectors to be projected upon a vector space where the basis vectors are the categorical emotions. This was done in a decoupled manner where a multi-input convolutional network broke down an image into its resultant Facial Action Coding System features and a secondary linear network classified the features into a probability vector for the basic Ekman emotions and neutral. This system was used to categorise more compound emotions and check if there was any relation between similar emotions in the vector space.
Robotics Final Project: A Personal Assistant
This project designs and implements an autonomous Personal Assistant which can use voice recognition to receive tasks and a recipient to deliver a message or an object to. Machine learning algorithms are used to detect and classify objects as well as people, and the assistant schedules its tasks dynamically based upon a known room’s location, a probabilistic location of a person, task priority and future tasks. The robot uses AMCL localisation as well as RRT Route Planning to navigate between locations.
Part II Project: Escape from a Hostile Terrain with Agents using Reinforcement Learning
A Deep Q-Learning algorithm is used to make heterogenous agents learn optimal prey policies when faced with a pseudo-intelligent opponent. It investigates the impact of different reward functions and the resulting agent behaviours when building from a ‘blank state’ to see if they are comparable to real world interactions and is written in Python.
IB Group Project: Anthropometrics today
This project takes two images of a person’s head and uses edge detection along with facial structure decomposition to get structural measurements and then finds the closest match to the person in the provided database.