Research Scientist Intern - Multimodal Sensing & On-Device Perception - Global Frontier Tech Recruitment Program - 2027 Start (PHD)

Responsibilities

• Design and prototype novel sensor or imaging architectures that move computation closer to the sensing front-end (e.g., near-sensor processing, event-driven capture, learned compression at the pixel level)
• Build and characterize imaging pipelines end-to-end: from optical/sensor physics through ISP to downstream perception models — identify where bits are wasted and where intelligence should be injected
• Leverage understanding of VLM/LLM and world models to inform what information the sensing front-end must preserve, discard, or transform — closing the loop between foundation model requirements and hardware design
• Develop or adapt machine vision models that are co-optimized with hardware constraints (power, bandwidth, latency)

Qualifications

• Currently pursuing a PhD in Computer Science, Electrical Engineering, Optical Engineering, Applied Mathematics, Physics, or a related technical field.
• Strong research background in computer vision and machine learning, with hands-on model training experience.
• Experience with at least one of: sequence modeling, language modeling, efficient neural network design, or signal processing.

Topic Content

The rapid advancement of AI Agents is set to have a far-reaching and transformative impact on daily life. As the hardware bridge linking AI Agents to the physical world and end users, intelligent hardware acts as a vital gateway for agents to perceive their environment and recognize user intent. Enabling AI Agents to maintain round-the-clock environmental awareness and efficiently capture real-time user intentions is essential to improving their everyday service experience. This topic seeks to overcome the limitations of conventional visual perception systems. By deeply integrating sensing and computing, it explores full-stack innovation spanning from low-level hardware to high-level algorithms. Key efforts include developing next-generation sensors with real-time understanding capabilities, investigating non-traditional visual capture and compression techniques, and designing novel hardware architectures to run advanced algorithms—enabling highly efficient coordination across sensing, processing, and communication.

Topic Challenge

Break free from the framework of conventional sensing systems, explore novel sensors, signal processing and compression schemes to achieve highly energy-efficient sensing tasks, while enabling seamless integration with large models.

Topic Value

Breakthroughs in this research direction will enable intelligent hardware to better connect AI, users, and daily life. Against the backdrop of the AI Agent era, this will unlock the gateway to the next generation of intelligent hardware terminals and open up broader technological possibilities.

Project Objective

Rethink visual perception from the sensor up. Instead of treating sensing and computation as separate stages, we're exploring architectures that fuse them — sensors that compress, select, and partially "understand" the scene before data ever leaves the chip. This requires deeply understanding both the hardware constraints and what modern vision-language and world models actually need from the sensing front-end. The goal: order-of-magnitude gains in perception efficiency by co-designing hardware and algorithms across the full pipeline.

Job Information

[For Pay Transparency] Compensation Description (Hourly) - Campus Intern The hourly rate range for this position in the selected city is $55- $55. Benefits may vary depending on the nature of employment and the country work location. Interns have day one access to health insurance, life insurance, wellbeing benefits and more. Interns also receive 10 paid holidays per year and paid sick time (56 hours if hired in first half of year, 40 if hired in second half of year). Interns who are not working 100% remote may also be eligible for housing allowance. The Company reserves the right to modify or change these benefits programs at any time, with or without notice.