Meta(?) Robotics of ZJU-UIUC Joint Institute was founded in 2018. I was one of the founding members. We targeted the RoboMaster competition and built our robots from scratch. As a new team at a new campus without any experience and technology heritage, unfortunately, in 2018, we didn't make it to the competition.

? Yes, Facebook was not yet Meta back then

In 2019, I took the role of the Control Group Lead. I led the development of embedded control programs and electric systems for robots, built from the ground up. We developed a cross-platform build system for STM32 microcontrollers, supporting Windows, Linux, and macOS. To ease the parameter tuning work, we also developed a wireless parameter tuning utility with a graphic user interface (GUI). During the 2019 season, I contributed 70k+ lines of C++ code and 20+ documents to the codebase.

Among the things we have built, I am most proud of the attitude and heading reference system (AHRS). In this subsystem, a gyroscope, an accelerometer and a magnetometer are integrated to calculate the absolute angle of the targeting gimbal. Together with the motor control system, our robots were able to stabilize the gimbal while rotating the chassis to evade incoming bullets, which provided us considerable advantage in the competition.

At the same time, I worked as the Project Manager in 2019 season. I was responsible for scheduling the overall development timeline and coordinating different groups. I arranged the weekly stand-up meetings, where members exchanged updates and settled the upcoming schedule. Together with other team leaders, we kept the whole team moving forward.

In the RoboMaster 2019 International Regional Competition, as our debut, we won the fifth place among all the international teams. It's only achievable with the joint effort of the whole team.

From 2020 to 2021, I worked as the Vision Group Lead of the team. I worked on a vision-assisted automatic aiming system, for which I have contributed 27k+ lines of C++ code.

On Windows, most joysticks are connected to games or applications using the DirectInput API. DirectInput is flexible, but it lacks a unified keymap standard. For example, the same virtual key in DirectInput can map to key "A" for one joystick, but to key "B" for another. The NetEase Android Emulator used the DirectInput API. Therefore, the default keymap may not be accurate. Users needed to manually configure the joystick when connecting for the first time.

When Microsoft rolled out their own joysticks—the Xbox controllers, a new interface called XInput was introduced. XInput is more well-defined and easier to use, but only supports the Xbox controllers. There are open-source tools that map other joysticks to XInput. However, they require a lot of manual configurations.

In addition, there was another challenge for the NetEase Android Emulator. As Nintendo Switch become popular, more and more users get their Nintendo Switch Joy-Cons and want to use them in the NetEase Android Emulator. However, as a pair of two separate controllers, Joy-Cons are recognized as two DirectInput devices on Windows. To use the two pieces as one full joystick, the program needs to support reading from two joysticks simultaneously, which is not supported back then. In addition, the advanced functions of Joy-Cons such as the gyroscope and the haptic feedback are not available through DirectInput.

We intended to provide a smooth user experience for joystick users of the NetEase Emulator. Therefore, we introduced a new adapter module. We integrate a community-contributed joystick keymap database of into the system. When a user plugs a joystick in, the module looks it up and applies the keymap automatically, unifying DirectInput, XInput and the Joy-Con-specific interface into an emulated XInput interface. The user can also manually configure the keymap through an intuitive graphic user interface (GUI). For Joy-Cons, two pieces are merged automatically (in fact, the module supports merging arbitrary numbers of joysticks). Gyroscope and haptic feedback are supported. Users can even map the gyroscope to axes or buttons, turning games into interactive ones.

During the course Mixed Reality at ETH Zurich, my teammates and I developed a system for scanning and reconstructing irregular geometries using a mixed-reality headset—Microsoft HoloLens 2.

Our goal was to provide near-real-time 3D scanning and reconstruction. The system consists of a HoloLens 2 and a host computer, connected by a cable or wirelessly. The user holds the object to scan in hands, and starts the scanning process with an eye-tracking button (the user can have both hands occupied). The system acquires depth data through the depth camera on HoloLens 2 and converts them to point clouds. The user is able to see the point cloud overlapping on the object with the mixed reality experience. To scan the complete object, the user slowly rotates the object. The point cloud follows. When the user is satisfied with the result, the scanning process can be stopped by the same eye-tracking button. The reconstructed mesh is further post-processed on the computer.

The whole processing pipeline consists of multiple steps. Without the joint effort of the team, none of these would have been achieved.

The first step is to acquire the depth data. The depth camera on HoloLens 2 can operate in two modes. The long-throw mode is used for environment modeling. The short-throw mode, known as AHAT (Articulated HAnd Tracking) mode, is originally used for hand tracking. We use the latter since it provides high resolution and high frame rates. A Unity-based app is deployed on HoloLens 2, which acquires data through the Research Mode API.

HoloLens 2 has limited computation capability. After some early-stage performance tests, we decided to put the remaining pipeline on a host machine. The depth data is streamed to the host via TCP through a wired or wireless connection. And then, the data is preprocessed. A lot of operations happen in this step, in addition to converting depth images to point clouds. For example, the depth image contains the user's hands. We remove the part using a hand model and the HoloLens 2 built-in hand-tracking functionality that provides coordinates of hand joints.

The next step is to register per-frame point clouds into a global point cloud. We adopt a local ICP (iterative closet points) algorithm. We do not make any assumption about the rotation direction. Instead, we assume small movement between frames and adopt a human-in-the-loop approach. When the algorithm decides the movement is above a threshold, it pauses the registration process and gives visual feedback to the user. Once the user re-aligns the object with the point cloud, the registration resumes.

Finally, once the point cloud is complete, the next step is to reconstruct the mesh. This step is computationally expensive and has a handful of parameters for fine-tuning. Therefore, this step is performed offline on the host machine. It also involves a handful of denoising, smoothing, and voxelizing algorithms.

Learning to program is difficult. As with all topics, students learn more quickly when given immediate feedback tailored to their efforts. However, while university staff—instructors and teaching assistants—are capable of providing such feedback, staff are not available 24/7 and lack the time needed to provide individual attention to each student's programs.

Automating feedback has been an important topic in CS education for years. A simple and widely adopted approach is to use systems that run test cases on student submissions. However, these systems typically fail to provide feedback tailored to each student's effort. For example, these systems cannot give out the test cases but only showing pass/fail results.

To address the need of rapid and tailored feedback, we develop a system for end-to-end automatic feedback. Specifically, we target LC‑3, a RISC-like educational assembly used in the courses at UIUC and other universities. At the heart of the system is KLC3, a concolic (concrete + symbolic) execution engine for LC‑3 built on top of LLVM/KLEE.

At high-level, symbolic execution is a technique that treats inputs as symbols rather than concrete values, operates them the same way as the program executes, and checks the result symbolically. Intuitively, it executes all the possible values concurrently. It provides additional power to the feedback system, not only to perform more thorough tests, but also to generate test cases tailored to the students' programs, which can be given back to the students without undermining the power of the system.

We also built tools around KLC3 into the system to cover the whole workflow of editing, testing and debugging when students do their programming. At the frontend, we develop a VSCode extension that provides real-time feedback to students while they are editing their code, leveraging static analysis to provide warnings of possible bugs or poor styles. At the backend, we develop a fully-fledged in-browser LC‑3 simulator with WebAssembly, freeing students from compiling the Unix LC‑3 tools locally.

We deployed the system to provide feedback for the assembly programming assignments in a class of over 100 students in Fall 2020. Students wrote a median of around 700 lines of assembly for these assignments, making heavy use of our tools to understand and eliminate their bugs. Anonymous student feedback on the tools was uniformly positive. Our work was published and presented at ASE 2021, the 36th IEEE/ACM International Conference on Automated Software Engineering.

Concolic execution is powerful, but the power doesn't come for free. It requires a lot of computation power. A lot of effort has been made into KLC3 to provide feedback in time after students submit their code. Those works also form parts of my Bachelor's thesis.

Nowadays, more and more mobile devices support wireless charging. Among them, the Qi standard is the most widely adopted protocol. Qi hotspots can be found in many public places, such as cafeterias and airports. However, the device being charged needs to be properly aligned with the charging coil (by the time we developed this system, Apple MagSafe for iPhones/AirPods was not a thing yet).

Our goal was to provide seamless wireless charging experience at Qi hotspots like cafeterias, where customers sit around tables. We designed a smart table that automatically aligns wireless charging coils with users' devices, which are detected with computer vision. The coils are moved by a mechanical system like the one used in 3D printers. Up to 3 devices can be charged concurrently. Phone cases with common thickness are allowed. The system is able to detect if a device on the table supports wireless charging with a few trials. If not, the device is ignored.

We designed and implemented the table. Our team won the most Interdisciplinary Project Award of ECE445 Senior Design at the ZJU-UIUC Joint Institute in 2021.

During Fall 2019, as a part of the UIUC ECE391 course, my teammates and I designed and implemented AuroraOS, an x86 operating system. The system was emulated in QEMU.

We implemented the basic kernel features required by the course, including memory management, real-time clock, interrupts, system calls, the user program loader, a read-only file system, terminal, and a round-robin scheduler.

In addition to these basic features, we implemented a handful of extra functions. We implemented a 1024x768 16-bit high color SVGA driver with hardware acceleration and alpha blending. On top of it, we built an interactive graphic user interface to support multiple terminals as well as mouse control. We also implemented an advanced scheduler with wait lists (similar to the one in Linux) and the CMOS clock. To ease the development and debug work, we also employed a cross-compiling toolchain from macOS to x86.

ECE391 hosted an OS design competition every semester. In Fall 2019, our team got the third prize in the competition.

At the Fall 2022 offering of the Advanced Operating Systems at ETH Zurich, my teammates and I built Ozone ("OS one"). Ozone is an ARMv8 operating system, built upon a research OS Barrelfish. Similar to Barrelfish, Ozone is a multikernel OS—each CPU runs an independent copy of the kernel, while the shared states are minimized and intercore communications are pushed to the user space. Being also a microkernel OS, Ozone (Barrelfish) delegates various services to user-space programs, while the kernel provides lightweight inter-process communication mechanisms. It also adopts a lot of ideas from previous OS research, such as the capability system, processor dispatch with upcalls, user-space RPC, etc.

As part of the project, I developed the virtual memory management system. The system supports memory mapping to given addresses, mapping to dynamic addresses, on-demand paging, and unmapping. Combined the buddy allocation algorithm and the red-black tree, all these operations can be done in logarithmic time complexities. Combining with the physical memory system developed by my teammate, which similarly adopts the binary search tree, Ozone is able to achieve efficient memory allocation for processes.

In addition, I worked on the process management system, intra- and inter-core communications, the naming service, etc. We ran Ozone on both QEMU and all four cores of the Toradex Colibri i.MX8 development board.

For the design project of UIUC ECE411 Computer Organization and Design in Fall 2020, my teammates and I designed and simulated an RV32I processor in SystemVerilog. The processor has a 5-stage pipeline with data forwarding and hazard detection. We implemented the Tournament Branch Predictor, combining both local and global branch histories. As for caches, we implemented a parameterized cache module with the pseudo LRU replacement policy, from which L1/L2 I-/D-caches are instantiated. We also implemented a next-line prefetcher.

As the design project of UIUC ECE385 Digital Systems Laboratory in Spring 2020, my teammate and I developed a dual-player video game on the Altera DE2-115 FPGA, simulating the well-known zombie-killing game Boxhead.

We developed a framebuffer-based hardware graphic engine. The engine supports 640x480 16-bit VGA output, 60Hz refresh rate, double buffering, auto background refresh, and direct memory access (DMA). On-chip memory is not large enough to hold the frame buffers, so SRAM is used. One challenge we encountered was that the SRAM is not fast enough to support the full refresh rate. VGA standard requires 25MHz output pixel rate. However, the main clock of the FPGA is only 50MHz. As SRAM does not support concurrent R/W, we only have two read and/or write operations per pixel per frame on average. Excluding one necessary read to generate VGA output, the remaining one write is not enough, since we must overwrite the background before drawing any sprite.

One possible solution is reducing the actual game refresh rate, for example, to 30Hz, half of VGA refresh rate, by switching the game frame every two VGA frames. However, this approach only provides one more write cycle per pixel, as read operations still take half of the time (we must output VGA color at 25MHz). On the other hand, since we are implementing a shooting game, reducing the refresh rate is likely to diminish the user experience.

Our solution to this problem is boasting the SRAM to 100MHz. We generated 100MHz clock signals from the 50MHz input clock with the PLL (phase-locked loop). The read/write operations are carefully distributed so that SRAM can operate in this extreme condition. Offsets are carefully adjusted to compensate for data delay. The 100MHz SRAM controller serves as the base of our high-efficiency graphic engine and allows a 60Hz game refresh rate.

In addition to the graphic engine, we implemented the USB keyboard driver and wrote the game logic in C, running on the Nois II system-on-chip (SoC).