Pick and Place Robot
MEAM 5200: Introduction to Robotics | Spring '25
Introduction
This project served as the final challenge for Penn’s graduate course on serial manipulator robotics.
I developed the software stack for a 7-DOF Franka Emika Panda arm to autonomously grasp, transport, and stack both static and dynamic blocks on a target platform within a four-minute time window.
The task required real-time perception, robust grasping, repeatable placement, and collision-free motion generation under time and workspace constraints.
The core of my approach emphasized manipulator controls and planning. I implemented numerical inverse kinematics with warm-start optimization, used the arm’s Jacobian to compute manipulability and avoid near-singular configurations, and designed deterministic yet safe motion sequences with adjustable approach and retreat offsets.
While the challenge allowed simple hard-coded motions, I framed the solution around principles used in real manipulation systems: constraint-aware motion generation, robust grasping strategies, and perception-to-control integration.
Pre-grasp alignment and approach vector
Orientation variability for static objects
Design Process
Approach
I structured the system around a modular manipulation pipeline:
perception → pose estimation → IK → trajectory execution → grasp → placement.
The motion subsystem was built around forward and inverse kinematics derived earlier in the course, and extended with null-space optimization to maintain high manipulability and avoid poor configurations.
The software architecture separated responsibility across two main components:
ArmController for kinematics, planning, and execution; and
ObjectDetector for AprilTag-based pose estimation and workspace monitoring through an overhead camera.
Static Blocks
Static blocks were handled with a structured top-down grasping sequence optimized for repeatability.
I used predefined scan poses that maximized AprilTag visibility, then generated approach trajectories using Cartesian interpolation followed by IK solving with joint-limit and orientation constraints.
Each placement pose included collision-free approach and retreat motions. Layer heights were parametrized to support flexible restacking or recovery actions.
Dynamic Blocks
Dynamic blocks rotated on a turntable, introducing timing constraints and changing relative orientations.
I implemented a hybrid reactive strategy: the robot maintains a ready pose in the turntable frame, then synchronizes the end-effector pose to intercept the block when its orientation falls within a feasible grasping window.
Once grasped, blocks were integrated into the same structured stacking pipeline used for static blocks, with roll corrections determined from the detected AprilTag pose.
Evaluation
The robot was validated in both simulation and hardware using metrics aligned with manipulator system evaluation: grasp success rate, trajectory feasibility, cycle time, stack stability, and behavior under pose estimation noise.
Simulation results yielded consistent maximum scoring runs with four static and two dynamic blocks.
On real hardware, we achieved three reliably stacked static blocks in 2 minutes and 51 seconds.
The discrepancy underscored typical sim-to-real gaps, camera calibration drift, friction differences, and contact uncertainty, which deeply informed our design choices.
Analysis
The structured manipulation pipeline proved robust for static objects but highlighted the importance of camera calibration, workspace modeling, and timing for dynamic interactions.
Block misses were often attributable to small errors in object pose estimation or changes in lighting conditions that degraded AprilTag detection.
The project emphasized a key lesson from real-world manipulation:
trajectory planning and kinematics are only as reliable as perception and calibration allow.
Even sub-centimeter differences between simulation and hardware affected grasp stability and approach paths, reinforcing the need for parameter tuning, redundancy, and fail-safe states.
Technical Takeaways
- Null-space optimization and manipulability metrics significantly improve robustness near workspace boundaries.
- Structured motions and conservative offsets reduce execution risk but require precise calibration to generalize.
- Sensor noise and pose estimation drift propagate directly into grasp success: perception must be tightly integrated with planning.
- Sim-to-real transfer requires explicit modeling of friction, delays, camera uncertainty, and contact mechanics.
- Testing on hardware early exposes failure modes that planning or IK alone cannot predict.
