Autonomous Dynamical Systems (DS) has emerged as an extremely flexible and powerful method for modeling robotic tasks. Task execution of DS models is typically done in an open-loop manner in combination with standard low level controller, e.g. position controller or impedance controller. Such an arrangement has two important drawbacks 1) it is not passive and 2) the DS model can not respond to physical perturbations on the robot body. These are severe limitations in tasks with uncertain physical contacts, e.g. object handovers. We propose a novel control architecture that closes the loop around the DS, ensures passivity and allows tuning of the impedance. We evaluate our approach in a comparative study in an uncertain manipulation task with unexpected contact.