Your Agents Are Only as Good as What You Tell Them to Care About