Hi Michael,
I am not using any field as a unique identifier. And YES, there are multiple records attached to one contact. So maybe that is the problem.
The work around I used was to use "Compare Contact Field" in the Dynamic Content rules rather than "Has Linked Record in Custom Object". This was a bit of extra effort since I had to update a contact field as well for this to work.
Regards
Apoorva