We had a bunch of these kinds of "follows", where ultimately the fish would see us and turn away. It helped when we had the sun at our backs.