The gray pieces of lint are from my dryer. I put them out for the birds to use, just didn't think about the sparrows using it too.

It is for this reason that I take out all sparrow nests.
"Male House Sparrows. House Sparrows are aggressive, and will attack and destroy bluebird eggs, nestlings and adults. It is better to have no nestbox at all than to allow House Sparrows to reproduce in one."