Well Haruki explained it to me like this: You're basically asking for the Spirits' help.

If you think of it that way, they're beings right? So a Filial Spirit can either comply with or deny your request. And if they can do that, then yeah. I guess they can get mad.