Then again, I know of plenty of adventure games that had action elements to it. To keep things from feeling too jarring though, you might want to consider making those points occur only when necessary, and maybe nix the "health bar" altogether. Make those sections either "instant death" or have a certain number of repeat tries where it is described how many shots you have left in that segment before meeting certain doom. What you should also do is make sure you either have checkpoints before each of those sections OR allow saving anywhere. That way it will decrease the frustration any players may have.
Also, make sure that the segments don't "sneak up" on the player: as Kyatric mentioned, adventure games are meant to be taken a bit slower with a bit more thought. Allow the player to take in the obstacle first before jumping into it, to think things through. If that's impossible for the scene, then make the solution to the action obstacle instantly recognizable.
Another way to make it feel less jarring in the "action" segments is to make sure the controls match with the controls from every other section of the game. If you swipe to move some curtains aside or swipe to wipe away cobwebs, maybe swipe to blow out a candle, then swiping to shoo attacking birds away would likely feel just as natural. If you're tapping the screen to throw a rock at a button earlier in the game, then later in the game if you're throwing a rock at a monster it's going to feel just as natural and feel like an extension rather than a completely different gameplay concept.
Just a few thoughts on the topic. Don't take my words as hard fast rules, they really are just suggestions.