One easy way to detect a plot device in a story is to consider whether something has the effects and consequences that would reasonably flow.
There's a thunderstorm that keeps the hero in his home one night. But if the next day, there are no downed trees, or flooded streams, or even leaves scattered over paths, the reader with a discerning eye can pick out that the storm was just a plot device to keep him home. Even without a discerning eye, it may bug the reader.
But not all plot devices can be discerned this way, and yours, too, can avoid it.
Carry through on your plot devices. If there is a storm, let the location be storm-racked the next day. If your character has mastered sleight of hand to pull off a card trick and impress the crowd at the inn, have the crowd gather round him and demand more and talk about it the next day. If a character is wounded, arrange for medical treatment and plausible recuperation time, and possibly permanent effects from scars to being crippled. If characters posing as old women and beggars habitually test young women, whether princesses or peasants, and when they find them courteous, let a pearl or diamond fall from their mouths with every word, pearls and diamonds will cease to be that valuable.
Skills and interests particularly need this. Your sleight-of-hand character has to keep in practice, or he will go rusty. If he has an interest in it, he will do so, and anyway, it gives him something to do in idle moments. A character is unlikely to play at cards exactly once, when the plot needs it, if there are other changes.
This goes double for plot devices that establish character. One piece of sage advice is that experts tend to be passionate about their subject, which gives motives for exposition. Consequently you could use a scene for reasonable info-dumping if the hero mildly observes that the ejection switch was introduced when spaceships used fission drives but still lingers on despite the conversion of fusion drives, and the chief engineer hears. A long rant about how ignorant people are and how even in the fission days the switch was mainly used to detach the drive for repairs, but people just watch their dramas and think it's the only use is for ejection of the fission drives going critical. (Or perhaps a rant against laws that refuse to change when circumstances do.) A lot of information, including tangential knowledge, can be dumped with little more than another character rolling his eyes and murmuring about how the hero got him going, BUT you have established that the chief engineer is passionate and prone to rants on the topic. Portraying him in all other scenes are mild and peaceable will give the game away.
This sort of long tail does limit their use for setting up plot twists. Foreshadowing is added by it, of course. A trivial display of abilities sets up that the ability will be important later, since you showed it, and thus makes it harder to surprise readers -- or characters -- when your character does it. Hence, the spy in the party will warn the Evil Overlord that your character with sleight of hand needs to be chained with particular care.
Fortunately, the plot device effect tends to wear off. A storm can be swept up from in a week or so unless you, say, damage a significant building, and even there, the repair goes on. The wounded character recovers and learns to cope with the injury. The townsfolk slowly stop pointing out the amusing sleight-of-hand man. The chief engineer runs out of steam and then is distracted by repairs, and possibly the only sign of his temper is shouts from the engine room.
Furthermore, their plot device use does help with distraction. The purpose of the sleight-of-hand plot device is to impress the townsfolk with the character -- or to make them hold him in contempt for his failure or the assumption he must be dishonest to know it. This helps lull the suspicions of the readers, because having served one purpose, they sink back into the background. When you pull them out for another -- the character is so deft that even heavier chains do not challenge him -- it helps with the surprise. Though it helps even more if the second use twists it in some way: the Evil Overlord puts the sleight-of-hand character in chains, only to realize that was the distraction while the other characters, who taught him the sleight-of-hand in the first place, escaped theirs.
Or perhaps the Evil Overlord is only distracted from what the others are doing, because none of them could escape by sleight of hand. It's amazing what can be done, but only if you carry through.