The Ballyclare player shared the lead with winner Marcus Fraser at the turn but lost his way down the stretch.
Maybin finished four shots behind the Australian in joint second on eight under after a 72.
Graeme McDowell, who won the tournament in 2008, ended in tied 28th on one under after a 71 while Shane Lowry's 75 left him on one over.
Maybin birdied the first, fourth and ninth to join Fraser at the top but then his problems began.
After a bogey at 10, he hit his approach into a bunker at 12 and missed his par putt from inside six feet, and although he atoned with a 20-foot birdie at the next, he dropped another shot at 16 and lost outright second place at the last when his putt lipped out.
"I didn't really come in with a whole lot of form, so to finish second is a great result any time," the 29-year-old said.
"I didn't play the back nine great so there are a few things there that need a bit of improvement but overall happy I'm enough."
Danielle McVeigh had to overcome a nasty dose of food poisoning before clinching her second top-class victory in 10 months on Scottish soil.
McVeigh played brilliantly on the back nine at Royal Troon to succeed Leona Maguire as Helen Holm Scottish Amateur Open Strokeplay Champion.
Winner of the British Ladies Amateur Championship last summer, the 22-year-old wrapped up a two-stroke victory over England's Nikki Foster on 10-under par after a 74 on the Championship links at Troon.
