Dear SSG,
Since the original invitation had a space for number of guests, this was obviously taken as an invitation to bring guests. The couple probably felt entitled.
Also, since you already asked them about it, your options are limited for handling this gracefully. Actually, there is no graceful way to exclude children, unless a tactfully worded invitation went out in the beginning.
Don't get me wrong, this isn't a matter of fault. The real issue is that the couple should have asked if children were invited before automatically including them on the RSVP.
You also need to figure out how important it is for you to have the couple at your wedding. They are grandparents and were looking forward to having their grandchildren for the weekend. So if they cannot bring the children, they likely will not show.
You might want to be prepared for another surprise. Unless you mentioned no children or adults only in someway on the invitation, then you may have other couples show up with children. Many people do not think of children as other guests.
In addition, many people look at weddings as family affairs and thing that children are automatically invited with the parents.
Would you turn people away at the door if they brought their children?
My recommendation comes from the Gracious Host.
The gracious host, since you have already mentioned it, would be to call and apologize, but that you had not prepared for children, that all the activities are adult activities, and you do not have a care provider. That would put the ball in their court. However, you need to be prepared that they may decline to come or tell you it is no problem, they will handle the children.
The gracious host then would hire a babysitter to handle the drop in kids. I am sure you will get a few kids, even if this couple decide not to come. Just set a side a room designated as a game room for kids. Perhaps you can get a teenager to babysit.
YOUR PAYMENT AND BONUS IF ANY MAY BE TAX DEDUCTIBLE

Edward M. Johnson