Hmm... I would never deliberately give my boys dry wall..... But somehow over Christmas break last year, Bryson managed to chew a very large hole in my dry wall. I freaked and watched him very closely and he was fine.
From my experience working with my grandfather, who is a kitchen and bath designer by day and handyman/farmer by night and weekends, drywall is very powdery and he always wears a mask when dealing with it. I believe plaster is involved in making it, and plaster is harmful if inhaled as a powder. It contains crystalline silica which can cause lung disease if inhaled long enough.
There is just so much stuff in there such as boric acid and fiberglass. It won't be fatal if ingested in small amounts, but definitely would over time. It also has something called gypsum in it which is a powder and if it becomes wet, it expands and clumps which harden. Just imagine what that would do inside their stomach.