Born in Kansas City, Missouri, Gadson played with the first line-up of Charles Wright's Watts 103rd Street Rhythm Band, and recorded three albums with them between 1968 and 1970. Along with other members of Wright's band, he went on to appear on many hit records including Dyke & the Blazers. Gadson started to become well known as a drummer following the release of the album Still Bill, by Bill Withers, released in 1972. He played on The Temptations album 1990, released in 1973. In 1975, he played with Freddie King on Larger than life and went on to record with Martha Reeves, Randy Crawford, Quincy Jones, Herbie Hancock, B.B. King, Albert King, Rose Royce, Elkie Brooks and many more household names. Gadson was also the drummer on Marvin Gaye's "I Want You" and appeared on two tracks on the 2005 Paul McCartney album "Chaos and Creation in the Back Yard", "At The Mercy" and "Riding To Vanity Fair".
In 2009, Gadson joined Willie Dixon's grandson, Alex Dixon on the release of his Rising From The Bushes, in which he appeared on two tracks, "Fantasy" and "Spoonful". Later that year, Gadson joined Beck, Wilco, Feist and Jamie Lidell covering Skip Spence's Oar as part of Beck's Record Club series, with videos appearing on Beck's website beginning November, 2009. He has drummed on Beck's albums Sea Change, The Information and Morning Phase, as well as Jamie Lidell's 2010 album Compass.