Conventionally, with a heap from which memory is allocated, the programmer must explicitly deallocate memory when it is no longer needed. Typically there are two complementary functions, alloc to allocate memory, and free to deallocate the memory. (C++ note: the corresponding statements in C++ are 'new' and 'delete' respectively.)
Briefly, garbage collection is a way to automatically deallocate memory, instead of program explicitly asking for memory to be deleted. The programmer need only alloc memory, and need never call free, since the system automatically tracks the use of all memory blocks in the system an deallocates blocks automatically when they are no longer used.
Garbage collection works by the system keeping track of references to each allocated memory block. While there are references (pointers) remaining which point to a block, it is kept in memory. Once there are no remaining references, the block should be deallocated.
Garbage collection depends heavily on the language being used, so no general-purpose operating systems use it---but some OSs which are designed for a specific computer langauge do use it: for example the Oberon system (which is designed for the language Oberon 2) and the JavaOS (for the language Java).
Most operating systems do not provide heap garbage collection, so mostly garbage collection must be implemented in the runtime system of the particular computer language. (For example, when the Java interpreter runs under UNIX, the Java interpreter itself is responsible for garbage collection.)