Assuming you're talking about a Unix machine running X Windows, perhaps the
Direct Rendering Infrastructure would help you find what you're looking for?
There are probably similar animals for Windows and OS X, but I can't tell you what they are off the top of my head.
Every device is different, so even if you manage to get something working for one set of hardware (there still is memory-mapped I/O graphic hardware out there), it likely wouldn't work on another system unless it was identical to the one you developed this code on. This is why things like X Windows were invented; to put you the programmer at a remove from the vagaries of the hardware.