A kernel is supposed to be a small set of necessary code to boot and manage some hardware. It's host to drivers, applications and UIs. Thus the name, 'kernel'.

To compare a driver with the kernel proper, seems very strange. There are other large codes in the world too.