Preprocessing sure. But the different formats are necessary. Depending on the data in the texture, the compression algorithm may alter the data in a way that behaves better or worse (for example, textures representing normals vs material vs albedo).
Before I discuss rendering with you, can you first disclose your experience with game engines or rendering? It's sort of pointless to go into that discussion unless you understand how things have changed in the last 5-10 years or so.
edit:
Forgot to mention that shipping with a scripting engine is sort of easy. It's just a matter of whether it's exposed to the customer or not. Tons of games (even decade old ones) have shipped with turing complete editors for modding or making custom levels or what have you. I've written a custom VM for a homebrew project that can do precisely this.
Sure. Started out with some school projects that were beautiful object-oriented single-thread vistor-pattern for rendering in fixed-function OpenGL. This predictably led to terrible performance. At the end of school, friend and I decided to extract that code and try to build a real engine around it.
Upgraded to thread-safe rendering in OpenGL, one thread handling rendering and windowing tasks, other threads game logic and submitting draw requests, and ultimately other threads handling resource loading onto the GPU (though that came later towards the end).
Upgraded that to actually use the programmable pipeline, and finally added support for FBOs and uniforms and buffers and everything so we could actually start on deferred rendering as outlined in Engel's blog and other places. Never did transparency, but if I remember correctly the goal was to probably do something like screen-door+deferred. For the life of me I can't remember the paper we were looking at.
Eventual goal was to do full deferred lighting and PBR, but by that point I'd moved over to web development. Every time I went back to that codebase, clean and organized as it was, it was just such a pain in the ass to do anything compared to splatting some JS and Three onto the screen and being done with it.
Concurrent with that, I was doing 3D CAD software development in Java with Ardor3D. So, big annoying retained-mode scenegraph with all the BVH nonsense you could ever want, and a bunch of legacy files for things that, when one rendered a building all at once, would kill the graphics card. And don't even get me started on trying to sort out order-independent transparency for that...one of my great failures.
By contrast, nowadays I do multiple web workers with websockets to provide 30-60 fps display of waveform data on just bare 2D canvases, with Angular wrapping the components.
EDIT:
I'm aware of the scripting thing. Note that, until perhaps FarCry, it was pretty well held that scripting languages were "too slow" for most games, nevermind the success of such things as JIT Quake 1 bytecode, or the stuff in Out Of This World, or what have you. Even UnrealScript was considered godawful dog slow and to be avoided if possible, and was finally shown the door. But, in the end, scripting was discovered not to be the super worst thing ever.
I think Eve finally hammered that point home to everyone.
The scripting languages we have now have come a long way in terms of ease of embedding, with a possible exception for Lua, which has been easy as hell for a while now.
Then again, I don't think that many scripting languages have the sheer size of interface with native code that, say, JS in the browser does.
It's a shame you didn't continue. Where you left off (prior to deferred ... shading I'm assuming and PBR) is still a bit a ways from where things get a bit more interesting. Physically correct rendering changes the game a good deal. We can't haphazardly blur textures and add things and lerp like we used to.
Before I discuss rendering with you, can you first disclose your experience with game engines or rendering? It's sort of pointless to go into that discussion unless you understand how things have changed in the last 5-10 years or so.
edit:
Forgot to mention that shipping with a scripting engine is sort of easy. It's just a matter of whether it's exposed to the customer or not. Tons of games (even decade old ones) have shipped with turing complete editors for modding or making custom levels or what have you. I've written a custom VM for a homebrew project that can do precisely this.