Not exactly - there are very clear areas where everyone agrees the dividing line exists when you look a full spectrum map. Even most colorblind will agree with the areas in general (there are lots of specific color blind types but most will agree what area of the map is which colors even if you don't put any scale indications on the map)

No. Babies don't know how to color bin. They learn the color binning of their language. There are a few languages with different color bins and people who grow up in those languages bin colors differently--having a hard time telling apart colors we obviously see as different (but which map to one bin in their mother tongue) but telling apart subtle stuff we see as one color but they see as more than one.

There are languages with more colors but the bins are the same.

Not exactly. It's more like the bins get added in a more-or-less predictable order as the language adds bins.

Within a particular culture that may be true, but for example the Japanese concept of blue/green is decidedly different from most Western concepts which consider blue and green separate colors.

Even then though we agree on the zones.

Until my wife and I are picking out home decor. Then all of a sudden nobody can agree on what color something is.

The dividing lines are the purest form of social construction. "arbitrary" for some people and "everyone agrees" for others.

I argued over a color names with a guy who later admitted he is color blind. So, no, we don't agree on zones. I mean, it was rather clear he is off. Basically, he has seen different color.