The 4 grids are supposed to solve to a unique set of solutions. If 1 clue from a group can be swapped with another in a different group, without changing the solution of either group, then it is not unique.
The more red herrings the better, but I can see at least 4 solutions, using the 16 clues given, that fit your solutions.
Please bear this in mind.

Really well put together, guessed all connections but could only get 1 because they all fit into each others groups

The downside of trying for many crossovers is the loss of uniqueness. Also I find your categories too obscure. Nice idea, though, all colours.

clearly very clever but far too obscure to even get started and more than one solution which made it even more impossible