Your scheme sounds like multimastering, and that is always a pain.

Two ideas come into mind, but I admit, both need to elaborate.

First, implement bus arbitration by protocol - either using a master/slave scheme with master periodically polling, or a token passing method. Both can achieve big gross data transfer rates but both will suffer from latency - the implementation together with the requirements of the application determine, if latency is acceptable.

The second idea is simpler, make two single-direction bus 4-bit-nibble-wide data (plus 2 control each - strobe and acknowledge).

Tons of fun... :-)

Jan Waclawek