As Erik suggests, these days it can be more economical to use a number of microcontrollers to solve your problem. Many times the microcontroller is a cheaper alternative to special ics. However, a classic solution might be to use a bcd->seven segment driver like a 4511 (4 port pins), 4 select lines (1 for each of the displays) to drive a transistor as well as connecting to each pushbutton switch. The other connection to the pushbuttons are connected together and go to another port pin. Nine port pins in total.

Is 3mS sufficient to refresh the displays? If you refresh all four displays faster than around 50 times a second then you avoid flicker problems.

Erik's solution also has possible positive side effects as you would only need 4 wires (0v,+V,SDA,SCL) between the main micro and the keypad/display micro. Easier to wire and a module you can re-use on other projects.

Also consider using a LCD module. These are quite economical these days and allow you to have a more informative MMI. Running in 4 bit mode and tying R/W to 0v, the control lines are EN and RS (6 in total). Another for turning the backlight on/off and one other for the pushbutton switches common (the other side of the switches go via a diode to the 4 bit data on the LCD). Thats used 8 bits.