Failed to Join Game Message

Merry Christmas!

I am hoping that someone may be able to help with this, I have 2 XB 1's in my house, both have Disney Infinity 3.0 and the Speedway game piece. I am trying to link the games so that both XB1's can play in the same game but every time one invites the other, it ends in a "failed to join" message.