It's just about poking at it and being persistent, until you figure everything necessary out. I've done similar reverse engineering efforts with a PoS terminal and an e-book reader/eInk display protocols. You don't know much at the beginning. Reverse engineering is mostly about persistence.

It may not always get you to the finish line, but without any documentation, persistence and some tools is all you have anyway. Knowledge is acquired during the process.