They had a proprietary decryptor chip that sat between the program roms and the CPU. The decryptor used a table held in a small SRAM that was powered by a battery. When the SRAM lost power (battery died, or someone was trying to reverse engineer the board), the decryption key would be lost.
We broke it because someone noticed that the FC1 pin was tied to the drcyptor chip. In retrospect this made sense because only the program roms were encrypted, not the gfx roms (FC1 and FC0 pins were 01 for data access and 10 for program access [1]). Whenever the 68000 was requesting an instruction, it would set FC0 low and FC1 high, and when requesting data it would be opposite - this is how the decryptor knew when to decrypt.
But there was an achilles heel!
If you use PC-relative addressing, this ALSO causes FC0 low and FC1 high (because PC-relative is assumed to be accessing more program code). So all you had to do was fetch each address using PC-relative MOVEs, and it would happily give you the decrypted instruction words.
But then there was a secondary protection scheme: The decryptor chip would mysteriously shut down after a short while! WTF!
It wasn't until Capcom released their home console based off the original CPS board, along with backports of games like Street Fighter Zero, that the penny dropped. These backports were full of mysterious reads of weird addresses that made no sense.
It turns out that these weird addresses were what the decryptor chip was waiting for on the address bus. If no such address appeared within a timeout period, the chip would stop decrypting until you reset.
But the guys backporting CPS2 games didn't take the instructions doing these accesses out - even though the original CPS board didn't need it (probably it would have been a lot of work), so that secret leaked out.
After that, it was a simple matter of writing a program that read via PC-relative addresses, sent those weird accesses from time to time to keep the watchdog happy, and then output to a dumper via IO pins.
[1] Section 3.8: PROCESSOR FUNCTION CODES (FC0, FC1, FC2)
The 68k was unique in that it included in its design the ability to act as a Harvard Architecture processor (where instructions and data have separate signal pathways rather than being together in the same address space [1])
It did this via the function code pins, which signaled differently depending on whether the chip was requesting instructions or data, in supervisor or user mode. No other consumer-grade chip had this as far as I know.
Sega's System 16 was also 68000 based (as was the Megadrive/Genesis), but I don't know anything about their protection scheme.
The 65816 has valid data address (vda) and valid program address (vpa) pins which are essentially the same as fc0/1/2 , minus the supervisor/user mode support.
vda vpa
0 0 internal operation, address bus may be invalid
0 1 valid program address, may be used for program cache control
1 0 valid data address, may be used for data cache control
1 1 opcode fetch, may be for program cache control and single step control
No other consumer-grade chip had this as far as I know.
The 8086/88 has status outputs which provide similar information, including which segment register is being used (CS, DS, ES, SS) and the type of bus cycle (interrupt acknowledge, read I/O, write I/O, halt, opcode fetch, read data, write data).
They had a proprietary decryptor chip that sat between the program roms and the CPU. The decryptor used a table held in a small SRAM that was powered by a battery. When the SRAM lost power (battery died, or someone was trying to reverse engineer the board), the decryption key would be lost.
We broke it because someone noticed that the FC1 pin was tied to the drcyptor chip. In retrospect this made sense because only the program roms were encrypted, not the gfx roms (FC1 and FC0 pins were 01 for data access and 10 for program access [1]). Whenever the 68000 was requesting an instruction, it would set FC0 low and FC1 high, and when requesting data it would be opposite - this is how the decryptor knew when to decrypt.
But there was an achilles heel!
If you use PC-relative addressing, this ALSO causes FC0 low and FC1 high (because PC-relative is assumed to be accessing more program code). So all you had to do was fetch each address using PC-relative MOVEs, and it would happily give you the decrypted instruction words.
But then there was a secondary protection scheme: The decryptor chip would mysteriously shut down after a short while! WTF!
It wasn't until Capcom released their home console based off the original CPS board, along with backports of games like Street Fighter Zero, that the penny dropped. These backports were full of mysterious reads of weird addresses that made no sense.
It turns out that these weird addresses were what the decryptor chip was waiting for on the address bus. If no such address appeared within a timeout period, the chip would stop decrypting until you reset.
But the guys backporting CPS2 games didn't take the instructions doing these accesses out - even though the original CPS board didn't need it (probably it would have been a lot of work), so that secret leaked out.
After that, it was a simple matter of writing a program that read via PC-relative addresses, sent those weird accesses from time to time to keep the watchdog happy, and then output to a dumper via IO pins.
[1] Section 3.8: PROCESSOR FUNCTION CODES (FC0, FC1, FC2)
https://www.nxp.com/docs/en/reference-manual/MC68000UM.pdf