We used a set of INT instructions in well-known low memory addresses that all jumped to the same place. We had an ASM file that you linked with, that had sixteen different address combinations for each.
The common entry point would look back on the stack and calculate from the return address which entry point had been called, and run the appropriate kernel call. We called it the CS:IP hack.
In the context of this post, the DOS INT10 and INTx(I forget) required the caller to load registers with the desired system call number, then perform the trap instruction in their code. Fortunately CTOS didn't need those particular software interrupts, so I could implement them for my purposes.
Windows 95 used a related hack. Whenever a v8086 program asked to create a call to protected mode code ("please give me a real mode address to call to, in order to start executing the protected mode routine at address 0x123456"), Windows would store the entry point in a table and hand out real mode addresses like FFD0:0, FFCF:10, FFCE:20, FFCD:30, FFCC:40 that all point to the same instruction (because the segment part is shifted left by 4 in real or v8086 modes).
The routine at 0xFFD00 could then enter protected mode and use the code segment to build the index into a table of entry points: FFD0 goes to index 0, FFCF goes to index 1, and so on. But for extra kicks, the address isn't actually pointing to valid code. It points to a random "c" character in the BIOS, which is an ARPL instruction - which in turn is invalid in v8086 mode and therefore invokes the undefined opcode exception handler. The exception handler, which handily enough is already running in protected mode, then takes care of doing the 32-bit call.
> It so happens that on the 80386 chip of that era, the fastest way to get from V86-mode into kernel mode was to execute an invalid instruction! Consequently, Windows/386 used an invalid instruction as its syscall trap.
I also read this part but I wonder how did they benchmark back then?
> Schulman’s Unauthorized Windows 95 describes a particularly unhinged one: in the hypervisor of Windows/386 (and subsequently 386 Enhanced Mode in Windows 3.0 and 3.1, as well as the only available mode in 3.11, 95, 98, and Me), a driver could dynamically register upcalls for real-mode guests (within reason), all without either exerting control over the guest’s memory map or forcing the guest to do anything except a simple CALL to access it. The secret was that all the far addresses returned by the registration API referred to the exact same byte in memory, a protected-mode-only instruction whose attempted execution would trap into the hypervisor, and the trap handler would determine which upcall was meant by which of the redundant encodings was used.
And if that’s not unhinged enough for you: the boot code tried to locate the chosen instruction inside the firmware ROM, because that will have to be mapped into the guest memory map anyway. It did have a fallback if that did not work out, but it usually succeeded. This time, the secret (the knowledge of which will not make you happier, this is your final warning) is that the instruction chosen was ARPL, and the encoding of ARPL r/m16, AX starts with 63 hex, also known as the ASCII code of the lowercase letter C. The absolute madmen put the upcall entry point inside the BIOS copyright string.
(Incidentally, the ARPL instruction, “adjust requested privilege level”, is very specific to the 286’s weird don’t-call-it-capability-based segmented architecture... But it’s has a certain cunning to it, like CPU-enforced __user tagging of unprivileged addresses at runtime.)
Ahhh.. probably my first program. Don't forget the int 20 at the end! It was beeping great. Still never unlocked the mysteries of those TSR programs though.
It's been a long time since I've touched any of this, so the details have slipped my mind. However, the general idea was that there were two different exit calls in DOS: terminate and terminate and stay resident. The difference between the two is that the stay resident option wouldn't release the memory used by your application. Further, the interrupt table, which told the processor how to handle each interrupt, was in RAM and therefore writable.
So, what TSRs would do is overwrite one or more interrupts to point to a routine that would check if the system call in question was one it wanted to handle (eg, to add a hotkey it would grab the keyboard handler and check for a special set of keys before passing control back to the normal handler). Once that was fine, it would call the TSR system call and control would be passed back to the OS with the hook still in place
Still never unlocked the mysteries of those TSR programs though.
I made a bunch of those, in TurboPascal. Just needed to save registers (including stack and heap segments) and hook some key combination. One of them was used commercially for installations by a very big company.
Testing was a little prone to spectacular failures. But once the general procedure was debugged, it was easy as pie.
We used a set of INT instructions in well-known low memory addresses that all jumped to the same place. We had an ASM file that you linked with, that had sixteen different address combinations for each.
The common entry point would look back on the stack and calculate from the return address which entry point had been called, and run the appropriate kernel call. We called it the CS:IP hack.
In the context of this post, the DOS INT10 and INTx(I forget) required the caller to load registers with the desired system call number, then perform the trap instruction in their code. Fortunately CTOS didn't need those particular software interrupts, so I could implement them for my purposes.