Custom Display Modes in Assembly

Question

I am designing a mini bootloader-kernel combo in 1 file as a fun project, but I have always wondered how to get a custom display mode fitted to the size of my screen, instead of premade display modes. My screen size is 1366x768, and I have looked on ctyme.com's Ralph Brown's List of Interrupts:

int 10h/ah=0 does not seem to work for me because there is no 1366x768 resolution to choose.

int 10h/ax=00F2h also does not have a 1366x768 option. I have considered using the bx=11Bh option, as it is close to what I need for the project with a large resolution and many colors, but I would like to know if it is possible to get the resolution perfectly, like just about every major OS has managed to do.

int 10h/ax=007Eh looked extremely promising on the ctyme website, but when I tried it out...

    [BITS 16]
    org 0x7C00
    mov ax, 0
    mov es, ax
    mov ah, 2
    mov al, 1
    mov ch, 0
    mov dh, 0
    mov cl, 2
    mov bx, stage2
    int 0x13
    jmp stage2
    times 510-($-$$) db 0
    dw 0xAA55

    stage2:
            mov ax, 0x007E
            mov bx, 1366
            mov cx, 768
            mov dx, 256 ;placeholder, was going to replace it with a higher number
            int 10h
            jmp $

...nothing happened. I assembled it using nasm and ran the program on qemu (My OS is the latest version of Ubuntu, 64 bit by the way), by doing:

    nasm boot.asm
    qemu-system-x86_64 boot

I didn't run it on a .iso file because qemu can run bin files, and I was planning to put it on .iso once I got a good amount of progress on the OS.

NOTE:
The pre-made graphics modes work for me (or at least some of them do, like the common but archaic int 10h/ah=0/al=13h)

I wondered if int 10h/ah=007Eh is just another way to load a premade graphics mode, but then I entered the resolution and colors for int 10h/ah=0/al=13h and it still didn't work.

I am aware the int 10h/ax=4F02h/bx=81FFh is said on ctyme to fill the entire screen, but I tried that and nothing happened.

ANOTHER NOTE:
In every instance where a mode did not work, there was no error shown, simply nothing happened. I know when it does work because it clears the original text that qemu writes about booting up when a mode successfully loaded.

Answer 1

Originally there was a variety of different video cards (CGA, EGA, Hercules, VGA.. ); and none of them were ever intended to be any kind of (hardware) standard. With each new video card the "int 0x10, ah=0x00" got extended to support new video modes; and this mostly worked at the time due to software being able to ask if the video card is CGA, EGA, VGA, etc (and newer cards providing limited backward compatibility with older cards).

However, in the late 1980s the market was flooded by a large number of video cards offering more capabilities (e.g. more than 256 KiB of video RAM, resolutions higher than 620*480, more than "256 colors"). When this happened different video card manufacturers started slapping random non-standard modes into the list of video modes that "int 0x10, ah=0x00" could support (which is why most video modes listed by Ralph Brown's list mention which specific video cards support each specific video mode); but it became impossible for software to figure out what the video card actually is and therefore impossible to determine what "int 0x10, ah=0x00" might support (and even more impossible for older software to support newer video cards well).

Eventually (very early 1990s) this disaster was fixed by the Video Electronics Standards Association (VESA) with a set of BIOS extensions known as "VBE" ( https://en.wikipedia.org/wiki/VESA_BIOS_Extensions ). Essentially, the woefully inadequate "int 0x10, ah=0x00" was kept for backward compatibility; but mostly superseded by functions (all sharing the "int 0x10, ax=0x4F" range, with al as sub-function) to determine the video card's capabilities (and get a list of video modes it supports), determine what each mode number actually is, set a video modes, and to do other things (e.g. bank switching, etc).

However; when using the more modern VBE you're still restricted to whatever video modes the video card felt like providing, and it might not provide a 1366x768 resolution.

The only way to bypass that limitation is to write a few thousand different video card drivers; starting with asking the monitor what it actually supports. Note that this requires understanding of the difference between "a video mode" and "a video mode timing" - there are millions of different video mode timings (with different vertical sync width, different horizontal/vertical margins, etc) that all work out to "1366x768 @ 60 Hz refresh rate" and out of those millions of possible video mode timings your monitor might support one of them; and if you get the timing wrong it won't work (and you probably won't know that it hasn't worked).

In other words; you can expect to spend several months getting the "monitor compatibility" working (learning about the relationships between video timing values and parsing the monitor's EDID to figure out what it supports); followed by 1000 months (writing 10000 very minimal drivers for 10000 different video cards); but while you're spending 83+ years trying to do this the video card manufacturers will keep releasing more new video cards that you don't support.

Basically, custom display modes (in any language) isn't even slightly viable for a lone developer (you'd need a team of 100+ developers); and isn't really suitable for a "mini-boot loader/kernel" either (you'd want a full kernel, preferably with device enumeration and video driver interfaces to support third-party device drivers as "run-time loadable" modules to make video driver development less painful).

I am aware the int 10h/ax=4F02h/bx=81FFh is said on ctyme to fill the entire screen, but I tried that and nothing happened.

Ralph Brown's Interrupt List just gives a very minimal/terse description (and is mostly "stale" - the information hasn't been updated for about 25 years now). It is not an adequate substitute reading the full VBE specifications (that can be obtained easily from the links at the bottom of the Wikipedia page: https://en.wikipedia.org/wiki/VESA_BIOS_Extensions#Further_reading ).