As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing.

In one instance I was reading a spec for an industry-specific protocol. At first I was looking at it and thinking "What the... they are padding stuff in a strange way, and using little-endian for the data". Then it suddenly dawned on me, that they've designed the spec to be whatever GCC on x86-64 Linux machine would do to layout C the structures in memory.

So someone very lazy could just define a struct and cast into it as data comes from the wire. Someone one a big endian machine, would have to do a lot more legwork to get the thing working. But given that there aren't many of those around, it was deemed an acceptable tradeoff.

I'm genuinely curious; what was the 'former' protocol? Was it encoded using FAST or zlib or something?

http://www.eurexchange.com/exchange-en/technology/t7/system-...

Specifically, compare the Enhanced Market Data Interface protocol (described in "T7 Market and Reference Data Interfaces") and the Enhanced Order Book Interface protocol. EMDI is FAST, a morass of tags, stop bits, presence bitmaps, and who knows what else (i don't). EOBI is structs.

In a previous job, the software I worked on had a "file format" that just consisted of dumping a raw array of structs to disk. It was obviously pretty fast, but it made cringe a lot.

What's worse, the compiler had flags to control if and how much struct padding to use, so reading a data file with a binary compiled with different padding from the one that created that file caused it to crash. Fun times...

[1] Unless you happen to access a field spanning two cache lines and miss, in which case the padded version would require accessing that second cacheilne anyway.

Unfortunately, the default for the compiler we used[1] to align struct members on DWORD (32-bit) boundaries, for some technical reason I am sure was totally reasonable. And these guys all built their code with a special make-variable that appended the "don't-pad-structs-I-really-know-what-I-am-doing" flag to the compiler's command line.

I did read somewhere, that x86 (at least Pentium III and later) like to make memory reads from addresses that are a multiple of four. But the profiling data I was able to gather showed that that part of the application had a negligible impact on overall performance. Since the vague job description I had gotten said my job was to "make things faster", I decided not to look into this any further.

[1] OpenWatcom (http://www.openwatcom.org/)

Even PowerPC64 has a little-endian mode - PPC64LE.

The machines are switching over because the expensive part is turning out to be software which is already working & the costs associated with migrating it over to a different endian system.

I wonder where this started... I always suspected it was Sun with their "The Network is the Computer" motto, and of course they would define "network byte order" as what they used.

Today, it drives me crazy that we're constantly swapping before sending across the network and then swapping again when it's received.

https://retrocomputing.stackexchange.com/questions/2652/when...

The internet was big-endian from the start, and that was probably because DEC machines were big-endian.

Therefore, it won't be feasible to try to re-introduce big endian to systems that need to be competitive at rendering the Web.

Big endian will stay alive for the time being on home routers and Sparc servers, and probably for a long time on IBM z systems.

The cost–benefit of software accommodating big-endian systems will look increasingly bad with legacy enterprise servers imposing a negative externality on everyone else. (E.g. refusal to consider bitcasts between SIMD vectors of the same bit width but different number of lanes as portable/safe operation in language design.)

TenFourFox and Leopard Webkit both emulate little-endian typed arrays on BE Power Macs and transparently byteswap integers. This mostly works for sites that care and has almost no overhead in JITted code for 16 and 32-bit int, but Facebook managed to break some of our assumptions with floats (example: https://github.com/classilla/tenfourfox/issues/453 -- we don't byteswap floats because it's expensive and probably would break things expecting native endian on the DOM side). Fortunately, TenFourFox doesn't support WebGL, so we don't need to worry about that.

IMHO, this is a broken promise about how the Web was supposed to be platform-independent. I get the feeling the general attitude on this thread is that such platforms are unimportant, but that doesn't mean it's not a broken promise.

I'm pretty sure many other fundamental protocols of the Web don't work so well on a non-2's-complement, non-8-bit-byte machine either.

WebGL's endian specification was the beginning of the end, after which folks started saying certain implementational details weren't worth caring about, and this just finishes it off. And that's not what was being sold in the beginning.

However, when the number's range is small enough, such code appears to work. Finding the bug is delayed until larger values occur, which is not until its deployed in the field or someone thinks of writing better test cases.

Anyway, if it is okay to truncate, the right thing to do (and ISO C conforming) is to access the underlying type as that type, and then cast the result, not to cast the pointer. I.e. rather than:

    int x = *(int *) ptr64; // breaks on BE; not kosher with ISO C aliasing rules

    int x = (int) *ptr64;  // works on LE and BE

If we want the bits for that int from some other part of the wider word, we can use shifting and masking.

Really I think consensus is that the industry focus on big endian systems was actually a mistake. It prioritized programmer comfort in a regime where it never really mattered.

I've always found LE vs BE to be somewhat like 0- vs 1-based array indices; one is more "conventional" to a human, but the other doesn't require a lot of contortions when expressed algebraically or algorithmically.

(With 0-based indices, element i is at address base + i * size; with 1-based indices, it's base + (i-1) * size, and it only gets worse from there as you add dimensions and other calculations.)

    0 array length <--- base pointer
    1 element 1    <--- array[1] == *(base + 1)
    2 element 2    <--- array[2] == *(base + 2)

Data structures which are frequently compared, e.g. to be sorted, benefit from BE ordering. Just binary compare them as byte strings, usually an efficient inline operation.

If I were implementing a packed wire format for SQL data, for example, I’d probably do it in BE order.

With practice it can be faster than entering on a calculator!

It's easy enough to deal with BE files and protocols by just swapping bytes. I'm referring to hardware architectures. Are there still any big-endian chips out there? Does it still make sense to support big-endian chips in new code as long as that code is never intended for embedded or very old systems?

(That requires a bit more than simply little-endian. It requires struct alignment and floating point formats to be the same. But with only a tiny bit of care it can be.)

For new code running in-core in 2018, I think little-endian is quite safe.

No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen.

The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you define a struct in C(++), there are not just endianness issues, but implementation-dependent issues of padding and alignment you have to consider. Recently, most compilers on most architectures standardized on self-alignment rules for all primitive types except char, but this is not guaranteed by the standard, it will bite you in the ass when you least expect it, and it will be decades yet before all the C code in the world is displaced by a single-implementation language like Rust.

The best way to write portable code that will work as intended is to treat all data coming from disk or over the wire as a bag of bytes, and not attempt to alias it to a struct.

When you’re on a PC, and working with files that you can reasonably expect will be exchanged (like doc, xls or pdf) — then your “No no no” heuristic is absolutely correct. As you pointed out, differences between compilers and between e.g. x86/amd64 are very likely to render these formats incompatible.

But when you’re working with files that only your software will access, unpacking it byte by byte will slow down IO by a huge factor compared to both mmap, and read/write of large blocks (the latter will likely translate to DMA i.e. the CPU will be free to do something else). When that’s the case (like for most videogames, even for PC ones), you don’t want to pay that performance penalty, you want to get that data in the memory ASAP.

Requirements change.

If:

* you're working with files that only your software will access AND

* you control, and understand, the CPU architecture of any machine that will touch that data structure and you KNOW that it will never change for the entire lifetime of that piece of data AND

* you always use the same version, or an ABI-compatible future version, of the same compiler that you KNOW will always lay out data the same way and you know what that way is AND

* you either don't need to touch this data structure in other programming languages, or you KNOW that this won't be an issue (for example because your programming language implementation was written in C, compiled against the same version of the compiler, and has an FFI that understands C structs)

THEN, you may proceed to mmap structs into memory. In practice these constraints are fairly commonplace; for example, NetBSD wscons device drivers report HID events which are specified in a struct, and because the HID events are likely never to leave the originating machine, only be passed from kernel space to user space, it makes sense to simply read(2) them straight into the struct.

And there's certainly nothing wrong with snarfing a large file into a char[], but when it comes time to extract meaning from it, unless you are absolutely sure that the underlying assumptions regarding data layout will never ever change, it will only cause marginal harm to do everything in byte offsets and shift and OR bytes to yield final values -- and this is the best portable way to access the data therein, agnostic of details about the compiler and CPU architecture.

In general it's a good idea to err on the side of caution by default, profile, and then optimize the hot paths as necessary bearing the constraints you're assuming in mind (and perhaps documenting them for good measure).

Maybe my initial statement was too strong, but I shudder whenever I see comments of the form "It's so convenient to just mmap() that sucker into a struct and access the fields!" Because they've never had to deal with the consequences of trying to access a struct from a Microsoft compiler serialized to disk, and finding out that GNU compilers have a quite different notion of how structure members are to be laid out in memory...

In general, changing data formats to something incompatible is one of the most expensive changes you can possibly make to your software. If you’re not sure write a prototype and profile. Implementing knowingly inefficient data format for a performance-critical application isn’t a good idea.

> it will only cause marginal harm to do everything in byte offsets and shift and OR bytes to yield final values

Huge harm, in both runtime performance, and code size & complexity.

P.S. For applications where portability matters and you’re OK paying performance cost of that, the industry has moved towards XML based formats. Not only it fixes byte order issues, also text encodings, globalization, it’s human readable, and it’s fast enough for many practical applications. E.g. doc/xls that you’ve mentioned are deprecated by docx/xlsx, the latter are XML based.

If you move text around you probably don't care about efficiency and can do it. If you care about efficiency and price(using cheap components) it is a very bad solution.

We use in all our serial-deserial a very fine grained controlled C library and it has been working flawlessly for years. This library is interfaced with C++, java, objC, swift,clojure, rust...

Look up FlatBuffers and CapnProto.

Cap'n Proto works by laying out data structures like a C compiler would, but following consistent, portable rules so that the layout is the same on all platforms. It then generates inline-able accessor functions to manipulate these structures which do pointer arithmetic similar to what a compiler would generate for accessing a struct. The end result is that accessing primitive fields from a Cap'n Proto struct is essentially identical in terms of machine instructions to accessing fields of a C struct.

To call that "deferring the parsing to access time" does not make sense.

(Note that for pointers, there are more differences: namely, because data will not always be loaded at the same address, pointers need to be relative rather than absolute. They also need to be bounds-checked for security. This adds a few instructions to pointer accesses, but those instructions look nothing like traditional "parsing" and certainly aren't byte-by-byte operations.)

Your statement earlier:

> The correct way to handle binary data is to unpack it into the struct byte by byte.

This is inaccurate. Byte-by-byte parsing is a valid way to do parsing but not the only way. Byte-by-byte parsers tend to be slow and -- arguably, more importantly -- overly complex and rigid. It is, for example, usually very hard to do "random access" with a byte-by-byte parser, because allowing out-of-order parsing tends to blow the code complexity through the roof.

On the other hand, with Cap'n Proto and similar approaches, you can trivially mmap() a very large data structure and traverse it randomly, and it "just works".

(Disclosure: I'm the author of Cap'n Proto, as well as the author of the first open source release of Google's Protocol Buffers, which does byte-by-byte binary parsing.)

As long as the Cap'n Proto or ProtoBuf data is properly aligned within of your custom file format, or you might end up with unusual slowness: https://blogs.msdn.microsoft.com/oldnewthing/20150116-00/?p=...

(Protobuf, on the other hand, fundamentally doesn't allow for multi-byte loads in the first place since integers use variable-width encoding, so alignment is irrelevant there.)

Basically, a decade ago a student started to work with some friends on an IRC client that integrates with a custom bouncer. Being a prototype, they just used Qt's serialization protocol between them. Over the years the project grew, at some point nokia funded development, it became Kubuntu's default IRC client (but only in the client+bouncer in one binary version, so without all the advantages), nokia was bought and closed and the department sold to BMW, and at some point.

Now, people tried writing third-party clients for this. And this became a minor issue, because the protocol was never documented. In favt, Qt's serialization was used for storing configs on disk, some blobs in the database, and over the network. It was later wrapped in TLS and deflate, and even at some point array-of-struct was turned into struct-of-array for the pattern during initialisation.

Either way, someone tried writing a mobile client for it, decided the protocol was insane, and instead built his own, almost identical client/bouncer system with an Erlang backend and json as protocol. This grew, and became IRCCloud.

Now, other people again tried developing third party clients for quassel. An android client was developed, but development was messy, and over the years, it stalled, because they reverse engineered the protocol, semi-successfully, and at some point didn't have enough time left, and gave up.

People reversed the protocol partially over the years again for pyquassel and quasselc/quasselbots/quassel-irssi.

Around then, another person tried reversing the protocol, and reimplementing it in JS for a webclient, which after a while became quassel-webserver.

Back then there was a lot of talk about replacing the protocol, but it was never done yet.

I, a user of irccloud around then, was annoyed with the costs (still being in high school myself, I couldn't afford the 4$/month, and the free tier wasn't enough), so I out for alternatives, and found quassel. But I hated the looks of Quasseldroid, so I forked it, and started working on the UI, and on features by reversing the protocol yet again. Not having actually programmed anything except for some delphi and VB.NET stuff, a tiny java project and one C# Windows Phone 7 app, my code was the worst, ever. Seriously, it was bad. After a while, discussion came about about turning this code into a PR, so I threw it all away, and rewrote it again, still bad, but it worked. This was merged, I became maintainer of Quasseldroid (because no one else was working on it anymore), and then, around 2015, I reversed the protocol, read the entire source of every implementation, wrote it all down on paper, studied every file format and every quirks and then I rewrote quasseldroid from scratch, in about 3 months, with every feature of the desktop version. I called this The Next Generation of Quasseldroid, jokingly quasseldroid TNG or later quasseldroid-ng.

A few weeks later, a new Android version came out, introducing Doze, and breaking everything about quasseldroid-ng. And so I rewrote it again, and before release, a new Android version broke it all again.

And that basically repeated, until I decided that it can't continue like this, and if we'll do major changes to the protocol, we might actually get a working version for Android that lasts longer than a few weeks in Beta before Google breaks it. So I learnt C++, and started contributing.

And that is basically my view of the story. Reverse engineering a protocol again and again, seeing variations of variations of the same protocol everywhere used, never properly documented. The Qt documentation is entirely different from what Qt actually puts on the wire. Blobs in the database.

But it also means backwards compatibility in the desktop client/core for every version from the past 10 years, and backwards compatibility on Android for every version of the past 6 years.

And now, maybe, I'll be able to help replace this, bit by bit. After the improvements to tge protocol that added major performance benefits, the next part is replacing the bouncer-side config format entirely, so I can properly containerise it.

TL;DR: no matter how good the support for a non-standard binary serialization format in your favourite language is, 10 years down the road people will reuse your protocol in a dozen more languages, and they'll have to reverse the protocol themselves, and will do a semi-good job at it, and because you never thought about backwards compatibility you now have a mess (we had luck because 99% of what we transmitted were key/value maps, and when reading we always used default values if the key didn't exist and ignored unused keys. Sometimes we did serialize structs, basically, though, and those places still cause me headaches today, and require workarounds to add features, e.g. the latest sendermode implementations)

Building upon an existing, well-documented, and relatively sane serialization format (protobuf, capn't proto, message pack, json, heck even bencode for all I care) is usually a good thing, and so is decoupling the messages from the details of an implementation's internals. Language and framework internal serializers (such as Python's pickle or, apparently, Qt's serializer) tend to make it harder to achieve both goals.

The problem with that is that whatever format seems well-documented and relatively sane today might become an obscure, unknown protocol 10 years down the road.

I unfortunately am not in a position to make such strong statements about Cap'n Proto. However, implementations exist in C++, Java, JavaScript, Rust, Go, Python, and a bunch of other languages, so it should at least be much easier to deal with than Qt serialization.

(Disclosure: I'm the author of Cap'n Proto and of the first open source release of Protobuf.)

When was the last time you saw a filename of the form NODE"accountname password"::device:[directory.subdirectory]filename.type;ver?

I have to agree here by experiences past. If the format in question has a chance of being performance sensitive, don't use FSM-based encodings [1]. It is inordinately difficult to optimize parsing these encodings even if you only have to handle tiny subsets, and it still won't be fast. A format like msgpack which prides itself on being very fast may be fast compared to JSON and other ways to express essentially arbitrary structures, but is DEAD SLOW compared to any direct encoding (be it a dedicated encoding you developed in literally a few hours or something like capnproto).

[1] Obviously, considering an encoding more complex than FSM means that you're an idiot and your application will almost certainly have security vulnerabilities related to the format in the future.

Assuming little-endian CPU arch. It's followed by a byte reorder on big-endian architectures. (And you assume all Windows instances are little-endian, which probably-is-but-may-not-be the case.) You made the decision to optimize for what you consider the common case, but it does not generalize without added translation code to all cases. You may have hidden the translation code behind CapnProto's generated accessors, but CapnProto's structs are translation-free the way AWS Lambda is "serverless".

> To call that "deferring the parsing to access time" does not make sense.

Except you are deferring translation work (like byte reordering) to access time. Either that or you're hiding it in the serialization APIs. Again, it's like "serverless" computing: just because you've hidden it doesn't mean it's gone away.

> Byte-by-byte parsing is a valid way to do parsing but not the only way. Byte-by-byte parsers tend to be slow and -- arguably, more importantly -- overly complex and rigid.

There's a performance cost, but hopefully you're only doing serialization/deserialization when you intend to hit the disk or wire to read/write into/out of your struct. All in-memory processing happens in whatever endianness and alignment makes your CPU and compiler happy.

There's nothing "overly complex and rigid" about understanding a binary format as a bag of bytes and fetching scalar values (including offsets into the data structure) from it accordingly. This is how shit gets done when it comes to portably handling arbitrary binary formats. CapnProto can score a few wins by assuming things about the target CPU/compiler, restricting the binary format to conform to some of those assumptions, and papering over the rest with code hidden behind some of its APIs. But it's not a general solution to the problem of extracting meaning from an arbitrary hunk of bytes that may or may not have come from a CapnProto-conformant application.

Basically all common CPUs are LE.

(And basically all BE CPUs have dedicated instructions for reading LE data. It's true I haven't yet added the inline assembly to use those instructions in Cap'n Proto's reference implementation, but that's only because no one actually cares about these architectures.)

So in basically all real use, there's no machine-instruction-level difference between accessing a field of a capnp struct and accessing a field of a C struct. If the instructions are identical, then how can you say one is "deferred parsing" and the other isn't? What meaning does any such distinction have?

> There's a performance cost, but hopefully you're only doing serialization/deserialization when you intend to hit the disk or wire to read/write into/out of your struct.

Disk is usually cached, meaning it's already in physical memory and you're wasting time making a copy rather than using the data in-place.

Over the network, within a datacenter, bandwidth is basically infinite (in that your CPU probably can't process bytes as fast as your network interface can). Time spent serializing and parsing is very real and wasteful. I've seen servers spending 30% or more of their CPU time parsing protobufs.

Over the long-haul internet, perhaps the CPU time spent parsing/serializing is not as relevant compared to the time spent transmitting. Still, I'd rather spend my CPU cycles elsewhere -- like in a dedicated compression algorithm -- rather than twiddling bytes needlessly in a parser.

> handling arbitrary binary formats

Yes, we all agree that some binary formats can't be handled any other way. But if you're in control of the format you use, then you can design it in a way that doesn't require a "bag of bytes" model, and your code can be much simpler and more adaptable as a result.

This. Optimizing performance on BE is a waste of everyone's time and resources.

(But the status quo on BE is that it does a load followed by a byte swap, which is probably pretty cheap anyway. The compiler might even already know how to optimize that into the appropriate LE-load instruction.)

We quickly discovered that a fair bit of Linux software (or at least Debian packaging of same) only worked properly on 64-bit DEC Alpha because the architecture was Little Endian; 32-bit pointers would typically "just work(™)" because the code was grabbing the first 32 bits, and the other half of the pointer addresses was typically 0x0.

Since we were porting to Big Endian hardware (just now discovered Itanium was selectable, not sure why it was Big Endian in our case) we were getting a fair bit of null pointer exceptions.

Anyway, ancient history, but that was my only direct exposure to endian-madness, so it has stuck with me.

I then worked on the FreeBSD port to DEC alpha. I helped with initial bringup and all the various issues around that. Then did a lot of the platform support, and the alpha-specific part of the Linux ABI compat layer, as well as the DEC OSF/1 binary compat layer. I ran a box on my desktop running FreeBSD/alpha as my primary workstation for years (API UP1000).

To make life more interesting, the later PDP-10/20 mainframes used PDP-11 minicomputers as front end processors and often as network processors so byte-swapping was the norm. Luckily the PDP-10 allowed bytes ranging from 1-36 bits wide - “byte” had not yet standardized on 8 bits

You kids sure know how to make someone feel really old... Much of the internet design predates the 68000, which in turn predates the RISC architectures you cite by half a decade and more.

Hands down no brainer.

Even the fact that headers come before payloads is a kind of "big endian", as is the fact that important information tends to occur earlier in headers. Look at a basic Ethernet frame. The destination address is first, so before even knowing the source address, or anything else, the switch can know where that frame will be sent.

On 100Gbit you're talking 0.01ns per bit, so 0.32ns for a full IPv4 address. This compared to, potentially, 15.24ns for a complete IPv4 frame.

If a 10Gbit network is being used precisely because 1Gbit didn't have sufficiently low latency, then it matters.

Perhaps the biggest mistake of IPv6 was copying IPv4's header order of source followed by destination. It would have been better if the destination address came before the source address in the header.

If big endian was effectively dead back then, I guess it's really dead now.

Of course, dealing with all of the Endian issues had not been planned for and was not in the project schedule :/

One time I had to make little- and big-endian machines talk nicely was for a message engine that ran on Intel/Linux. All the messages in the ICD were glorified packed structs that were native on the IBM/AIX side.

I used the ICD wire format as my storage format. I defined containers for all the primitive types with correct sizes and that swapped values on assignment and read (except for strings, of course). I added the appropriate pragmata to pack the underlying structs and locked down their sizes with static_asserts that would fail at compilation time until everything fit perfectly.

These classes gave me what looked like native structs on the Linux side, e.g.,

    msg.foo = 3;

    msg1.date = msg2.date + 1;

A cool aspect to the story was it all worked correctly the first time out of the box even though I did not have access to the AIX box during implementation. When it came time to integrate with the AIX side, all that was left to do was some low-level socket stuff to hook up to the message flow.

I’d moved to a different project full-time and worked this message engine during evenings and weekends. We got the code unpacked, so I run make check like I had been so frequently.

“Wow, you had time to write unit tests and everything?” asked one of the guys from behind my shoulder.

A moment of Zen hit.

“I didn’t have time not to.”

https://en.wikipedia.org/wiki/FFmpeg#Supported_formats

> When more than one byte is used to represent a PCM sample, the byte order (big endian vs. little endian) must be known. Due to the widespread use of little-endian Intel CPUs, little-endian PCM tends to be the most common byte orientation.

https://wiki.multimedia.cx/index.php/PCM

I think you meant to write “Little-endian is slightly more confusing for humans who use left-to-right languages” as all the R-L languages also put the least significant digit on the right.

Also for the part that is L-R, that is not the rule, as some people still read all the number as R-L (actually in a lot of historical documents that was the case), so the would read 1925 as five and twenty and nine hundred and a thousand. Where is now most people would read it as a thousand, and nine hundred, and five and twenty.

Source: native Hindi speaker.

Take 12,345,567 for instance, if you want to read it aloud you first have to go all the way to the end to figure out that there are three groups of numbers, so the first one is million, then thousands, then units. If you were only given a truncated version of the number it's impossible to read it aloud. So in a way the arabic represents number actually makes some sense and it could be argue that we write them backwards.

Furthermore IIRC from a discussion with an arabic friend (from the middle east, not sure if it's different in NA) the most formal way of reading numbers aloud is actually backwards (or little endian I guess) compared to the way we do in english. So if I understood correctly for 1,234 you'd say "four and thirty and two hundred and a thousand" or something similar. Don't quote me on that though because I know almost nothing about the arabic language and this is from a discussion I've had many years ago.

Other than that little endian makes more sense. The fact that Arabic numbers are written in big endian is weird. Nobody questions it because it's all anyone is taught.

from a bytes as representing a bitstring perspective, big endian make sense, and little endian is a terrible scramble.

And nearly every powerpc I've seen is big endian, FWIW.

The Atom needs instructions that make bigger, more complex operations simpler and lower powered, and with transistors the size they are these days they have ample silicon space to burn on these types of accelerators.

(disclaimer: IBMer)

  Little-endian is slightly more confusing for humans

0x12345678 stored big-endian, numbering bytes left-to-right:

  12 34 56 78

The solution is to show dumps with bytes numbered right-to-left. This is coherent with how we number bits, and also how we relatively position digits in any other number.

0x12345678 in little-endian, but written right-to-left:

  12 34 56 78

The endianness at the bit level is irrelevant because the bits are chunked into bytes, and are usually not even addressable.

A storage format exhibits endianness only when it is addressable.

In the C language, bits are only "addressable" via the shift operators. These are rooted in pure arithmetic, so that 1<<1 is always 2, regardless of whether you're on a big or little endian platform. Whether you call the value 1 "bit 7", "bit 8", "bit 1" or "bit 0" is just, pardon the pun, word semantics.

The only time you deal with "bit endianness" is with certain compressed data formats, and with bitfield layout rules.

It's extremely important to your mental model to understand how the bits are arranged, or left shift (<<) is going to produce different results in your head. If you think the value 1 is bit 7/8, you get:

1 0 0 0 0 0 0 0

Which would left shift to:

0 0 0 0 0 0 0 0

Which would not be equal to 2.

And the shortcut for that is simply regard bytes as big-endian. On all platforms. Whether you're on a PPC or x86, the byte 0x80 is going to go out on the wire as 1 first, followed by 7 zeros.

So if you're writing a data compressor and the spec says that the variable-length bit strings (huffman or whatever) are stuffed into bytes in network order, that means you fill bytes from the left down. That is done with code that works the same way on BE or LE platforms.

It is counterintuitive to swap pieces of it into some locally opposite order.

Yet, that's what has to be done so that numbers are readable.

That's why "od" has modes for that.

  $ od -tx1  /bin/ls | head -1
  0000000 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  $ od -tx2  /bin/ls | head -1
  0000000 457f 464c 0102 0001 0000 0000 0000 0000
  $ od --endian=big -tx2  /bin/ls | head -1  # GNU extension, probably.
  0000000 7f45 4c46 0201 0100 0000 0000 0000 0000

  A dump of memory (several kilobytes, megabytes or whatever) is inherently big endian: it proceeds from the base address and goes up.

Start each line with bytes from the right instead to make the bytes and bits numbering consistent, and you can print large little-endian dumps with all bits and bytes are where you expect them to be.

The number 0x12345678 is actually the nybbles 1 2 3 4 ... which are the bits 0001 0010 0011 0100 and so on.

In the hex dump 12 34 56 78 we just understand the bytes to be big-endian also at the nybble level (and bit also).

That is to say the "1" can be understood to be at the lower "nybble address", relative to the "2".

If that buffer is sent over a serial communication channel or network, the bits actually go in that order 0001 0010 0011. The 1 nybble goes out first, as 0001 (three zeros out the door, then a one), then the 2 nybble and so on.

    00000000: 6C 6F 77 09 30 0A 66 72 65 65 09 31 32 32 0A 65 low.0.free.122.e

    e.221.eerf.0.wol 65 0A 32 32 31 09 65 65 72 66 0A 30 09 77 6F 6C :00000000

[1] My first computer was a Tandy Color Computer, which had a serial port driven directly by the CPU. I learned pretty quickly which bit goes first.

Personally I prefer the little-endian representation.

That the documentation thinks this is bit 7 going to bit 6 is immaterial.

Calling the MSB "bit 1" is a tip of the hat to serial communications. In serial communication and networking, it is predominant to transmit the MSB first.

If the documentation is about a wire format, then using that numbering is correct down to the data link layer and (modulo framing considerations and such), physical.

The main practical problem with MSB0 is that you need to consider the width of whatever field or register you're looking at to work out the correct bit shift.

Why? Because you can use lexicographic sorting with big-endian bytes.

I can't explain it other than presumable there was some guy on the design committee who was like "but.... Unix... the 70s... network byte order... guys!" or something.

[1] https://github.com/spc476/CBOR

Like 6-bit bytes and octal, BCD, segments, and other such memories, it’s simply a relief that we’ve moved on.

So the one that ends big (most significant byte last) should be big endian and the one that ends little should be little endian. But it's actually the opposite. It makes it difficult to remember which is which.

    Little endian puts the
    Least significant part at the
    Lowest address(offset)

Is it really that different? It matters on x86 if the value is in some register already, since there's no subdivision of the high part, but it shouldn't matter for memory. Most instructions support math in addressing, so whether `(char)*x` gets compiled to `[eax]` or `[eax+3]` seems not very different.

I still have PowerPC and MIPS machines that are BE, and which are occasionally handy for seeing if pointer math is being done wrong.

Right, and also because modern RAM is essentially a block device. You just can’t access individual bytes anymore. E.g. on a typical modern PC with a dual-channel DDR, the block size is 128 bits = 16 bytes.

That's not correct. Ubuntu is LE only and it looks like RHEL is also switching from BE to LE.

"The base RISC-V ISA has a little-endian memory system, but non-standard variants can provide a big-endian..."

SPARC and MIPS are dead.

RISC-V is little endian.

See Wikipedia on Bi-endian machines: https://en.wikipedia.org/wiki/Endianness#Bi-endian_hardware