Why does the compiler understand ASCII by default?

4 months ago by printf("%s", name); to c/c_lang

Environment\
Compiler Clang on Termux on Samsung's One UI 7 on a Samsung Galaxy Tab A9+

The title is an assumption on it's own, so feel free to correct it/me!

I was experimenting with sanitizing user input, read to a character array with fgets. Specifically, I was trying to have a for loop remove (skip) certain input. Here is the code:

for (n = strlen(input) - 1; n >= 0; n--) { if (input[n] >= 0x30 && input[n] <= 0x39 || input[n] == ' ' || input[n] == '\t') { input[n] = 0x18; } }  

While the program does behave as I want it to, I don't understand why it seemlingy by default understands that the various hex codes refer to the character encoding as per the ASCII table. I cated my tablet's filesystem encoding at /sys/fs/f2fs/dm-44/encoding, which yielded UTF-8. If I understand it correctly, the first 128 code points of Unicode are the same as ASCII's. But according to this article on Wikipedia, there are no hexadecimal references in Unicode, only octal and decimal.

If the underlying filesystem uses UTF-8, and Unicode code points are not referred to by hex, how then does my compiler (Clang) understand what ASCII code points I'm referring to?

Is there some conversion going on under the hood that I am not aware of? I did find a libxml2/libxml/encoding.h, which contains comments about some conversion to and from UTF-8. Is this it? I can't make head or tails of it because of my limited C knowledge...

modeler 7 points 4 months ago

I C (and most other languages) you can enter a value like thirty-two using many different number bases: ‘ ‘ as a char, 32 in decimal, 0x20 is hex, 040 in octal or 0b00010000. All of these mean thirty-two, just written differently using different characters.

Note that each of these representations use different digits and/or letters to specify the same number - the mathematical entity thirty-two. These letters and digits are characters stored as numbers, eg the digit zero (0) is coded in ASCII as 48.

In the end, the C compiler will compile the text ‘32’, ‘0x40’ or ‘0b00010000’ into binary and store it in memory the single byte where the 5th bit set to 1 and the others to 0. This is the way computer memory is used to represent the number, and it’s linked to the way the electric circuits do things like add numbers together.

The compiler is the thing that understands that the characters like ‘3’, ‘+’ or ‘F’ and works out what they represent (a number, a variable name or a sum). When it understands that, it will turn the code into bits that the CPU can work with, for example passing it to the compare instruction CMP used to test whether the input is less than the number thirty-two.

This is very meta, and I hope this clarifies rather than confuses. Compilers are notoriously confusing, such as the thought ‘the compiler compiles the compiler code into an executable that can compile itself’.

path: 0 24138234, hotness: undefined, score: 7, children: 0
CameronDev 3 points 4 months ago

You are inspecting a byte, which contains 0x39. That it corresponds to "9" doesn't matter at all to the compiler at all. If you were on a system that didn't match "9" <=> 0x39 it would still replace those byte values.

path: 0 24137976, hotness: undefined, score: 3, children: 2
akunohana 2 points 4 months ago

Thanks! I'll have to noodle this around a bit because it's hard for me to understand. xD

path: 0 24137976 24138150, hotness: undefined, score: 2, children: 1
CameronDev 3 points 4 months ago

I dunno if this helps, but this screenshot shows the memory view of a program with a string. The hex representation in the middle is what is actually stored in memory. Each pair is one byte/char, and that is what your input[n] <= 0x39 is comparing against.

path: 0 24137976 24138150 24139021, hotness: undefined, score: 3, children: 0
kubica 3 points 4 months ago

Maybe this is a bit too much if you are just starting but read the first paragraph at least and then also look at "Relationship to ASCII" and "Relationship to Unicode"

https://en.wikipedia.org/wiki/Code_page

path: 0 24138132, hotness: undefined, score: 3, children: 1
akunohana 1 point 4 months ago

Thanks! Every piece of advice is appreciated! :D

path: 0 24138132 24138165, hotness: undefined, score: 1, children: 0
avidya 2 points 3 months ago

You can actually change this.

The compiler has an 'execution' character set that describes how characters should be interpreted at run time, and a 'source' character set, and they need not be the same.

Check this out

$ cat test_ebdic.c ; gcc -fexec-charset=EBCDIC-US test_ebdic.c ; ./a.out | xxd
#include <stdlib.h>
#include <stdio.h>
int main(void)
{
        char data[] = "Hello";
        printf("%s\n", data);
        return EXIT_SUCCESS;
}
00000000: c885 9393 960a                           ......

The file was typed in UTF-8, and the compiler automatically translated the literal to EBCDIC-US to emit at run time. My terminal can't display this, so I view it with a hex editor.

path: 0 24617079, hotness: undefined, score: 2, children: 2
akunohana 1 point 3 months ago

Sic!!! Thank you so much! I'll try this as soon as my new Gentoo system is up and running! Just to have another excuse for me to set it up already. 🤣

Is it gcc or the OS that determines what encodings I can pass as an option to the compiler?

path: 0 24617079 24622835, hotness: undefined, score: 1, children: 1
avidya 2 points 3 months ago

GCC uses libiconv to figure out how to transcode the bytes.

path: 0 24617079 24622835 24649996, hotness: undefined, score: 2, children: 0
amake 0 points 4 months ago

@akunohana Apologies if I'm missing something, but 0x30, 0x39, and 0x18 are merely hex notation for integers. You could have written 48, 57, and 24 instead for the same effect. Or you (probably?) could have used char literals like '0', '9' (I guess there isn't one for U+0018).

UTF-8 determines how the characters are encoded, i.e. what sequence of numbers (ints) they are represented by. So there's no special understanding of Unicode or hex going on here; you're comparing numbers (as you should).

path: 0 24138258, hotness: undefined, score: 0, children: 7
akunohana 0 points 4 months ago

LOL (please excuse my early 2000's slang)

Thanks for that clarification.

What I still don't quite understand is, does this mean that there is no "checking against an ASCII table" or "lookup" going on? But how then does it know that 0x18 is CAN (disregard)?

path: 0 24138258 24138259, hotness: undefined, score: 0, children: 6
kubica 1 point 4 months ago

Imagine you want to cypher a text, you can have a table with a column with the char you really want to be the meaning and another column with the obscured representation, what you write is the obscured thing, but when you read you take out that table to translate to the original meaning. Computers do sort of the same thing with the translation from bits to letters. But there are many tables, it depends on the languages to read and write.

path: 0 24138258 24138259 24138396, hotness: undefined, score: 1, children: 0
amake 0 points 4 months ago

@akunohana The compiler doesn't need to know that 0x18 is CAN; that knowledge is embedded in whatever decided that the data you're inspecting is UTF-8 or ASCII.

The content of your original post has been replaced with a link that I can't open so I can't go back and confirm where you said the data was coming from. But if the data was some other exotic encoding then 0x18 would mean something else in the context of that data.

path: 0 24138258 24138259 24138288, hotness: undefined, score: 0, children: 4
akunohana 1 point 4 months ago

Oh, maybe I messed something up when editing... Here's what I wrote:

I was experimenting with sanitizing user input In other words, I'm simply prompting for user input, read to a character array with fgets.

What it "that thing" that decided the encoding?

path: 0 24138258 24138259 24138288 24138289, hotness: undefined, score: 1, children: 3
amake 1 point 4 months ago

@akunohana OK that's a pretty good question then. In that case the encoding is determined by your terminal (or if not terminal then execution environment). Try invoking env (or locale) and looking at LANG and LC_ALL; those should tell you what your terminal accepts as input and passes along to your program.

path: 0 24138258 24138259 24138288 24138289 24138381, hotness: undefined, score: 1, children: 2
c_lang
c_lang

@programming.dev

login for more options
1357
103
28

Welcome to the C community!

C is quirky, flawed, and an enormous success.
... When I read commentary about suggestions for where C should go, I often think back and give thanks that it wasn't developed under the advice of a worldwide crowd.
... The only way to learn a new programming language is by writing programs in it.

© Dennis Ritchie

🌐 https://en.cppreference.com/w/c

go to feed...