10 min read
WASM from Scratch
23 August 2023
I recently was trying to figure out how to compile my C code to WASM.
If you search the internet for this, you'll find a
ton of resources that say to use
emscripten
. Even
MDN says this.
While emscripten is not a bad choice, it does
a lot for you.
Not to mention that installing it can be a bit of a pain.
If you're like me and prefer to take a more
handmade approach to building software, it can be difficult to figure out everything you have to do to just compile some C code to WASM and run it.
Here is a comprehensive guide to what I learned after searching the internet and talking with some friends:
Basics
Writing for Web Assembly is similar to writing for an embedded system.
So if you've already taken care to keep your
dependencies minimal, this should be pretty straightforward.
If you're starting a new project I
strongly recommend this.
Let's say you have the world's most basic C program:
typedef signed int i32;
__attribute__((export_name("add")))
i32 add(i32 a, i32 b) {
return a + b;
}
clang
can compile our C code directly to WASM with a few additional command-line arguments.
If you run
clang -print-targets
you can see the available compile targets. And indeed, we can see both
wasm32
and
wasm64
are present.
Great! Let's try compiling with
clang
and specify we want to use the
wasm32
build target:
> clang --target=wasm32 -o hello.wasm hello.c
But if you just try to run this, you'll get the following error:
C:\dev\myspace> clang --target=wasm32 -o hello.wasm hello.c
wasm-ld: error: cannot open crt1.o: no such file or directory
wasm-ld: error: unable to find library -lc
wasm-ld: error: cannot open C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\Llvm\lib\clang\12.0.0\lib\libclang_rt.builtins-wasm32.a: no such file or directory
clang: error: linker command failed with exit code 1 (use -v to see invocation)
This is because, much like embedded software, there is no C standard library!
OK, no problem, we've got the
--no-standard-libraries
flag. So let's try that and get another error:
C:\dev\myspace> clang --target=wasm32 --no-standard-libraries -o hello.wasm hello.c
wasm-ld: error: entry symbol not defined (pass --no-entry to suppress): _start
clang: error: linker command failed with exit code 1 (use -v to see invocation)
This is clang telling us that there is no
_start
function (the function that eventually calls
int main
in a typical C program). Well it looks like there's a suggestion there too so let's just using that. The thing is
wasm-ld
is giving us the error. To pass a
wasm-ld
linker flag you have to prefix it with
-Wl,
, so the actual flag looks like:
-Wl,--no-entry
.
I will now take the liberty to add a few more compiler flags that will matter later:
-Wl,--no-entry
- no entry point (we're just building a library)
-Wl,--import-memory
- import our memory from the JS environment
-mbulk-memory
- tell the compiler not to be too clever and replace our loops with
memset
(because remember, no C standard library)
I also added the
-O1
optimization flag to make the output a little bit more concise.
Note that we explicitly told
clang
to export our function with the special
__attribute__((export_name("add")))
decorator.
If you prefer to export all the symbols in your program, you can do that with the
-Wl,--export-all
linker flag.
If you're curious, here's a list of
LLVM linker flags you can use.
So now we can compile our program again:
> clang -O1 --target=wasm32 --no-standard-libraries -mbulk-memory -Wl,--no-entry -Wl,--import-memory -o hello.wasm hello.c
And that works!
💡 Web Assembly is a binary format, and similar to machine assembly there is a text format called WAT (Web Assembly Text).
You can view the WAT of the WASM binary directly in the browser. Go to the Network tab, then click Preview:
(module
(memory $env.memory (;0;) (import "env" "memory") 2)
(global $__stack_pointer (;0;) (mut i32) (i32.const 66560))
(func $add (;0;) (export "add") (param $var0 i32) (param $var1 i32) (result i32)
local.get $var1
local.get $var0
i32.add
)
)
You can also use a CLI tool like
wasm2wat to disassemble the binary.
Using the WASM binary
To use the WASM binary, we need a runtime environment.
We can use the browser for this, but there are
other WASM runtimes.
Here is a simple HTML file to load our WASM:
<!doctype html>
<html>
<body>
<script>
const run = async () => {
const response = await fetch('/hello.wasm');
const bytes = await response.arrayBuffer();
const numPages = 4;
const memory = new WebAssembly.Memory({ initial: numPages });
const wasm = await WebAssembly.instantiate(bytes, {
env: {
memory,
},
});
const { instance } = wasm;
console.log(instance.exports.add(42, 42));
};
run();
</script>
</body>
</html>
Unfortunately, because of "security reasons" you can't just open this HTML file in your browser. You need to run a local web server to dynamically fetch content.
Any of the following commands will work:
> python -m http.server 8000
> npx serve -p 8000
Now navigate to
http://localhost:8000/hello.html and open the developer console. You should see:
84
Built-Ins
Clang has built-in functions for a few things such as:
f64 sqrt(f64 x) {
return __builtin_sqrt(x);
}
This will
just work in WASM because there are
f32.sqrt
and
f64.sqrt
instructions!
There are a few more notable built-ins:
__builtin_sqrt(x);
__builtin_wasm_memory_size(0);
__builtin_wasm_memory_grow(0, blocks);
__builtin_huge_valf();
There are no functions for
sin
and
cos
for example. For these you can: copy their source from
MUSL, write them yourself, or expose them from JavaScript.
When in doubt, you can actually
look at what how
emscripten implemented a feature you want.
Emscripten does a lot of weird and clever things to emulate certain native features.
For example, threads don't exist in WASM. Everything is single-threaded!
Imports
Because our WASM code is running in it's own sandbox, it will need to talk to the outside world.
In order to expose some JavaScript callbacks, you use the following syntax:
__attribute__((import_module("env"), import_name("js__sin")))
f64 js__sin(f64 x);
This tells
clang
that there is some external function that will be provided when the WASM module starts up.
Note that you
must provide the function now otherwise the WASM module will fail to initialize.
And then add the corresponding
js__sin
function to the
env
property of the WASM module:
const wasm = await WebAssembly.instantiate(bytes, {
env: {
memory,
js__sin: Math.sin,
},
});
Now you can call
js__sin
from C.
This is also, for example, how you would do WebGL bindings.
Memory
When we booted up our WASM code, we told it exactly how many memory pages it has. Each page is 64KB.
Our program must run with the memory that it's given.
While there is a builtin to request more memory from the WASM runtime, I prefer to just declare all the memory I will ever use upfront.
You can get the heap size and base pointer with:
extern u32 __heap_base;
u64 wasm__heap_size_in_bytes()
{
return __builtin_wasm_memory_size(0) << 16;
}
void *wasm__heap_pointer()
{
return (void *)&__heap_base;
}
⚠️ If you just try to write directly into your WASM program's memory you will be writing over the stack from [0, &__heap_base)
.
Now we can define our own global allocator with a simple bump allocator:
static u32 offset = 0;
u8 *wasm__push(u32 size)
{
u8 *result = NULL;
size = (size + 15) & ~15;
if (offset + size < wasm__heap_size_in_bytes())
{
result = (u8 *)(wasm__heap_pointer()) + offset;
offset += size;
}
return result;
}
void wasm__pop(u32 size)
{
size = (size + 15) & ~15;
if ((i64)offset - size > 0) {
offset -= size;
} else {
offset = 0;
}
}
⚠️ Note that we always align our allocations because some data structures like Float64Array
require that the memory is 8-byte aligned!
Pointers will be
u32
-sized because we are using
wasm32
.
As far as I can tell, the most you can return from a native function via the WASM FFI is a
u64
and this gets returned to JavaScript as a
BigInt
.
If you need to read or write more than that then you can use the
wasm__push
and
wasm__pop
functions.
Another strategy would be to just agree on a reserved address space for JS-WASM communication, sort of like a global scratch buffer.
There is no
malloc
or
free
.
If you absolutely can't live without them, you can define your own or find an implementation online.
In practice, I literally just use my
Arena
allocator on top of the
wasm__push
and
wasm__pop
functions.
Strings
Strings in WASM are a special case of dealing with exchanging large amounts of information back and forth (I mean, larger than a
u64
).
You might require special handling to get the UTF8 part correctly, but otherwise we are literally just writing the bytes to some shared location on the heap and passing the pointer to them.
The browser has two built-in classes for dealing with UTF8 strings:
TextDecoder
and
TextEncoder
.
Let's define a couple of functions for reading and writing UTF8 strings:
const decoder = new TextDecoder("utf8");
const DecodeString = (u8Array, idx, length) => {
if (!idx) return "";
const endPtr = idx + length;
return decoder.decode(u8Array.subarray(idx, endPtr));
}
const StringLength = (u8Array, ptr) => {
let endPtr = ptr;
while (u8Array[endPtr]) ++endPtr;
return endPtr - ptr;
}
const encoder = new TextEncoder("utf8");
const EncodeString = (u8Array, base, string) => {
if (!string.length) return 0;
return encoder.encodeInto(string, u8Array.subarray(base)).written;
}
Then we can use this code to read strings from WASM like this:
const ptr = instance.exports.secret_message();
const result = DecodeString(heap, ptr, StringLength(heap, ptr));
console.log("The secret message is:", result);
If we want to pass a JS string to WASM we can do this:
const ptr = instance.exports.wasm__push(3 * message.length);
const count = EncodeString(heap, ptr, message);
instance.exports.ping(ptr, count);
instance.exports.wasm__pop(3 * message.length);
Structs
If you want to return a struct from a function you can either pack it into a
u32
or a
u64
and re-interpret the bytes, or you can return a pointer to the struct and decode the struct fields. I think returning a pointer is a bit easier to work with so that's what I normally do.
For example:
typedef unsigned int u32;
struct Foo {
u32 x;
u32 y;
};
const ptr = instance.exports.new__foo();
const dataView = new DataView(memory.buffer);
let at = ptr;
const x = dataView.getUint32(at, true);
at += 4;
const y = dataView.getUint32(at, true);
at += 4;
const foo = { x, y };
DataView
has a bunch of
methods for interpreting different kinds of data.
The second argument should be
true
because according to
Rasmus the bytes are always little endian independent of the host-machine's endianess.
This strategy also works for pointer fields.
Arrays
We already saw a couple of examples for reading and writing strings. Arrays are similar.
To send an
f64
array to WASM:
const jsArray = [1, 2, 3, 4, 5];
const cArrayPtr = instance.exports.wasm__push(8 * array.length);
const cArray = new Float64Array(memory.buffer, cArrayPtr, jsArray.length);
cArray.set(jsArray);
instance.exports.compute_square_roots(cArrayPtr, jsArray.length);
const result = Array.from(cArray);
instance.exports.wasm__pop(8 * array.length);
To read an
f64
array from WASM:
const ptr = instance.exports.push_f64_array(length);
const dataView = new DataView(memory.buffer);
let at = ptr;
const count = dataView.getUint32(at, true);
at += 4;
const data = dataView.getUint32(at, true);
at += 4;
const result = new Float64Array(memory.buffer, data, count);
return result;
Note that because we just returned a
Float64Array
into WASM's memory, these values will actually change if WASM overwrites that memory. So you should take care to copy the array if you need to, or make sure the memory stays at a fixed address.
For small arrays you can use
Array.from(float64Array)
to convert the
TypedArray
into a plain JavaScript array.
You can also copy the typed array itself with
float64Array.slice()
.
⚠️ One caveat with TypedArray
s is that Array.isArray
will actually return false
for them!
You should use ArrayBuffer.isView()
instead.
Source code
Here is the full source code of our program, also available on
Github:
#define NULL 0
typedef unsigned char u8;
typedef unsigned int u32;
typedef signed int i32;
typedef signed long long i64;
typedef double f64;
__attribute__((import_module("env"), import_name("js__output")))
void js__output(char *str, i32 size);
__attribute__((import_module("env"), import_name("js__sin")))
f64 js__sin(f64 x);
f64 sqrt(f64 x)
{
return __builtin_sqrt(x);
}
f64 sin(f64 x)
{
return js__sin(x);
}
u32 wasm__heap_size_in_bytes()
{
return __builtin_wasm_memory_size(0) << 16;
}
extern u32 __heap_base;
void *wasm__heap_pointer()
{
return (void *)&__heap_base;
}
static u32 offset = 0;
__attribute__((export_name("wasm__push")))
u8 *wasm__push(u32 size)
{
u8 *result = NULL;
size = (size + 15) & ~15;
if (offset + size < wasm__heap_size_in_bytes())
{
result = (u8 *)(wasm__heap_pointer()) + offset;
offset += size;
}
return result;
}
__attribute__((export_name("wasm__pop")))
void wasm__pop(u32 size)
{
size = (size + 15) & ~15;
if ((i64)offset - size > 0) {
offset -= size;
} else {
offset = 0;
}
}
typedef struct f64_Array f64_Array;
struct f64_Array
{
u32 count;
f64 *data;
};
__attribute__((export_name("add")))
i32 add(i32 x, i32 y)
{
return x + y;
}
__attribute__((export_name("secret_message")))
char *secret_message()
{
return (char *)"Hello, Sailor!";
}
__attribute__((export_name("ping")))
void ping(char *message, i32 count)
{
js__output(message, count);
}
__attribute__((export_name("compute_square_roots")))
void compute_square_roots(f64 *array, i32 count)
{
for (i32 i = 0; i < count; i += 1)
{
array[i] = sqrt(array[i]);
}
}
__attribute__((export_name("push_f64_array")))
f64_Array *push_f64_array(u32 count)
{
f64_Array *result = (f64_Array *)wasm__push(sizeof(f64_Array));
result->count = count;
result->data = (f64 *)wasm__push(sizeof(f64) * count);
for (i32 i = 0; i < count; i += 1)
{
result->data[i] = 0;
}
return result;
}
<!doctype html>
<html>
<script src="./hello.js">
</script>
</html>
const decoder = new TextDecoder("utf8");
const DecodeString = (u8Array, idx, length) => {
if (!idx) return "";
const endPtr = idx + length;
return decoder.decode(u8Array.subarray(idx, endPtr));
}
const StringLength = (u8Array, ptr) => {
let endPtr = ptr;
while (u8Array[endPtr]) ++endPtr;
return endPtr - ptr;
}
const encoder = new TextEncoder("utf8");
const EncodeString = (u8Array, base, string) => {
if (!string.length) return 0;
return encoder.encodeInto(string, u8Array.subarray(base)).written;
}
const Kilobytes = (x) => 1024 * x;
const Megabytes = (x) => 1024 * 1024 * x;
const load = async (wasmPath) => {
const response = await fetch(wasmPath);
const bytes = await response.arrayBuffer();
const numPages = Megabytes(64) / Kilobytes(64);
const memory = new WebAssembly.Memory({ initial: numPages });
const heap = new Uint8Array(memory.buffer);
const wasm = await WebAssembly.instantiate(bytes, {
env: {
memory,
js__output: (str, size) => {
const result = DecodeString(heap, str, size);
console.log("[hello]", result);
},
js__sin: Math.sin,
},
});
const { instance } = wasm;
return {
add: (x, y) => {
return instance.exports.add(x, y);
},
secret_message: () => {
const ptr = instance.exports.secret_message();
return DecodeString(heap, ptr, StringLength(heap, ptr));
},
ping: (message) => {
const ptr = instance.exports.wasm__push(3 * message.length);
const count = EncodeString(heap, ptr, message);
instance.exports.ping(ptr, count);
instance.exports.wasm__pop(3 * message.length);
},
compute_square_roots: (array) => {
const cArrayPtr = instance.exports.wasm__push(8 * array.length);
const cArray = new Float64Array(memory.buffer, cArrayPtr, array.length);
cArray.set(array);
instance.exports.compute_square_roots(cArrayPtr, array.length);
instance.exports.wasm__pop(8 * array.length);
return Array.from(cArray);
},
push_f64_array: (length) => {
const ptr = instance.exports.push_f64_array(length);
const dataView = new DataView(memory.buffer);
let at = ptr;
const count = dataView.getUint32(at, true);
at += 4;
const data = dataView.getUint32(at, true);
at += 4;
const result = new Float64Array(memory.buffer, data, count);
return result;
},
};
};
const main = async () => {
const hello = await load('/hello.wasm');
console.log(`add(42, 42):`, hello.add(42, 42));
console.log(`secret_message():`, hello.secret_message());
console.log(`ping("hello, good sir"):`, hello.ping("hello, good sir"));
console.log(`compute_square_roots([1, 2, 3, 4, 5]):`, hello.compute_square_roots([1, 2, 3, 4, 5]));
console.log(`push_f64_array(10):`, hello.push_f64_array(10));
};
main();
Debugging with Sourcemaps
To step through your C code in the browser with sourcemaps you need to do the following:
1) Install the Chrome extension:
C/C++ DevTools Support for DWARF
2) Add
-gdwarf-5
to the CLI arguments (and probably disable optimizations)
3) Set up a Path Substitution in the extension's Options. For compiling on Windows, I needed to add a substitution from
/home/nick/
to
C:/
and then it just works!
Wrap Up
The code I've shared should cover just about everything you could ever want to do in WASM.
While it would be a lot of work to emulate
everything emscripten
is doing, I hope this has shown you a viable alternative.
Overall I have enjoyed working with WASM despite some of it's limitations.
Links
https://rsms.me/wasm-introhttps://depth-first.com/articles/2019/10/16/compiling-c-to-webassembly-and-running-it-without-emscripten/https://dassur.ma/things/c-to-webassembly/https://github.com/llvm-mirror/clang/blob/master/test/CodeGen/builtins-wasm.chttps://github.com/emscripten-core/emscripten/tree/main/system/lib/libc/musl