@rovarma@mastodon.gamedev.place

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 6mo ago

New article! A user *cough* @aras@mastodon.gamedev.place *cough* is reporting full system freezes while using Superluminal on Linux. What do you do? Cry? Well, we did a little bit.

But we also dove into the kernel...again, this time finding & fixing several issues in eBPF's spinlock implementation. Read all about it:

https://rovarma.com/articles/a-tale-about-fixing-ebpf-spinlock-issues-in-the-linux-kernel/

rovarma.com

92

18

61

6

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 10mo ago

I've been wanting to start a blog for a while, and finally decided to bite the bullet.

The first article of hopefully many more to come is about, you guessed it, profiling & optimization.

Boosts appreciated!

https://rovarma.com/articles/optimizing-libdwarf-eh-frame-enumeration/

rovarma.com

Optimizing libdwarf .eh_frame enumeration | Ritesh Oedayrajsingh Varma

For the Linux version of Superluminal we rely on unwind information stored in the .eh_frame section in a binary to perform stack unwinding. We’ll go over optimizations we made to libdwarf that greatly improve the performance of retrieving this information.

33

18

28

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 6mo ago

Replying to

@aras@mastodon.gamedev.place I’m planning on switching my dev machine to Arch, which is on the bleeding edge kernel usually, so that we’ll spot issues like this faster locally in the future

6

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 6mo ago

Replying to

@aras@mastodon.gamedev.place Good point! Added a bit about that in the conclusion. TL;DR: usage of rqspinlock in the ebpf ring buffer was introduced with kernel 6.15, and our development machines (and most of our alpha testers) are on Ubuntu, which runs an older kernel than that (6.14). It would eventually have shown up for us too, once Ubuntu caught up.

5

2

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 10mo ago

Replying to

@sinbad I am never pressing this button

4

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 6mo ago

Replying to

@michaelc@mastodon.gamedev.place Yep, that’s exactly what happens. It’s fine in this case because with the high sampling rate and the sporadic nature of the race, you’ll lose a few samples at most. And, the race is between the context switch and the sample, and we also collect stacks for cswitches, so the sample wouldn’t even give us any extra information since it would be the same stack (remember this is a same-CPU issue, so it effectively means you’d be sampling the cswitch).

1

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 6mo ago

Replying to

@aachrisg@mastodon.gamedev.place @aras@mastodon.gamedev.place True! In my case, my machines are right next to each other, so the cable is fine. There is a KGDBOE (“KGDB Over Ethernet”) project that attempts to generically support kernel debugging over ethernet transports, but due to the complexity of the ethernet driver, it’s severely limited, and I never got it to work. If the ethernet driver is “external” to the machine being debugged, it should work in principle I think. One way to find out! ;-) https://github.com/sysprogs/kgdboe

GitHub

GitHub - sysprogs/kgdboe: A network interface for GDB for Linux Kernel

A network interface for GDB for Linux Kernel. Contribute to sysprogs/kgdboe development by creating an account on GitHub.

1

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 10mo ago

Replying to

@questor@mastodon.gamedev.place It's not 3MB of info; there are 3,373,919 DWARF bytecode *instructions*. The actual size of the binary in this case is ~91MiB. The .eh_frame info is emitted by default, yes, even if the language doesn't support exceptions/exceptions are disabled. This is because the exception unwinding machinery needs to support unwinding *through* code that doesn't support exceptions (consider, for example, C code calling into C++ code via a callback).

2

1

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 10mo ago

Replying to

@iralmeida@mastodon.gamedev.place

Does it match your guess?

Yep! It’s one of the things I was worried about.

I originally had a section at the start explaining the basics of stack frames, frame pointers etc. But a single section can’t really do it justice, it needs a full article by itself, so I scrapped it. The end result is indeed that some basic knowledge about stacks is needed :-(

Awesome that you read up on it and were able to follow it afterwards! Glad you liked it :-)

1

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 10mo ago

Replying to

@questor@mastodon.gamedev.place There *are* flags to prevent the .eh_frame info from being emitted, but you have to go out of your way to do so. By default they'll be there in both debug & release builds.

1

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 10mo ago

Replying to

@aras@mastodon.gamedev.place Partially inspired by our conversation over lunch a while back, so, thanks! :-)

1

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 6mo ago

Replying to

@michaelc@mastodon.gamedev.place The sample interrupt is the only NMI we care about. There might be other NMIs on the system, but we don’t capture them.

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 10mo ago

Replying to

@Themikina@mastodon.gamedev.place I’m using https://astro.build/ with some customization. I like that it allows you to keep articles in markdown, but it’s not the only static site generator that can do that. I mainly ended up with it because there was a template available for it that I liked. In previous years I’ve gone through the typical “I’ll build it myself” phase multiple times, but never got to the writing phase. My advice to my past self would be: don’t overthink it, it just spits out html in the end :P

Astro

Astro builds fast content sites, powerful web applications, dynamic server APIs, and everything in-between.

0

1

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 6mo ago

Replying to

@aachrisg@mastodon.gamedev.place @aras@mastodon.gamedev.place it might, but honestly a regular rs232 card is not expensive either, so it doesn’t matter much :)

0

3

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 10mo ago

Replying to

@iralmeida@mastodon.gamedev.place what did you struggle the most with? I think I know, but would be good to know for sure :-)

0

6

0

Ritesh Oedayrajsingh Varma @rovarma@mastodon.gamedev.place

· 10mo ago

Replying to

@zeux@mastodon.gamedev.place Congrats!

0