Replying to
@SecureOwl@infosec.exchange A small switch that you put there, as an urgent short term fix, 9 years ago.
Remote
AndresFreundTec
@AndresFreundTec@mastodon.social
mastodon 4.8.0-nightly.2026-10-06Long time postgres developer, working at Microsoft.
Account about tech, not politics. For the latter look to @AndresFreundPol@mastodon.social
6576 Followers
81 Following
34 Posts
Joined November 18, 2022
Bluesky:
Replying to
I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol. Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates.
Really required a lot of coincidences.
Scam attempt just now.
Supposedly somebody from google checking in about a change to my account recovery details. Contact via both phone and email, with both a robot and a brit sounding guy.
Knew a bunch of personal details (just stuff one could easily get via leaks etc).
Interesting bit is that it's not clear what they were after. Not a straight attempt at relaying SMS "2FA" or such. Played along until they wanted me to repeat a case number back to them, which seemed too risky.
A few days ago, at @pgconfdev@mastodon.social, I gave a talk about some pitfalls when profiling (and benchmarking) postgres. Turbo boost, iTLB, cpuidle, ...
Slides are here:
https://anarazel.de/talks/2026-05-21-pgconf-profiling-postgres-perils/profiling-postgres-perils.pdf
Replying to
@thesamesam @mgorny The "you must update to the latest release
to get all fixes needed to keep a system secure of all currently-known
issues" bit really makes my head explode.
How does GKH expect folks upgrade all their prod systems every ~6 days (the rough average release pace of -stable kernels), with sometimes as much as four releases in a week.
That's unrealistic CYA language, and GKH has to know that.
Replying to
@gregkh@social.kernel.org @joshbressers@infosec.exchange Of course companies hate it.
Plenty for bad reasons.
But also for reasonable ones: Who can afford to reboot all machines every few days? 6.18 averaged a stable release every ~5.6 days, 6.12 averaged one every ~6.15 days.
If you continually ask for unrealistic things ("All users of the xyz kernel series must upgrade." > once a week), folks *have* to stop listening after a while.
What do you expect folks to actually do with prod systems?
Replying to
One more aspect that I think emphasizes the number of coincidences that had to come together to find this:
I run a number "buildfarm" instances for automatic testing of postgres. Among them with valgrind. For some other test instance I had used -fno-omit-frame-pointer for some reason I do not remember. A year or so ago I moved all the test instances to a common base configuration, instead of duplicate configurations. I chose to make all of them use -fno-omit-frame-pointer.
Replying to
Afaict valgrind would not have complained about the payload without -fno-omit-frame-pointer. It was because _get_cpuid() expected the stack frame to look a certain way.
Additionally, I chose to use debian unstable to find possible portability problems earlier. Without that valgrind would have had nothing to complain.
Without having seen the odd complaints in valgrind, I don't think I would have looked deeply enough when seeing the high cpu in sshd below _get_cpuid().
Replying to
There are more coincidences that are even less interesting. But even the above should make it clear how unlikely it was that I found this thing.
Replying to
@tomasv @thesamesam @mgorny I think it's ok to have very frequent releases. There are some problems around how much testing that realistically allows, and that does seem to show up in the frequency of needing fixup -stable releases.
IMO the problem is having very frequent releases without providing *any* usable information about who needs to update how urgently, by saying that everyone needs to update immediately.
If you continually make unrealistic requests, nobody listens to you.
Replying to
For our own compute we've been averaging daily:
- 1464 core hours (full cores, not SMT)
- 396 of which were windows (visible due to the licensing cost)
- 40GB of artifacts
- doesn't include macos, which I can't track as easily, due to being self hosted runners
So we will likely need something where we can continue to provide compute ourselves, to keep this affordable.
Replying to
@Viss @Cs137 @WoodpeckerCI
I suspect our CI usage would very quickly make codeberg not like us, it's probably too much given their size.
Looks like woodpecker doesn't quite have the support for running full VMs, but I guess they do have a plugin architecture for that...
Replying to
@briankrebs@infosec.exchange It was preceded by a robot voice, yes. I don't *think* I pressed 1- I had grimy hands and expected a call, so I didn't check who was calling but just accepted the call via headset and also hung up the same way. A human sounding voice called a few minutes later. Picked up for the same reason as before... Listened for a minute and hung up again...
Replying to
@alien@mastodon.green You're right. I should have hung up earlier than I did. Normally I do. While I was *quite* sure it was a scam, I wasn't immediately *entirely* sure. And it felt a bit different / better done than the very easy to detect day-to-day stuff, so I probably was a bit too curious...
Replying to
@funkylab@mastodon.social @ondrej@mastodon.rfc1925.org Doesn't really answer your questions, but there's https://jasone.github.io/2025/06/12/jemalloc-postmortem/
Somehow the number of cases in which one needs to slap __attribute__((always_inline)) on static inline functions to prevent gcc from creating a non-inline [partial] versions in a TU seems to be steadily increasing.
That'd work, but the better fix is to simply not have the lock in the first place :). Which we did a few months ago...
Replying to
@jawnsy@mastodon.social @bcantrill@mastodon.social @ahl@mastodon.social I'd call it "one of the few long-term OSS developers having a podcast" prowess. A podcast I happen to listen to :)
Replying to
@axboe@fosstodon.org @bert_hubert@eupolicy.social Same number for the robot. The callback supposedly was from 818-934-0683
Replying to
Any suggestions / experiences where to look next?
Replying to
@josefbacik@hachyderm.io Heh. I spent surprisingly large amounts of time making postgres' AIO with io_uring competitive with the worker process based model.
We now have somewhat complicated heuristics for when to tell io_uring to process IOs asynchronously, even if they could be processed synchronously (mostly because to be read data is already in the page cache).
It's really annoying because in quite some cases io_uring is trivially faster, but in others it's slower.
Replying to
@gabrielesvelto@mas.to Nice thread!
You seem to imply that bugs have become considerably more frequent, largely due to the increased complexity. Right?
To me it's not obvious that the larger number of known issues isn't to a large degree due to much better visibility (we didn't have anywhere close to today's automatic crash collection systems in the past) and due to the vastly increased number of CPUs... Do you have any gut feeling about that?
Replying to
@noahm@chaos.social @corbet@social.kernel.org The vulnerable version was already present in debian test/unstable before that, it was just 5.6.0 instead of 5.6.1. And it was uploaded by the in-fact maintainer of the debian package for ~5 years.
Replying to
@bcantrill@mastodon.social @ahl@mastodon.social There might be more agreement on that :)
Replying to
@bcantrill@mastodon.social @ahl@mastodon.social Now we just need to avoid derailing into a heated debate about solaris/illumos being good/bad...
Replying to
@pinskia@hachyderm.io Unfortunately a trivial reproducer does not end up doing it... So I need to figure out how to get the real case reduced...
Replying to
@pinskia@hachyderm.io It's definitely some size heuristic - if the version of pg_get_ticks() that uses rdtsc is used the problem doesn't happen.
Replying to
@bert_hubert@eupolicy.social Heh. Maybe.
Just not sure what a recording of me saying a few numbers would be worth. I don't think I use anything that allows recovery via voice codes or such.
There's also plenty recordings online where I do say numbers...
Replying to
@pinskia@hachyderm.io Will try.
There are cases where I get it, the compiler can't know that yes, I do want this large piece of code duplicated just to constant-propagate with different values.
But the case at hand was generating a partial of
inline instr_time
pg_get_ticks(void)
{
if (timing_tsc_enabled)
{
instr_time now;
now.ticks = pg_rdtscp();
return now;
}
return pg_get_ticks_system();
}
where pg_get_ticks_system() is just a clock_gettime() converting to ns.
Replying to
@pinskia@hachyderm.io Well, that does make it easier.
Do you just need a .i or would you like to be able to execute it to see that yes, actually inlining is better?