That's a lot of attitude from a very small rodent. "Are you lookin' at me!?"
Remote
Scott Laird
@laird@hachyderm.io
mastodon 4.7.3Temporal, ex-Figma, ex-Google SRE. Prone to fits of extreme geekiness on many topics. Seattle-ish. He/him.
Random collection of this year's interests:
- Photography
- Siberian cats
- Networking
- Time synchronization (NTP, PTP, etc)
- Go (the language, not the board game)
- Amateur Radio (WA7SL)
- CNC / 3D printing / woodworking
433 Followers
328 Following
38 Posts
Joined November 20, 2022
Blog:
Github:
It's time for Saturday-morning moss, apparently.
There's a *distinct* lack of #marmotodon in my timeline here. They were all over the place at Mount Rainier today, as usual.
#photography #mountrainier #pnw
Wow, okay -- looks like there's been a big improvement in camera sensor technology lately when I wasn't paying attention. CineD just posted a review of the DJI Osmo Pocket 4P (https://www.cined.com/dji-osmo-pocket-4p-lab-test-rolling-shutter-dynamic-range-and-exposure-latitude/) and found that it gives *17* stops of dynamic range, which is legitimately more than any camera that they've tested, including the ARRI Alexa 35, which is pretty much *the* Hollywood "real cinema" camera. Most *great* cameras give 12-14 stops of dynamic range. The tiny little Osmo gives 17, and it doesn't seem to be due to AI gimmicks.
This seems to be the first non-phone camera with one of Sony's new "LOFIC" sensors. It sounds like it's able to handle overflowing photosites better, so it can handle extreme highlights without clipping, which has always been a fundamental problem of digital sensors.
It'll be interesting to see what these do once they grow to bigger-than-phone-sized sensors.
Replying to
One last moss picture from this morning. I'm not quite sure what it is, maybe some sort of fern moss? Plus a bonus mushroom, and a few douglas fir needles for scale.
#mosstodon #photography
Tulips at dawn this morning.
Today's morning fun: playing with creating a Netbox config template for my Arista devices.
- Good: it's actually pretty simple to get started. Almost no new infrastructure needed, unlike the time I tried to do this in Ansible and felt like I was pushing a rock uphill the whole time.
- Good: the data model (shockingly) matches reasonably well.
- Bad: oh god, not jinja2 again.
- Meh: you're mostly just left with a blob of text; there's no actual automation for diffing/deploying/etc. Although plugins are available
- Bad: the mismatch between jinja2 templates and Actual Python(tm) gets *really* annoying. It's like *half* python and half not. It's just fine for simple stuff -- "iterate over all interfaces and add `mtu {{ interface.mtu}}`" is easy. Iterating over all IPs for each interface and adding `ip address` or `ipv6 address`, depending, isn't too bad. Iterating over all IPs for each interface *plus* FHRP (VRRP, etc) addresses for each interface, plus dealing with corner cases is a making me lose hair.
- Meh: there are missing things in Netbox's data model that would be really nice, like the MLAG ID. There are plugins for some of this, and people have built giant complicated models for other bits; I'm tempted to just add a custom field for the ~dozen cases where I care.
- Meh: avoiding excessive newlines around `{% if ... %} command{% endif %}` in jinja2 is really annoying me. I guess I should probably re-read the manual -- I want "output what I say, without trying to reformat anything, but don't emit totally blank lines". Lots of blank lines: easy. No newlines at all: easy. Newlines in the right place: not so much.
Replying to
I've already upgraded 4 systems, I'll do a couple more this morning and then maybe hit two birds with one stone and write a new "import and preprocess video and still images from cameras" tool that gives the open-source flavor of my employer's product a workout. I should really be more familiar with it, and it's probably the right tool for this job. Ish.
The problem is that my NAS isn't really fast enough to let me edit 8K or 12K video directly from network shares, and I don't usually want to wait for an ~hour for a TB of video to slowly copy from a flash card onto the network before I can edit things. But I don't have enough local SSD space to keep everything local. And telling Resolve to start editing when files are on flash drives means that the projects break.
Plan:
- Produce a tree full of symlinks on a local disk.
- When a new video is detected on an external drive, first add a symlink that points directly to the external drive.
- Then copy the video from the flash drive onto a local SSD. When complete, update the symlink.
- Then copy the video from the local SSD onto the NAS. Don't update the symlink.
- When the local video-storage SSD is nearling full, delete videos from the SSD, after verifying that they've been copied to the NAS correctly. Update symlinks to point to the NAS copy.
This is mostly pretty straightforward, but getting the details right and not losing data is (as always) the fun part.
It's officially spring -- the tulips are blooming.
Replying to
@azonenberg@ioc.exchange @whitequark@social.treehouse.systems a better question might be "how hot does the heat sink need to be to keep the egg from seeping into it and getting gross?"
Replying to
@azonenberg@ioc.exchange @whitequark@social.treehouse.systems IIRC sous vide eggs work in the 60-70C range, depending on the texture that you want. I'm not sure quite how hot things need to be to count as "frying", although I could see a solid argument for 100C+. Supposedly you can get the egg to set up in a pan around 80C, although I'm not sure how long it'd take, or if it'd actually be something that I'd want to eat.
Replying to
@whitequark@social.treehouse.systems nVidia's cards don't usually start griping about heat until ~105C and don't shut down until north of 115C, depending on the card. So way hotter than needed to cook eggs.
I'm using USB-powered case fans to cool several nVidia devices, mostly ConnectX NICs. It really doesn't take much airflow to keep them under 80C, although I haven't tried a BF3. OTOH, without extra airflow they'll almost all shut down within minutes. I find it amazing that NICs (even basic, Doom-free, non-k8s-running NICs) need forced airflow to stay operational.
I finally bought a new photo printer; my last one was an Epson Stylus Photo 1270, which was introduced in 2000. The 1270 was impressive at the time, but the ink wasn't very stable and it *loved* to get clogged nozzles, which meant that it wasn't actually all that useful. I managed to produce a few prints from it that I liked, but it took a lot of iteration and wasted a lot of paper and ink, and in the end it was more of a matter of luck than anything.
The new one is a Canon imagePROGRAF PRO-1100, which obviously needs at least one more "pro" in the name. It's a beast that can print on 17" wide paper, and massively more computerized than the old Epson. It's smart enough to detect and fix clogs on its own (supposedly), and includes a bunch of self-checking and calibration tools that probably didn't exist anywhere 25 years ago.
For being ~25 years newer and at least one rung up the model ladder, the specs aren't actually *that* different from the old Epson. The Canon has somewhat higher resolution (2400x1200 vs 1440x720 dpi), but they both claim the same minimum droplet size (4 pl). Other than the paper size (13" vs 17"), the biggest spec difference is that the Canon is a 12-ink printer while the Epson was a 6-ink model, which gives is a *much* wider color gamut, so landscapes and portraits look more natural.
The biggest difference in actual use, though: I haven't had a bad print yet from the Canon. I've printed 6 or 7 prints in different sizes on a few different papers, and pretty much every single one of them matched what I'd expected to see. Shockingly, it seems to actually print correctly and reliably. Almost every photographic paper manufacturer ships ICC profiles *plus* printer config profiles (for paper thickness, ink handling, etc) for the 1100, and so far everything has actually worked. Download the profile, load the ICC into soft-proofing in Photoshop, load the paper profile into Canon's media tool, tweak so that it looks ok on the monitor, then hit print. A few minutes later there's a perfectly workable print sitting right there.
I'd really expected this to be harder, and was expecting to be frustrated for a few days before I actually found a printing process that worked for me.
Replying to
Ah, so Mastodon *does* strip GPS EXIF tags. I suspected that it did, but I rarely have GPS tags in my pictures anyway.
FWIW, this is from the Sultan Basin Road, a bit north of US-2 about an hour out of Seattle. Here's the location if anyone is looking for a bit of moss:
https://www.google.com/maps?q=loc:47.938295,-121.7121867
Replying to
And yes, I've seen outages (or near-outages) that involved every one of those problems.
Replying to
@Netux Yeah, certainly. But even small datacenters can have huge problems too. Not everything is fixable with a crash cart or gets an easy ETA, even once you know what's wrong. The hosting company that I worked for around the turn of the century had an issue that pushed back one of my projects from "after lunch today" to "maybe late next year, if we're lucky" when we discovered that we were *seriously* oversubscribing our DC's power system. We went into "*nothing* new goes in" mode and stayed there until we could get our local power company to pull additional power into the middle of an office tower.
It's the kind of (near) outage that no company really has more than once, because you immediately throw engineering resources at it and everyone involved probably *still* asks about the power supply situation whenever they're dealing with DCs, even after 25 years. I know I do.
(It was a repurposed mainframe machine room, with overengineered everything, but it was designed for an era when computers were huge but not particularly power-dense. Even the "wow, this server draws 100W in 4U!" era it wasn't hard to overload things.)
Replying to
@nixCraft@mastodon.social for memory at least this makes sense; by putting RAM in the SoC they get *much* more bandwidth than they could with SODIMMs. The M4 Max Mini has roughly the same bandwidth as 6 DDR5-5600 channels, or about 3x any non-workstation/non-server PC. There's no way they could fix the pins for 6 DDR5 channels into a cheap package, much less room for the DIMMs themselves without doubling the size of the computer.
SSD: that's just Apple being Apple.
Replying to
A couple miles further along, the Cascade River Road enters the North Cascades National Park. I grew up in Western Washington, and *technically* I think this was the first time that I've ever actually been *in* this National Park. I've driven over the North Cascades Highway repeatedly, but it never actually enters the park -- it's in the Ross Lake National Recreation Area, not the park itself. According to the park service's map, even the park's visitor center isn't actually *in* the park.
From what I can see, this is the only actual road in the entire park, although the very top of the Stehekin Valley Road *might* make it into the park as well, depending on how much of it has washed away lately. Since Stehekin isn't connected to the rest of the country's road system, though, it's hard to count it as a road into the park.
Replying to
Compared to updating the base OS, getting netbox working again is, as always, a total pain.
Like all Python apps, it's pretty tightly coupled to the version of Python installed on the system. Ubuntu 26.04 upgrades to Python 3.14, which, like all Python upgrades, breaks tons of things.
So I got to spend an hour or so iterating through Netbox's requirements.txt and randomly upgrading dependencies until they stopped complaining about missing methods, and then I had to modify a few legacy migration scripts so they didn't depend on parts of the language that no longer exist.
16 dependencies later, it seems to at least start up now.
Replying to
Okay, 4 more hosts upgraded to 26.04 today, 9-ish left. Mostly not very exciting, but:
- I had to make minor changes to Kea's DHCPv4 config to get it to restart.
- One host had been set to use legacy v1 cgroups, so I had to change that before the updater would run.
- Rebooting takes too long every time. I always decide that it's not going to boot and I'm going to need to drag a monitor and keyboard over about 3 seconds before it boots.
- Netbox failed to start, which is what netbox does whenever you even think about changing the version of Python on the system. This is going to be fun, but it can wait a bit.
Replying to
$ uptime
18:04:54 up 436 days, 23:15, 2 users, load average: 1.82, 1.01, 0.51
$ reboot
Sniff.
Replying to
In any case, my general opinion of Kea from when I first set it up 3 or 4 years ago hasn't really changed at all. It's massively overengineered for what I need, but *still* isn't actually able to do what I want it to do without jumping through major hoops.
Fortunately, most of the hoops that I had to jump through at the time have stayed jumped-through. Other than the static lease issue and the "oops, transient database issue so turning off DHCP" issue I can mostly just ignore it all.
I define DHCP pools and static leases in Netbox, and then periodically extract those and write them into a git repo, which gets synced onto my DHCP servers. In theory, this means that if anything breaks it'll just stop updating and it's relatively easy to fix.
In practice, it means that I can set up new static reservations by just reserving the IP address for it in Netbox (which is where I manage IPs anyway) and adding a MAC address to the IP via a custom field. Every IPv4 address that has a MAC listed gets added to Kea automatically within a few minutes. This is pretty much the absolute least amount of work needed to accomplish the goal. I just need to paste the MAC into the IP reservation.
Replying to
The first of these is easy to fix, just add something like
"reconnect-wait-time": 3000,
"max-reconnect-tries": 3,
to the lease-database block in kea-dhcp4.conf. That'll let it try reconnecting after a 3s delay, but it'll still exit if it fails 3x in a row. Getting systemd to restart it is at least a bit better documented.
I just realized that images from my camera, when saved as a 16-bit TIFF (which is almost required if you're going to edit or print them) are just *barely* small enough to be able to fit one photo onto a CD-ROM. Roughly 611,401,222 bytes.
It's sort of like the inverse of the old Sony Mavica cameras that used a floppy disk drive ("save icon") for saving images in the camera.
Replying to
@LapTop006@aus.social @azonenberg@ioc.exchange Yeah, ~all of Mellanox's SN[2345]* switches can be used as L2/L3 switches just using the stock kernel. You add routes/bridges/vlans/etc to the kernel like normal and it offloads to the switch ASIC. The SN2xxx models are really low-power as well, although they're getting pretty long in the tooth these days.
I've been working on getting VyOS working on mine, which gives them a decent switch-like UI. There are some philosophical differences with the VyOS core people around interface naming that will make this difficult to upstream, but I'm trying to find a way around it.
Replying to
@mattblaze@federate.social ok, now I'm curious. You usually mention the lens used for most pics, and I think you said that you usually use an Arca-Swiss C1, but which tripod legs?
Replying to
@mattblaze@federate.social Ah, I was going to guess a 5-series Gitzo systematic. I have a knockoff that's similar in height and rarely end up using either of the bottom two leg segments. I'll keep it in the car if I'm going someplace to shoot, but mostly use a mid-sized set of FLM legs that are way easier to carry around. Not as nice as the RRS, but nice enough.
Replying to
@swhobbit@hachyderm.io I had a similar-vintage HP MFP, I think it was the same basic printer. It died with multiple error codes that I wasn't able to fix after swapping multiple parts and staring at the service manual. So I ended up getting a brother to replace it, because current HPs are problematic.
You can't really compare a modern photo inkjet to a color laser, though. There's basically no overlap in use cases. I'm not convinced that the new Canon is even able to load more than a couple pages of paper at a time, and it really wants you to manually feed paper one sheet at a time via the manual feed slot. It'd be terrible as a "regular printer."
The results when printing pictures go exactly the opposite way, though. I've never seen a color laser that's better than "not horrible" with pictures, while modern photo inkjets are probably the highest-quality way to print images, ahead of traditional wet photographic printing.
Replying to
@swhobbit@hachyderm.io sounds reasonable. I'm up to 917 IP addresses and 175 IP prefixes in netbox right now, counting 482 DHCP pool addresses and a zillion PTP links. Not particularly big by any sort of enterprise standard, but way too big to want to manage without tooling. "Find an unassigned IP on network X" is a common operation and mostly-trivial now.
Generating the kea config happens automatically. Generating DNS is automated but doesn't run from cron; I like to look at diffs. For example, I just noticed that Netbox 4.x has a "DHCP" address status, so I moved all of my DHCP entries from status "Active" to "DHCP", and discovered that my DNS updater wants to delete all of my DHCP DNS entries, presumably because it depends on "Active". Oops. Reverted for now.
On the showing work front, https://github.com/scottlaird?tab=repositories&q=netbox2&type=&language=&sort= although it's unlikely that it's actually useful for anyone else.
Replying to
@paul_ipv6@infosec.exchange yeah, that's kind of what it all feels like. I really just want *some* way to have a relatively simple network with redundant DHCPv4 servers. This is all using RFC 1918 space, and I'm in *absolutely* no danger of running out of it. I can grow networks enough to let me have multiple, redundant DHCP pools per network. Except if I actually run 2 independent servers with their own pools then individual systems end up bouncing back and forth between blocks every time they reboot, and that ends up making debugging suck.
Turning on Kea's 2-way replicated DHCP seems to be the *easiest* solution to this, and it was a pain to get right. Honestly, I'd be perfectly fine with running one of them with a "wait 500ms before responding" config or even a "don't respond unless your healthcheck of the master DHCP server fails" setting.
Unfortunately, I've never seen a single DHCP server with either option. Not that either would be *that* hard to throw together, but it's harder than living with Kea, now that it's working.
Fundamentally, there doesn't seem to be much between Kea (very heavy-weight) and things like dnsmasq (mostly aimed at cheap consumer routers and so forth). It's kind of odd that there are a ton of perfectly viable BIND alternatives but no alternatives for ISC's DHCP code.