@gavin@fosstodon.org

· 11mo ago

I love #nix because it allows me to reproduce software states so reliably that I now take it for granted.

I dislike #nix because I cannot encourage teams I work with to use it. The main limitation is the lack of a universal workflow that I can advocate for - particularly for non-nix developers.

Personally, I use non-overlapping nix workflows for different projects. And that is just with 'classic' nix. Flakes, lix, tvix, snix, flox et al just add to the overwhelm for new people.

fosstodon.org

Fosstodon

2

1

0

Gavin Masterson @gavin@fosstodon.org

· 11mo ago

My last foray into containerising #nix development shells was a partial success. It worked for simple shells, but not the more complicated version of shells that are required for bigger projects.

Are there any premium-quality examples of the process in repos or blogs that I have not found yet?

Thanks in advance. (I will keep searching/reading, but like many things with #nix, it seems when you know a workflow, you know it - but you can't easily reproduce that path for others)

fosstodon.org

Fosstodon

2

0

Gavin Masterson @gavin@fosstodon.org

· 24mo ago

Replying to

@ryanmcshane@fosstodon.org @wardrup@babka.social @rstats@a.gup.pe @hadleywickham@fosstodon.org The TL;DR of all conversations I have digested is: The subtle differences in the behaviour of <- and = mean that reserving = for argument assignment, and <- for variable assignment will prevent most annoying consequences.

7

0

2

0

Gavin Masterson @gavin@fosstodon.org

· 31mo ago

Replying to

@brodriguesco@fosstodon.org @jimgar@hachyderm.io @rabaath@fosstodon.org @matthewbadger@fosstodon.org Hot off the press: The {pointblank} package has once again protected my client's data pipeline from potentially problematic inputs to the database.

4

20

0

Gavin Masterson @gavin@fosstodon.org

· 31mo ago

Replying to

@brodriguesco@fosstodon.org @jimgar@hachyderm.io @rabaath@fosstodon.org @matthewbadger@fosstodon.org I have used pointblank for type checking dataframes in data pipelines. The API is fantastic for in-line and schema-driven testing ('agent-based'). You can use strict assertive testing or threshold-based failure when data quality cannot be guaranteed or is not mission critical.

2

21

0

Gavin Masterson @gavin@fosstodon.org

· 31mo ago

Replying to

@jimgar@hachyderm.io @brodriguesco@fosstodon.org @rabaath@fosstodon.org @matthewbadger@fosstodon.org A unit is each row of the input dataframe. Each row that fails is returned in the CSV/object.

1

0

Gavin Masterson @gavin@fosstodon.org

· 31mo ago

Replying to

@jimgar@hachyderm.io @brodriguesco@fosstodon.org @rabaath@fosstodon.org @matthewbadger@fosstodon.org Not these functions. In the Validation workflow, the idea is that the functions are running in an automated job. In this scenario, the failure of the job is the necessary outcome. (I reran the function for the purpose of demonstration - my log message was not displayed alongside the code trace in a way that would have been clear.) The Data Quality workflow with agents does allow you to view the details of all failing unit tests for all checks.

1

15

0

Gavin Masterson @gavin@fosstodon.org

· 31mo ago

Replying to

@jimgar@hachyderm.io @yjunechoe@fosstodon.org Thanks, June! Only saw your message after I replied with the same info about agents. Using {tidyselect} is going to be amazing! 🤩

1

0

Gavin Masterson @gavin@fosstodon.org

· 31mo ago

Replying to

@jimgar@hachyderm.io @brodriguesco@fosstodon.org @rabaath@fosstodon.org @matthewbadger@fosstodon.org 'Agents' and their 'reports' are incredible. All failing unit tests are accessible via the CSV link (when the report is viewed in html). In an interactive setting, the dataframes containing the failing unit tests can be extracted as a whole or individually from the agent object. https://rstudio.github.io/pointblank/articles/VALID-I.html

rstudio.github.io

Introduction to the Data Quality Reporting Workflow (VALID-I)

0

14

0

Gavin Masterson @gavin@fosstodon.org

· 6mo ago

Replying to

@hayesstw @GrahamDowns There is this one: "lolsob"

0

4

0

Gavin Masterson @gavin@fosstodon.org

· 10mo ago

Without providing solid evidence of poor performance, the timing of the request for removal is highly suspect given the role of Dion George in COP30.

Dion George axed, despite ‘doing my job without fear, favour or prejudice’ https://share.google/yal7cD55Y92GXhQkY

share.google

0

1

0

Gavin Masterson @gavin@fosstodon.org

· 20mo ago

Replying to

@jimbob@aus.social I don't have time to test the use of nix for this, but if it is archived in nix, you could run the {insol} code in a nix-shell, and get around the CRAN issue.

0

2

0

Gavin Masterson @gavin@fosstodon.org

· 31mo ago

Replying to

@danwwilson@rstats.me @jimgar@hachyderm.io @brodriguesco@fosstodon.org @rabaath@fosstodon.org @matthewbadger@fosstodon.org Currently in a script. I am interested in knowing more about how it works in {targets}, though. I assume it can be used to interrupt the sequence of calls, if say, an input to one of the steps has been changed in an incorrect fashion.

0

10

0

Gavin Masterson @gavin@fosstodon.org

· 28mo ago

Replying to

@stefan@social.lol My barrier to the entire #nix project is the scattered, poor/outdated/conflicting documentation.

0

2

0