Planet Scheme

November 26, 2020

GNU Guix

Music Production on Guix System

The recent release of Guix 1.2.0 was accompanied by a release song (lyrics). Let me tell you how this happened and how you can use Guix System for your own music productions!

Collage of cables, a rack-mounted USB audio interface, a Chapman Stick, and a screenshot of Ardour

It all started only three days before the 1.2.0 release when someone on the #guix IRC channel brought up the tradition of OpenBSD people to write, record, and publish at least one song alongside their system releases.

A wistful look at my neglected instruments later I felt compelled to take this as a challenge: with less than three days to go could we actually compose, record, and publish a song using nothing but free software despite a jam-packed weekend schedule? I wanted to find out.

Inspiration and Planning

The working title “Ode to One Two Oh” was an obvious choice, being a quasi-palindrome, and its five syllables suggested a time signature of 5/4. Where to from here?

As I stared at my Emacs session with a Guile REPL (read, eval, print, loop) buffer I tried to recall what the letters “REPL” stand for. Clearly, in my case the “P” was for “Procrastination”, but what about the others? I had stumbled upon the chorus: a description of the Guix development process. Contribute as others before us have shared their contributions (Reciprocation), review patches and discuss (Evaluation), hack on something else (Procrastination), and repeat (Loop).

The words suggested a simple descending melody, which would need to be elevated by a somewhat less simple chord progression. After trying out a few harmonies on the Grand Stick I remembered how terrible my memory was and decided that I would need to scatter the harmonies onto a canvas, listen to the whole progression, and adjust the lines as needed — all without having to build up muscle memory for harmonies and progressions I may very well end up discarding in the process.

This is where my composition workflow probably deviates from most other people. Many would use a MIDI sequencer for that kind of approach, whereas I decided to hone in on the exact harmonies with an unlikely tool: the unparalleled music engraving application Lilypond. Lilypond sports a versatile language that covers primitive note input, the means of combining them to larger phrases and musical ideas, and the means of abstraction — it allows for musical ideas to be named and recombined in different shapes. For everything the language doesn’t account for with specialized syntax I can simply switch to Guile Scheme. No other notation software is as flexible and malleable as Lilypond. I let it generate both sheet music and a MIDI file — the sheet music is displayed in a PDF viewer in Emacs and the MIDI file sent to fluidsynth (because I trust my ears over my eyes).

lilypond draft.ly && \
  fluidsynth -r 48000 -i -n -a alsa \
    ~/soundfonts/FluidR3GM.sf2 draft.midi

I always try to keep the duration of my stay in the MIDI world at a minimum, because a composition workflow that is firmly rooted in MIDI tends to result in music that sounds sterile or robotic, an undesirable quality that can be hard to eradicate later. So I put them aside and focused on another part of the song. Mirroring the quasi-palindrome of the title, the song’s structure would be A B C B A. With the smooth chords of the B section locked down I walked up to the Grand Stick (I mounted it on a modified microphone stand for more flexibility) and tapped out a contrasting two-handed funky bass line to the click of a metronome.

Time to record!

Audio Recording

How does the signal of my stringed instrument make it into a file on my computer’s disk? The Stick’s stereo signal feeds into two daisy-chained Axoloti boards with hand-crafted signal processing tuned to the peculiarities of my instrument; the stereo output of the DSP boards is connected to two inputs on my USB audio interface (the Tascam US16x08, though any class-compliant audio interface will work just fine), which presents itself to the system as a 16 channel input sound card.

Exposed Axoloti boards feeding on a Chapman Stick audio signal

The computer runs JACK 1 to shuffle the incoming signals on all channels to other JACK clients (audio applications and plugins). I use patchage to conveniently wire up inputs and outputs in an interactive graphical signal flow diagram.

audio and MIDI signal flow in patchage

The center piece of my recording session is the venerable Ardour, an incredibly flexible and polished digital audio workstation (DAW) that natively supports JACK and also serves as an LV2 audio plugin host.

Okay, the Stick is ready to be recorded, but I prefer to record a drum track first instead of playing to the click of a metronome. But wait, I really can’t record the drums now! This is an apartment and the neighbors are asleep. Oh, and I’m a lousy funk drummer. Let’s cheat!

Custom drum patterns in Hydrogen

My drum machine of choice is Hydrogen. It lets me create any number of arbitrarily long patterns, combine them, and — that’s crucially important — sync up with other JACK applications such as Ardour. I toggled a button in Ardour to let it take control of the JACK transport, meaning that when I start recording in Ardour the drum machine starts playing in sync — perfect! Let’s roll!

Rough Processing

After frustrating minutes had turned into exhausting hours of recording the bass line (eventually resorting to punching in and out instead of re-recording everything whenever I made an audible mistake) I put my arranger’s hat on and tried to get a sense for what might still be missing. The recorded track sounded rather “flat”; after all this was the “raw” signal straight from the instrument’s pre-amplifier. Time to spruce it up and approximate the final sound!

Over the years the selection of audio plugins that I’m using to process my recordings has narrowed down to few more than the Calf suite of LV2 plugins. I will admit to being an extremely superficial person, so a big factor in choosing the Calf plugins is the consistent, very pretty and intuitive user interface they present. What made me stick to the Calf plugins, though, is that they also sound really good. Furthermore, they make it easy to tweak the many parameters and come with helpful diagnostics (such as listening only to the changed frequencies, or spectral visualizations, etc).

Selection of Calf plugins

First thing I do is to use light compression to even out slight variations in dynamics. Calf Mono Compressor does the job here. The principle is simple: a compressor lowers the volume when a sound gets loud and it raises the volume of the resulting signal according to the dialed in makeup gain. It has a bunch of other knobs to control how far ahead it will look to watch out for loud sounds, another to prevent “pumping” (too rapid activation and deactivation), another to smooth out the “knee” of the threshold curve, etc — but what it really comes down to is: at what level should the signal be reduced, by how much, and what’s the gain to make up for the overall loss in volume.

One piece of advice: resist the temptation of overdoing this! Compression may make the sound punchier, but don’t throw away all dynamics here or you’ll risk draining the life from your hard-earned waveforms. So I stick with a ratio of no more than two, keep the makeup gain fairly low, and the threshold somewhat high to only trigger the compressor in the upper fifth of the dynamic range.

Next: equalization. This is to scoop out an unpleasant character by attenuating certain frequency ranges. Again, the Calf suite has got me covered. A five band EQ is sufficient as I don’t want to be tempted to make sharp cuts in the frequency spectrum of the signal. As a general rule I don’t boost any frequencies but primarily cut. Here, too, the key is to be gentle. Don’t cut more than two or three decibels or you’ll end up with a comically filtered sound. (Exceptions apply for drum recordings, but thankfully we dodged that bullet by using Hydrogen.) If you find that you need to cut or boost more than that, chances are that there are problems with your recording. Don’t try to compensate for problems with post-processing that could have been avoided by improving the recording quality. I scoop out the mids a tiny bit and cut out the very low bass frequencies that don’t contribute anything more than a rumble.

MIDI Resurgence

With the bass line and the drums in place it was time to revisit the harmonies for part B. I really did want to record them with my very red virtual analog hardware synthesizer, but when I brought it into my tiny work room I realized that I had ran out of space on my lap — I had run out of space on my desk months ago, and with all those audio cables piling up on the floor even that was no option.

Lilypond generated a MIDI file earlier, so I sighed and told Ardour to import it. I assigned the ancient electric piano plugin mdaEPiano as a synthesizer to the track, and expanded the track vertically to drag around every MIDI note event by hand, refining the notes and adding tiny timing and velocity imperfections to make it seem as if a human played them. To paper over the clearly audible undesirable imperfections of the plugin’s sound generation I added some Calf Reverb (for the graphics people: adding reverb is roughly equivalent to using the blur tool in a graphics application). I could have picked one of the many better synthesizers and emulators, but we go way back, this plugin and I.

MIDI support in Ardour is really well thought-out as you don’t need to ever leave the main window, so you are always surrounded by audio context. You just expand the track vertically and have as much access to the events as you want. This approach may be a little unusual for those who are used to MIDI sequencers, but for my purposes it is close to perfect.

Audio and MIDI tracks in Ardour

Lyrics and Vocals: Human vs Machine

At this point things were coming together nicely, but we only had lyrics (a handful of words, really) for the chorus. Not enough for a song, but just a little too much for a laundry list. So I turned to IRC again where our resident Debian ambassador and assuredly human person Vagrant Cascadian happened to volunteer a stroke of genius: what if the lyrics were composed entirely of typos in package descriptions that had been fixed since the last release? Vagrant carefully reordered the words to a poignant poem, bringing them alive as the voice of the adversarial machine, a symbol of the waning hold of the bugs and errors on our way to the next release. What would be more fitting than to let the machine croak these words in the release song?

I started up the Festival speech synthesis software and a few Scheme expressions later the inimitable monotone of none other than the entity known only as cmu_us_fem_cg droned the words into an audio file that I promptly imported into Ardour, and processed with the GxDetune plugin, Calf Vintage Delay, and some reverb.

I did my best between bites of a Monday pre-release lunch to perform the role of the human contemplating this cycle of creation through hacking by recording three vocal lines (two low, one high) for the chorus.

Final touches

With about half an hour or so left before the release announcement it became clear that there would be no “audio mastering” in the traditional sense. I put on my monitoring headphones (flat frequency response for an “objective” mixing experience) and adjusted the volume levels, plugin settings, and faded in and out the noisy edges of each take. I also added a few more effects: a phaser on the Chapman Stick track to emphasize that this is supposed to be funk; a crossover (Calf X-Over 2 Band) and three audio busses to first process the low bass frequencies separately from the high frequencies (e.g. to use different amplifier simulations) and then to rejoin them; and a limiter on the master bus to avoid any volume spikes above 0dB and to compress the mix.

Mixing tracks in Ardour

I exported the final mix as a Vorbis OGG audio file and sent it off to Ludovic to have it included in the release announcement.

Concluding Thoughts

Some of the tools used in the production

So what’s the verdict? Is the result perfect? Absolutely not! But I do want to point out that any and all mistakes and imperfections are entirely due to my own lack of skill and time — they are not the result of the limitations of free software. A more skilled audio engineer would certainly not be limited by a system like the one I used, a system composed entirely of freedom-respecting software that is made extra reliable by the user-empowering and liberating features of Guix System.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Ricardo Wurmus at November 26, 2020 03:30 PM

November 23, 2020

GNU Guix

GNU Guix 1.2.0 released

Image of a flight of the Guix.

We are pleased to announce the release of GNU Guix version 1.2.0, right in time to celebrate the eighth anniversary of Guix!

The release comes with ISO-9660 installation images, a virtual machine image, and with tarballs to install the package manager on top of your GNU/Linux distro, either from source or from binaries. Guix users can update by running guix pull.

It’s been almost 7 months since the last release, during which 200 people contributed code and packages, and a number of people contributed to other important tasks—code review, system administration, translation, web site updates, Outreachy mentoring, you name it!

There’s been more than 10,200 commits in that time frame and it is the challenge of these release notes to summarize all that activity.

Before reading any further, sit back and play this very special release tune, Ode to One Two Oh (lyrics) brought to you by your friendly Guix team—see credits below!

Security

A major highlight in this release is the ability to authenticate channels, which probably makes Guix one of the safest ways to deliver complete operating systems today. This was the missing link in our “software supply chain” and we’re glad it’s now fixed. The end result is that guix pull and related commands now cryptographically authenticate channel code that they fetch; you cannot, for instance, retrieve unauthorized commits to the official Guix repository. We detailed the design and implementation back in July. The manual explains what you need to know as a user and as a channel author. There’s also a new guix git authenticate command to use this authentication mechanism for arbitrary Git repositories!

Example commit graph.

Coupled to that, guix pull and guix system reconfigure now detect potential system downgrades or Guix downgrades and raise an error. This ensures you cannot be tricked into downgrading the software in your system, which could potentially reintroduce exploitable vulnerabilities in the software you run.

With these safeguards in place, we have added an unattended upgrade service that, in a nutshell, runs guix pull && guix system reconfigure periodically. Unattended upgrades and peace of mind.

Another important change from a security perspective that we’re proud of is the reduction of binary seeds to 60 MiB on x86_64 and i686, thanks to tireless work on GNU Mes, Gash, and related software.

On the same security theme, the build daemon and origin programming interface now accept new cryptographic hash functions (in particular SHA-3 and BLAKE2s) for “fixed-output derivations”—so far we were unconditionally using SHA256 hashes for source code.

User experience

We want Guix to be accessible and useful to a broad audience and that has again been a guiding principle for this release. The graphical system installer and the script to install Guix on another distro have both received bug fixes and usability improvements. First-time users will appreciate the fact that guix help now gives a clear overview of the available commands, that guix commands are less verbose by default (they no longer display a lengthy list of things that they’ll download), and that guix pull displays a progress bar as it updates its Git checkout. guix search, guix system search, and similar commands now invoke a pager automatically (less by default), addressing an oft-reported annoyance.

Performance improved in several places. Use of the new “baseline compiler” that landed in Guile 3.0.4 leads to reduced build times for Guix itself, which in turn means that guix pull is much less resource-hungry. Performance got better in several other areas, and more work is yet to come.

We’re giving users more flexibility on the command line, with the addition of three package transformation options: --with-debug-info (always debug in good conditions!), --with-c-toolchain, and --without-tests. Transformations are now recorded in the profile and replayed upon guix upgrade. Furthermore, those options now operate on the whole dependency graph, including “implicit” inputs, allowing for transformations not possible before, such as:

guix install --with-input=python=python2 python-itsdangerous

Last, the new (guix transformations) module provides an interface to the transformation options available at the command line, which is useful if you want to use such transformations in a manifest.

The reference manual has been expanded: there’s a new “Getting Started” section, the “Programming Interface” section contains more info for packagers. We added code examples in many places; in the on-line copy of the manual, identifiers in those code snippets are clickable, linking to the right place in the Guix or Guile manuals.

Last but not least, the manual is fully translated into French, German, and Spanish, with partial translations in Russian and Chinese. Guix itself is fully translated in those three languages and partially translated in eleven other languages.

Packs, GNU/Hurd, disk images, services, …

But there’s more! If you’re interested in bringing applications from Guix to Guix-less machines, guix pack -RR now supports a new ‘fakechroot’ execution engine for relocatable packs, and the ability to choose among different engines at run time with the GUIX_EXECUTION_ENGINE variable. The fakechroot engine improves performance compared to the proot engine, for hosts that do not support unprivileged user namespaces.

Support for whole-system cross-compilation—as in guix system build --target=arm-linux-gnueabihf config.scm—has been improved. That, together with a lot of porting work both for packages and for the Guix System machinery, brings the hurd-vm service—a cross-compiled Guix GNU/Hurd system running as a virtual machine under GNU/Linux. This in turn has let us start work on native GNU/Hurd support.

Related to this, the new (gnu image) module implements a flexible interface to operating system images; from the command line, it is accessible via guix system disk-image --image-type=TYPE. Several image types are supported: compressed ISO-9660, qcow2 containing ext4 partitions, ext2 with Hurd options, and so on. This is currently implemented using genimage.

In addition to those already mentioned, a dozen of new system services are available, including services for Ganeti, LXQt, R Shiny, Gemini, and Guix Build Coordinator.

2,000 packages have been added, for a total of more than 15K packages; 3,652 were upgraded. The distribution comes with GNU libc 2.31, GCC 10.2, GNOME 3.34, Xfce 4.14.2, Linux-libre 5.9.3, and LibreOffice 6.4.6.2 to name a few. There’s also a new build system for packages built with Maven (bootstrapping Maven in Guix was the topic of a talk at the Guix Days last week).

The NEWS file lists additional noteworthy changes and bug fixes you may be interested in.

Try it!

You can go ahead and download this new version and get in touch with us.

Speaking of which, our Debian ambassador told us that you will soon be able to apt install guix if you’re on Debian or a derivative distro!

Enjoy!

Credits

Ricardo Wurmus (grand stick, synthesizer, drums, vocals, lyrics) — Luis Felipe (illustration) — Vagrant Cascadian (Debian packaging, lyrics) — Festival (back vocals)

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Ludovic Courtès at November 23, 2020 03:45 PM

November 17, 2020

Programming Praxis

Sales Commission

Today’s exercise is simple drill for beginning programmers; it was inspired by a student asking on a beginning programmer’s platform for someone to do his homework for him. The requested answer was in C++, and it was a few weeks ago, so I don’t feel too bad posting a Scheme solution:

Write a program that inputs each employee’s sales, then computes and prints the employee’s commission according to the formula $200 plus 10% of sales. Stop the input using a sentinel value. At the end of the input, display a one-dimensional array containing all the commission amounts, and the total amount of commission paid. You may not use a switch statement or multiple if statements.

Your task is to write a solution to the student’s homework; don’t use C++. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at November 17, 2020 10:00 AM

November 16, 2020

GNU Guix

Online Guix Day Conference: schedule released!

The Guix hackers are very happy to announce the first online Guix Day Conference on Sunday November, 22nd. This conference is open to everyone (no registration fee) and will be held entirely online. Want to know the schedule, read on!

There will be no presentation on the 22nd! Please watch the talks beforehand.

Guix Days logo

Join us live on the 22nd to participate in the various sessions!

Live discussions will take place on Sunday, November 22nd, and the agenda is the following (UTC+1):

(break)

(long break)

(break)

Chinese users will find a mirror hosted at https://guix.org.cn. You will also find alternative links below for different formats, and downloading through IPFS.

Each session will be question/answer and discussion related to the topic via the BigBlueButton instance generously hosted by Fosshost. Warm thanks to them!

The slots are short so please watch the videos beforehand to better enjoy the discussions. The term BoF means open discussion to address prospects. The last discussion may be longer depending on what you have to share.

The main channel for the day will be the video chat and questions will be asked via the chat hosted there or––because we love it––via #guix on irc.freenode.net then the floor might be shared, opening more mics. The discussions will not be recorded because we would like to keep them informal––where people are less impressed to share their point of views.

The Code of Conduct applies for all the channels of communication.

GNU Guix in psychology research and teaching

Presented by Lars-Dominik Braun. (video webm, video mp4, doi, slide, ipfs)

The Leibniz Institute for Psychology supports psychologists in adopting open science practices by providing them with free infrastructure services. One of these services is PsychNotebook, a web platform providing access to shareable and reproducible R and Python programming environments, using RStudio and JupyterLab in particular. PsychNotebook is used by researchers for analyzing research data and by instructors to teach psychology students script-based analyses.

The session covers why psychology among other research field needs this platform, how it is designed and what role GNU Guix plays in all of this. In particular, four challenges are addressed: user management, project management, web app deployment/proxying; as well as usability and how GNU Guix supports or provide reproducible environments.

Fixing the CI

Presented by Mathieu Othacehe. (video webm, video mp4, ipfs)

The session covers the following points:

Nix and Guix: similarities and differences

Presented by Andrew Tropin. (video, video mp4, ipfs)

The session covers an high-level overview and comparison of Nix and GNU Guix package managers or NixOS and Guix System distributions. The comparison had been initiated to understand the differences between those two great projects. It may inspire people from both communities to implement missing features or help someone to decide, which package manager or operating system to pick.

From v1.2 to release process

Chaired by Simon Tournier.

The session covers a proposal to smooth the release process; ironic for a rolling-release project, isn’t it? Make a release means:

  1. how and what to do: tools
  2. schedule / track
  3. who do: people

The #1 is roughly described in the file maintenance/doc/release.org. Even if a non-negligible part is based on experience and cannot be documented; see #3. However, tools are still missing: going further than guix weather --coverage or --display-missing.

The #2 means track what is going on between 2 releases. It seems easier to write down important changes when they happen than parse all the log history one week before releasing in order to publish the NEWS file. More importantly, #2 means stay on track with the schedule: release when it is ready? at fixed date? what must be in? does it make sense to synchronize with staging merges? how to synchronize with the branch core-updates?

The #3 means who take the responsibility to do the job. And it appears easier to divide the workload. More importantly, how to share the skill? Guix could take inspiration from Nix or GNU Glibc or your-name-it.

Porting Guix to modern PowerPC

Presented by Tobias Platen. (video webm, video mp4, ipfs)

The sessions covers how to port of Guix to modern 64-bit Little Endian, since that one is best supported by the Talos II and its graphics card, the AST2500. The final aim would be a self hosting version of Guix that runs on the Talos II, the Blackbird and the upcoming Libre-SOC. Such port may also be useful to support older PowerMacs including the G4 and G5.

Just build it with Guix

Presented by Efraim Flashner. (video webm, video mp4, ipfs)

The session covers how to use Guix as build plateform. Creating custom packages is ubiquitous with Guix and packaging with Guix is fairly straightforward. But what about working with packages where you want to package a non-release version? Or if you're hacking on another package which either isn't packaged or you want to test your changes before sending off a patch set or a pull request? The file guix.scm is the unofficial filename for Guix build instructions for this case. It provides a target for creating an environment for hacking on the package, and it creates a recipe to build what's currently in that repository; meaning you can use the power of Guix for builds even while working on other projects. A combination of a little bit of boiler-plate for building “this here repository” and standard package definitions allow for easy building and rebuilding without dirtying the source tree. And also for building multiple versions of the package in one go.

Progress so far on the Guix Build Coordinator

Presented by Chris Baines. (video webm, video mp4, ipfs)

The session looks at the Guix Build Coordinator, a tool for building lots of derivations, potentially across many machines, and doing something useful with the results. This is a new tool that might be able to help with patch review, quality assurance as well as substitute availability. The talk will cover the motivation, design, implementation and future, along with a small demo of the Guix Build Coordinator.

Peer-to-peer substitutes and sources

Chaired by David Dashyan.

The session covers the status of the peer-to-peer substitutes distribution. Especially the almost 2 years old first draft adding support to distribute and retrieve substitutes over IPFS; see the wip-ipfs-substitutes branch. Moreover the branches wip-ipfs and wip-ipfs2 are attempts to add the Go part of IPFS. The discussion will address the next steps to merge the branch wip-ipfs-substitutes or how to add decentralized substitutes distribution.

Guile Hacker Handbook

Presented by Jérémy Korwin-Zmijowski. (video webm, video mp4, ipfs)

The sessions covers Guile Hacker Handbook (GHH). The purpose of the GHH is to show Guile the way modern programming languages are shown, i.e., demonstrating its tools and following development approach we often stick to professionally.

Lengthy manuals are often hard to grasp at first; especially when learning new materials from scratch. Instead, it seems easier to rely first on tutorials or blog posts. Writing style and direct application sometimes helps to understand the underlying concepts). Then reads the reference manual feels more comfortable. GHH is an attempt to address this. For example, GHH is about Guile, not Scheme.

GHH is also about Test Driven Development and focuses on tests as first-class citizen.

(BoF) Rust and Cargo

Chaired by John Soo.

The session covers the various issues with the Rust ecosystem in Guix. The discussion is about:

  • packaging efforts
  • build systems
  • incremental compilation/shared libraries
Bootstrapping the Java Ecosystem

Presented by Julien Lepiller. (video webm, video mp4, ipfs)

The session covers the Maven bootstrap and the Maven Build System and how this Maven story may inspire directions to implement similar bootstrap stories for other ecosystems.

Ensuring that software is built entirely from source is an essential practice to ensure user Freedom, as well as for auditability and security. Unfortunately, the Java ecosystem is very complex and presents some interesting challenges when building from source.

One of these challenges is Maven, a build tool and package manager that is used by many if not most of the Java developers nowadays. One key challenge is that Maven is itself a Java package, that is built with Maven and has a lot of dependencies, that themselves use Maven.

The discussion presents the current state of the bootstrap and how we break the various dependency cycles that occur. The recent addition to Guix of the maven build system is a major step towards a good support of the Java ecosystem in Guix. We will discuss how Maven works, what it expects, and how Guix can accommodate it to build offline, reproducibly, with no trusted binary.

The ways forward (roadmap and beyond)

Chaired by GNU Guix maintainers.

The session covers the medium- and long-term goals that may or may not look realistic. Pragmatic dream!

Code of Conduct

This online conference is an official Guix event. Therefore, the Code of Conduct applies. Please be sure to read it beforehand!

If you witness violations of the code of conduct during the event, please email guix-days@gnu.org, a private email alias that reaches the organizers (Simon zimoun Tournier and Julien roptat Lepiller) and the GNU Guix maintainers.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Guix Hackers at November 16, 2020 12:00 AM

November 10, 2020

Programming Praxis

Memfrob

Memfrob is a light encryption algorithm that works by xor-ing 4210 = 001010102 with each input byte to create encrypted output; decryption is symmetric with encryption. Memfrob is roughly equivalent to ROT13 in cryptographic security.

Your task is to implement memfrob. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at November 10, 2020 10:00 AM

November 08, 2020

GNU Guix

Introduction to the Guix Data Service, the missing blog post

The Guix Data Service processes, stores and provides data about Guix over time, at least that is what the README says. It's been around since the start of 2019, and while there have been plenty of long emails to the guix-devel mailing list about it and a blog post about a related Outreachy project, this is the first blog post covering what it is and why it exists.

Why?

The initial motivation came from trying to automate aspects of reviewing patches for Guix. If you have some patches for Guix, one aspect of review might be to apply the patches and then build the affected packages. How do you know what packages are affected though?

You could try and guess based on the content of the patches, and this could work some of the time, but because Guix packages relate to one another, changing one package may cause dependent packages to change. Additionally, there are places in Guix where small changes could affect a large number of packages, build systems for example. The guix refresh -l command is really helpful when testing packages locally, but it can in some cases miss some packages that are effected by changes, as it only explores the package graph.

The approach taken to working out what packages are affected by a set of patches, was to record information about all the packages in the "base" revision of Guix, prior to applying the patches, and also record information about the "target" revision generated from applying the patches. With all that information about the two revisions, you can then compare the data to determine what's changed. This goes beyond finding out what packages are affected, and includes things like looking at changes to lint warnings, channel news entries, and more.

Screenshot of the comparision between two commits

This approach of storing information about revisions has applications beyond reviewing patches, which is another reason why this approach was taken. While the Guix Data Service doesn't bring new knowledge to the world, it can make information that is out there more accessible, and that improved accessibility is a feature.

Applications

Say you want to know when the previous version of a package was available, and what that version was. You could look through the Git repository history, or inspect previous revisions to find out, but because the Guix Data Service can store the available package names and versions in a range of revisions, it can provide this information more quickly and with less effort.

Screenshot of a Guix Data Service package versions page for emacs

Now, questions about package versions is something a user of Guix might have. However, so far I haven't seen the Guix Data Service as something that users of Guix should necessarily use or be aware of. Instead, I think it has a place to provide information to enable things that users of Guix would directly use.

There are a few applications of data from the Guix Data Service in varying states of development. I've been attempting to automate parts of a weekly news publication about Guix through using the Guix Data Service, I've also been writing a service for building derivations, which I've been using in conjunction with the Guix Data Service to provide substitutes. As part of an Outreachy internship on improving internationalisation support in the Guix Data Service, Danjela worked on creating a package search page for the Guix website, which wrapped the package search functionality in the Guix Data Service.

Screenshot of the prototype weekly news site

While I'm cautious about having the Guix Data Service attempt to address individual user needs, there are some applications where it alone is sufficient. I've been using the Guix Data Service to gather up data about which packages in Guix don't build reproducibly. Hopefully the Guix Data Service is well positioned to help with technical questions like this.

Screenshot of the Guix Data Service package reproducibility page

Architecture

The Guix Data Service is written in Guile, and uses PostgreSQL for the database. There's plenty of SQL queries in the code, including some quite long ones.

There are several scripts which act as entry points to different parts of the Guile codebase:

  • guix-data-service

    • Provides the web interface
  • guix-data-service-process-jobs

    • Polls the database for new jobs, and forks guix-data-service-process-job
  • guix-data-service-process-job

    • Processes an individual job, loading data for a single revision
  • guix-data-service-process-branch-updated-email

    • Processes an email to find out about new revisions

There's also other scripts which perform a range of functions, like backing up the database, generating a minimal database which is hopefully small in size and querying build/substitute servers for information.

When running on a Guix system, there's a service to help with deployment.

Getting information in

Rather than polling the Git repository to find out about new revisions, the methodology used so far has been to receive emails. In the case of the main Guix Git repository, this can work as follows:

  • New commits are pushed to the Guix Git repository on Savannah
  • A post-receive hook sends an email about the branch that's been updated, plus emails about each commit
  • A dedicated email account is used to subscribe to guix-commits, and this receives the emails
  • getmail is running as a service on the machine running the Guix Data Service, it receives the emails and calls guix-data-service-process-branch-updated-email passing the contents in on stdin
  • The script reads the email and inserts the relevant data in about the branch that was updated, the time it was updated, and also inserts a new job, representing the new revision to be processed

Compared to polling the Git repository, this approach has a few advantages:

  • The time the email was sent is a good proxy to when the branch was updated
  • Receiving emails promptly, having getmail use IDLE for example helps with learning of changes quickly
  • The email account provides some reliability, so messages aren't missed if the Guix Data Serivce is down, they'll just be processed later
  • The mbox files for the guix-commits mailing list can be processed to provide data for past revision, in years when the Guix Data Service wasn't running for example

When the guix-data-service-process-job script runs, it goes through a long process to extract information about that revision of Guix, and store it in the database.

The first part of this is to actually fetch and build the relevant revision. The Guix Data Service uses channels and inferiors, the same code used by guix pull and guix time-machine for communication with another revision of Guix. It's through the inferior REPL that information from the target revision is extracted.

In addition to receiving information about new revisions, the Guix Data Service can accept POST requests to receive information about builds. There's some support in Cuirass and the Guix Build Coordinator to send these requests.

Storing all that information

Following on from one of the initial motivations for the Guix Data Service, comparing two revisions to determine which packages have changed, the schema for the database is organised to facilitate fast comparisons between two arbitrary revisions. The compromise here is the storage space taken up.

Similar to version control systems, an alternative schema would have been to store the differences between revisions in a linked list or tree. This would avoid storing lots of information that generally doesn't often change between subsequent revisions, but at the expense of making both determining the entire state of individual revisions and comparing arbitrary revisions more complex and costly.

Even though all the information about each revision is associated with that revision, there is some indirection, and deduplication involved. For example, each revision is associated against entries in the package_derivations table, which represents a package plus derivation for a specific system and target. If this information doesn't differ between two revisions, they'll just reference the same entries in this table.

Making information available

Currently, the Guix Data Service provides a web interface. The HTML pages and forms are designed to help potential users of the Guix Data Service find and explore the available data. When using the Guix Data Service, you'd probably want a more machine readable form for the data, rather than HTML, and at the moment that's JSON. You should be able to request JSON either through the HTTP Accept header, or by using the .json extension on the URL path.

Deployments

There's not just one deployment of the Guix Data Service, currently I know of two. There's data.guix.gnu.org which just tracks the master branch of Guix, and has data going back to roughly the start of 2019 (with some gaps). There's also data.guix-patches.cbaines.net which isn't limited to just the master branch, and has additional branches constructed from patches that are submitted, but doesn't have much historical data.

Looking forward

There's still lots of areas where the Guix Data Service can be improved.

It would be convenient if getting data in to the Guix Data Service was faster, currently there's quite a delay between the Guix Data Service finding out about a new revision, and it completing processing it.

The processing could also be improved, there's some notable current omissions like the package graph (inputs, propagated-inputs and native-inputs) as well as package replacements (grafts). It would also be interesting to see if the Guix Data Service could be generalised to process other channels, instead or in addition to the main Guix channel.

In the future, I'd like to make the data available in formats other than JSON, like RDF. I'd also like for it to be possible to watch/subscribe to particular things that the Guix Data Service knows about, the Guix Data Service would then notify you via some means that there's be a change. This could enable all sorts of applications to respond to changes connected to Guix.

Additional reading

To provide some information, and to help get my own thoughts in order, I sent out semi-regular emails about the Guix Data Service over the last two years, I've linked to most of these below:

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Christopher Baines at November 08, 2020 08:30 PM

November 06, 2020

Programming Praxis

Surfraw

For the last few days I’ve been playing with surfraw, a command-line program that allows easy access to a variety of search engines. About a hundred elvi (search agents) are provided, and it is easy to add your own. You can get surfraw from your favorite code repository (for me, it was sudo apt install surfraw). I use lynx for the browser, so everything works on the command line.

Your task is to install surfraw on your machine, write a custom elvi, and share it with the rest of us. When you are finished, you are welcome to read a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at November 06, 2020 10:00 AM

Vladimir Nikishkin

Let us discuss SRFI-216: SICP Prerequisites

Abstract

001-Sicp_js.png

This post is a not-so-technical introduction to the Scheme Request for Implementation 216: SICP Prerequisites, that I have written and made available for discussion. (Please, contribute!)

SICP stands for the “Structure and Interpretation of Computer Programs”, a world-famous introductory programming textbook.

It’s aim is to make the exercises and examples from the book be available for any Scheme system bothering to provide it, and not just MIT/GNU Scheme and Racket (which doesn’t even consider itself a Scheme any more). Before this SRFI an issue tracker request asking for SICP support would have been looking vaguely. Now you can just write “Could you consider providing SRFI-216 (and 203)” in your implementation.

In order to write this SRFI, I went through the whole book and solved all the exercises. However, my experience is just mine, and to make a truly good common vocabulary, community feedback is required.

For technical detail and more background, I am inviting you to read the whole article.

What is SICP and why is it good?

002-800px-Massachusetts_Institute_of_Technology_logo.svg.png

SICP used to be an introductory course in programming in the Massachusetts Institute of Technology.

Its aims are twofold.

On one hand, it aims to provide sort of a bridge between the level of programming at which programmers reason about basic computational units, such as half-adders and XOR gates, and the level at we can reason about everyday things, such as “please highlight all misspelled words”.

On the other hand it tries to expose the students to as many software design/architectural patterns (that are actually thought out to add value rather than overcome the limitations of a language) as possible.

Both of these goals are enormous, and the course is itself enormous in size.

Still, passing it gives that (false) feeling of almightiness that is just so seductive that trying it once, the student is almost bound to pursuing it later on.

And apart from the sheer difficulty of the course, there are technical problems too.

Which technical problems?

003-800px-MIT_GNU_Scheme_Logo.svg.png

To keep the long story short, the Scheme world is not a place for the weak.

SICP challenges the student extra hard in both the questions it asks, and in the questions it does not ask. In particular, it speaks almost nothing about any particular implementation of Scheme, apart from a few words about MIT/GNU Scheme. Recalling that MIT Scheme was initially conceived as a software simulator for a hardware chip will already give you a sketch of an emotion that a serious student tackling SICP is experiencing all the time.

Implementations matter a lot for a language as high-level as Scheme. In C you can get by with a lot of things, because the language itself has direct memory access, so you are allowed to bend the rules for your own benefit pretty much in an unlimited fashion.

However, Scheme is a high level language, and the developers paid a lot of effort to make sure that it remains as abstractly formulated, as possible, to avoid “designing themselves in the corner” from which it would be very hard to escape.

But a side-effect of this is that a lot of features that modern programmers take for granted, such as random numbers or time and date query are unavailable to pure Scheme programmers.

SICP uses several of those, as its aim as a programming language textbook is to introduce the student to as many programming concepts, as possible, even if they have not yet been as cleanly and abstractly formulated, as is desirable to be included into the Scheme standard.

Why would I even want to solve SICP in some other Scheme?

One of the main benefits of the SICP is that it teaches the readers how to build an “artificial intelligence system” (in this case, a high-level programming system) on almost any Turing-complete substrate.

But the more frustrating then is to realise that completing it entirely is only possible on two systems, one of which is highly peculiar and does not support Windows (MIT, officially), and the other one doesn’t even call itself Scheme.

Moreover, nowadays the main strength of the SICP is not the strength of a general-purpose language (even though writing standalone programs is still completely possible, and Cisco still maintains its own implementation), but as an extension language that can be built into almost any software product imaginable, written in any language.

There are Schemes working on top of JVM, CLR, Fortran, JavaScript. Scheme is an extension language for GNU Debugger, GNU GIMP and GNU Guix.

For an interested programmer, it’s the most reasonable to choose the programming system for mastering SICP that is most like to fit well into the daily workflow.

This is one of the main goals of this SRFI.

My contribution

Since may years has passed since the release of SICP, Second Ed., the language has evolved, and several widely accepted libraries have emerged, that now make it possible to pass the course without digging into the gory details of interpreter implementation, as was required years ago, if you wanted to use any Scheme other than MIT/GNU.

Some of the features still remain a no man’s land in the sense that no overwhelmingly acceptable abstraction has emerged to be included into the Scheme. The most noticeable missing jigsaw bit is the graphics output. Frankly speaking, in the age of extra powerful HTML, it is unlikely that it will ever emerge.

The author of this text, therefore, had to deal with the graphics separately, as described in https://lockywolf.wordpress.com/2020/07/18/proposing-programming-language-features/ and the SRFI-203.

This work, however, speaks about the features that either have been included into the standard, of have widely accepted abstractions.

The artefact

004-srfi.png

The result of my work is a Scheme Request For Implementation, a non-normative Scheme community standard.

It includes two constants, five functions, and one syntactic extension to the core language.

Boolean values.

Many people would remember that it took quite a while until the C language got a dedicated logical type. Well, for Scheme it took even longer, and therefore SICP has to refer to the two variables true and false, that are an abstraction over some (unknown) logical types.

Now that Scheme standard has #t and #f, it makes a lot of sense to bridge this gap by equating SICP values and standard values.

Time queries.

Scheme hasn’t had a standard function to find current time for quite a while. Now it does have one, it is possible to implement the most basic tool for performance profiling (timer) portably. This way the runtime function was implemented.

Random numbers.

How would you implement random numbers on a machine that has not access to a clock, an entropy source, or a decaying atom? It’s actually a non-trivial question! Therefore, the Scheme standard still does not have a way to generate (pseudo-)random numbers.

Luckily, the need in random numbers is so large that a widely accepted abstraction has appeared ages ago, and for me it was necessary to just repackage the code.

This way the random procedure was born.

Multi-threading.

005-1200px-Multithreaded_process.svg.png

Multi-threading was a hot topic about ten years ago, when I was still at a university. It has not yet been fully researched, deadlocks and race conditions still arise quite a lot, despite several languages that claim to be very fit for multi-threaded programming being available on the market.

The two most basic tools that are required for properly understanding multi-threading are something that “allows programs to run in parallel” and something that “allows one program to make sure that it is not interfering with the some other one”.

Of course, there are numerous other tools, but the parallel-execute, and test-and-set! (also known as atomic-compare-and-swap) are the basic foundation on which everything else can be built.

I implemented them on top of the “SRFI-18”, which itself borrows inspiration from Java. It is funny to implement mutexes on top of other mutexes, but that’s the price of two models not being completely corresponding.

Streams.

Streams are infinite lists. They would not have appeared here at all, if SICP had spoken at least a tiny bit about native mechanisms for syntactic extension.

Alas! SICP was written in the years when Scheme had so-called “non-hygienic” macros, but still says nothing about them.

The most useful cons-stream syntactic structure is, therefore, impossible to implement without writing your own Scheme, which by the Chapter 3 is still not possible.

I had, therefore, implement it myself, on top of the last (R7RS) standard’s syntax-rules.

How can I help?

Firstly, no good standards appear unless a lot of people examine them. Code and document review are extremely needed.

Secondly, if you have a “favourite” Scheme implementation, you could try lobbying the provision of this SRFI, in order to both make the implementation more attractive to the users, and to promote SICP.

Tell your friends, students, professors, and enthusiasts, that studying SICP does not have to be a process full of pain.

If you are teaching programming, functional programming, or informatics (or your friends do), suggest that they have a look at the proposal, their feedback would be the most useful.

Conclusion

The SRFI-216, and the SRFI-203, combined, provide all the features that are required from a Scheme implementation to host the book.

I’m hoping that this work may extend the period of relevance for Scheme, SICP, and help popularise such an in-depth and unorthodox course.

The discussion is done through the mailing list. For details, please, consult https://srfi.schemers.org/srfi-216/.

Contacts and Donations

This post was not funded by anyone, so if you liked it and/or find it useful, please consider donating or subscribing.

Donate

Subscribe

by lockywolf at November 06, 2020 05:07 AM

October 30, 2020

Mark Damon Hughes

Reinventing the Wheel

Sure, there's existing code. Somebody Else's Code. It works fine, maybe not as fast as you'd like, or the interface isn't quite right. That's how it often is with me and SRFI-13. Olin Shivers is a skilled Schemer, back when that wasn't cool (OK, it's still not cool), but some of his APIs enshrined in early SRFIs drive me a little nuts, and the implementation is slow because it's so generalized.

So after a few false starts and failed tests, I now have these pretty things: (updated 2020-11-10, enforced hstart, hend boundaries)

;; Returns index of `needle` in `haystack`, or  if not found.
;; `cmp`: Comparator. Default `char=?`, `char-ci=?` is the most useful alternate comparator.
;; `hstart`: Starting index, default 0.
;; `hend`: Ending index, default (- haystack-length needle-length)
(define string-find (case-lambda
    [(haystack needle)  (string-find haystack needle char=? 0 )]
    [(haystack needle cmp)  (string-find haystack needle cmp 0 )]
    [(haystack needle cmp hstart)  (string-find haystack needle cmp hstart ) ]
    [(haystack needle cmp hstart hend)
        (let* [ (hlen (string-length haystack))  (nlen (string-length needle)) ]
            (set! hstart (max 0 (min hstart (sub1 hlen))))
            (unless hend (set! hend (fx- hlen nlen)))
            (set! hend (max 0 (min hend hlen)) )
            (if (or (fxzero? hlen) (fxzero? nlen))
                
                (let loop [ (hi hstart)  (ni 0) ]
                    ;; assume (< ni nlen)
                    ;(errprintln "hi=" hi ", ni=" ni ", hsub=" (substr haystack hi hlen) ", bsub=" (substr needle ni nlen))
                    (cond
                        [(cmp (string-ref haystack (fx+ hi ni)) (string-ref needle ni))  (set! ni (fx+ ni 1))
                            ;; end of needle?
                            (if (fx>=? ni nlen)  hi  (loop hi ni) )
                        ]
                        [else  (set! hi (fx+ hi 1))
                            ;; end of haystack?
                            (if (fx>? hi hend)    (loop hi 0) )
                        ]
        ))))
    ]
))

;; Test whether 'haystack' starts with 'needle'.
(define (string-has-prefix? haystack needle)
    (let [ (i (string-find haystack needle char=? 0 0)) ]
        (and i (fxzero? i))
))

;; Test whether 'haystack' ends with 'needle'.
(define (string-has-suffix? haystack needle)
    (let* [ (hlen (string-length haystack))  (nlen (string-length needle))
            (i (string-find haystack needle char=? (fx- hlen nlen)))
        ]
        (and i (fx=? i (fx- hlen nlen)))
))

Written for Chez Scheme, caveat implementor. BSD license, do what thou wilt. If you find a bug, send me a failing test case.

I don't normally bother with fx (fixnum) operations, but in tight loops it makes a difference over generic numeric tower +, etc.

by mdhughes at October 30, 2020 08:03 AM

October 27, 2020

Programming Praxis

List Slices

During some recent personal programming, I needed a function to slice a list into pieces: Given a list of sub-list lengths, and an input list to be sliced, return a list of sub-lists of the requested lengths. For instance:, slicing the list (1 2 2 3 3 3) into sub-lists of lengths (1 2 3) returns the list of sub-lists ((1) (2 2) (3 3 3)). Extra list elements at the end of the input list are ignored, missing list elements at the end of the input list are returned as null lists.

Your task is to write a program that slices lists into sub-lists according to a specification of sub-list lengths. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at October 27, 2020 09:00 AM

October 20, 2020

Programming Praxis

Doubled Pairs

Today’s exercise is somebody’s homework problem:

You are given an array containing positive integers, all distinct. Write a program that determines if there exist in the array any numbers such that one is double the other. For instance, in the array [1,2,3,4], the pairs 1,2 and 2,4 are both doubles, so the program should return True; in the array [1,3,5,7,9] there is no such doubled pair, so the program should return False. For extra credit, return a pair that proves the result is True, if it exists. For extra-extra credit, return all such pairs. Your program must run in linear time.

The student who asked for help realized there was a simple nested-loop solution that runs in quadratic time, but that doesn’t meet the constraints of the homework problem.

Your task is to solve all three levels of the homework problem. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at October 20, 2020 09:00 AM

October 16, 2020

GNU Guix

Announcing the first online Guix Day Conference

The Guix hackers are very happy to announce the first online Guix Day Conference on Sunday November, 22nd. This conference is open to everyone and will be held entirely online. Want to speak? Submit your proposal!

Important dates:

  1. November 6th: Deadline for talks proposal.
  2. November 14th: Deadline for releasing your pre-recorded talks.
  3. November 16th: Release of the schedule.
  4. November 22nd: Conference day!

Guix Days logo

The agenda of the day is:

  • pre-recorded talks with live question and answer sessions
  • birds of a feather (BoF) sessions
  • lightning round talks, if possible
  • hack together

There will be no presentation on the 22nd! And no registration fee.

Until November 6th: talks proposal

Propose your talks by sending them to guix-days@gnu.org. Feel free to drop in #guix on irc.freenode.net to discuss what you would like to talk about before submitting. :)

Please describe with 10 lines or more what your proposal is about. Even if it is a BoFs topic (smaller group who want to talk about specific topics).

Once you have sent your proposal, you will be notified in the coming days whether your talk be part of the Guix Day.

Good topics include your own experience with Guix and what you feel is important to share with your other fellows, for example a non-exhaustive topic list is: installer, Maven build system, Data Service, GNU Hurd and cross-compilation, Cuirass and continuous integration, authentication, secret services, website translation, translation infrastructure,… It is a single day so we won't be able to cover all. ;-)

November 9th-14th: prepare your talk

The aim of the pre-recorded talk is to demonstrate new features, what you are hacking on, introduce the subject for easing the live question and answer sessions or BoFs. These pre-recorded talks should be 15-45 minutes long. Feel free to ask if you need help with the recording.

You are free to choose whichever storage platform you want (e.g., your own website, a PeerTube instance, a Nextcloud instance, etc.), but we will need to have access to the original file so we can publish it later on audio-video.gnu.org. Your video must be released under a license that at least allows anyone to copy and share it, for any purpose.

You will have to release the video publicly before November 14th, so everyone has a chance to see it before the conference. If you are not able to do so (for instance your server cannot handle a huge load), you can alternatively send us a private link to the video and we will upload it on audio-video.gnu.org. If you decide to do so, you will need to have the video ready by November 12th.

November 16th-22nd: watch the talks

Be sure to watch the pre-recorded talks before the conference. There will be no presentation on the 22nd.

November 22nd: participate

Coming soon! Stay tuned.

Code of Conduct

This online conference is an official Guix event. Therefore, the Code of Conduct applies. Please be sure to read it beforehand!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Guix Hackers at October 16, 2020 12:00 AM

October 15, 2020

Joe Marshall

Apropos of Nothing

Lisp programmers are of the opinion that [] and {} are just () with delusions of grandeur.

by Joe Marshall (noreply@blogger.com) at October 15, 2020 10:18 PM

Vasilij Schneidermann

State of Emacs Lisp on Guile

Update: Factual corrections to Robin Templeton’s work.

Update: Added an extra set of benchmarks for Guile 3 in a Debian Sid Docker container.

Disclaimer: I don’t use Guile. I hardly know it. There are other Scheme implementations I know far better. But since Guile Emacs is a hot topic with much hopes and unproven claims, I experiment with it every now and then. All “benchmark” results here are to be taken with caution, they’ve been created using Guile 2.2.6 and Emacs 26.3 on a Thinkpad X230t running Arch Linux.

With that out of the way, laurus from #emacs[1] reminded me that one of the reasons why Guile Emacs was possible in the first place is Guile’s language tower, with Emacs Lisp being one of the supported languages. But what does that mean? How complete is the Emacs Lisp support? What can it be used for? These and a few other questions are the topic of this blog post.

In need of a spec

Standardized programming languages have the great benefit of being based on a specification one can consult whenever in doubt of how things are supposed to behave. This allows several competing implementations to be developed, with their own unique strengths and benefits. if you adhere to the standard, switching between implementations isn’t hard and can help shaking out bugs, for example when compiling your C programs with different compilers.

Things get considerably harder if your chosen language decided to forego this approach and the correct behavior is defined by it, yet this didn’t stop people from writing alternative implementations for programming languages such as Python and Ruby. Emacs Lisp got into a similar situation ever since Guile was extended to the degree of supporting Emacs Lisp as an additional language. Provided your version of Guile is new enough, you can evaluate trivial code in the REPL:

scheme@(guile-user)> (define foo 1)
scheme@(guile-user)> foo
$1 = 1
scheme@(guile-user)> ,L elisp
Happy hacking with Emacs Lisp!  To switch back, type `,L scheme'.
elisp@(guile-user)> (defvar bar 2)
$2 = bar
elisp@(guile-user)> bar
$3 = 2

So far so good. But how much of Emacs Lisp is supported? Not much apparently, many common functions like message and error are unbound. It doesn’t seem possible to do anything with buffers or files either. This greatly limits the possibilities of writing useful scripts in Emacs Lisp[2]. One way of determining what exactly is supported would be consulting the source code, but where’s the fun in that when you could instead test it programmatically, thereby creating an executable spec…

Generating the spec

The usual test approaches fail me. Reading test inputs via stdin with read-string? Accessing the arguments with argv? Reading from a file with insert-file-contents? Obtaining an environment variable with getenv? None of that is supported. At least you can print to stdout with princ. I went for a slightly different approach instead, obtain a list of functions/variables[3] in a minimal Emacs environment, then generating a test file that checks their existence and prints a test summary. Here’s the code generation part:

(defun printf (fmt &rest args)
  (princ (apply 'format fmt args)))

(printf ";; elisp spec adherence test
(defvar passed 0)
(defvar failed 0)
(defun test-sym (pred sym)
  (if (funcall pred sym)
      (setq passed (1+ passed))
    (setq failed (1+ failed))))
(defun test-fun (sym) (test-sym 'fboundp sym))
(defun test-var (sym) (test-sym 'boundp sym))\n\n")

(mapatoms
 (lambda (atom)
   (when (fboundp atom)
     (printf "(test-fun '%S)\n" atom))
   (when (and (not (keywordp atom)) (boundp atom))
     (printf "(test-var '%S)\n" atom))))

(printf "\n")
(printf "(princ \"Passed: \")\n")
(printf "(princ passed)\n")
(printf "(terpri)\n")
(printf "\n")
(printf "(princ \"Failed: \")\n")
(printf "(princ failed)\n")
(printf "(terpri)\n")

Assuming it’s been saved as gen-elisp-spec.el, the executable spec itself is generated with emacs -Q --batch --script gen-elisp-spec.el > elisp-spec.el. Here’s a test session using Emacs and Guile:

[wasa@box ~]$ time emacs -Q --batch --script elisp-spec.el
Passed: 9567
Failed: 2
emacs -Q --batch --script elisp-spec.el  0.10s user 0.02s system 99% cpu 0.117 total
[wasa@box ~]$ time guile --language=elisp elisp-spec.el
Passed: 137
Failed: 9432
guile --language=elisp elisp-spec.el  77.62s user 0.27s system 103% cpu 1:15.04 total

This is kind of surprising. I didn’t expect Emacs to fail its own test and didn’t expect Guile to implement this little either. Most surprising is the abysmal speed of the test[4], I’m looking forward to anyone being able to explain that part to me. Here’s one more test using the official Debian Sid Docker image with Emacs 26.3 and Guile 3.0.2:

root@d27668492764:/# time emacs -Q --batch --script elisp-spec.el
Passed: 9108
Failed: 2

real    0m0.104s
user    0m0.097s
sys     0m0.007s
root@d27668492764:/# time guile --language=elisp elisp-spec.el
Passed: 137
Failed: 8973

real    6m20.950s
user    10m32.294s
sys     0m7.846s

This is not exactly an improvement. At least the numbers are small enough to print out the offending symbols, for Emacs it’s atom and printf (which polluted the test environment), for Guile I’ve taken the liberty of annotating the list:

;; binding
let let*
;; functions
lambda apply funcall
;; evaluation
eval load eval-and-compile eval-when-compile
;; sequences
aref aset make-vector nth
;; sequencing
progn prog2 prog1
;; iteration
dolist while
;; control flow
if when unless cond
;; short-circuiting
or and not
;; explicit nonlocal exit
throw catch
;; exceptions
signal condition-case unwind-protect
;; input
read-from-minibuffer
;; output
prin1-to-string print princ send-string-to-terminal terpri
;; cxr
car cdr caar cadr cdar cddr
car-safe cdr-safe
nthcdr
;; associations
assoc assq
;; search
member memql memq
;; destructive list processing
nreverse setcar setcdr rplaca rplacd
;; other list processing
cons list make-list `
mapcar mapc
append concat
reverse
length
;; symbols
defconst defvar defun defmacro
get put
fset set setq setplist
symbol-function symbol-name symbol-plist symbol-value
intern make-symbol
fmakunbound makunbound
quote function
;; plist
plist-get plist-put
lax-plist-get lax-plist-put
plist-member
;; strings
string string-match substring
upcase downcase
;; predicates
zerop floatp stringp numberp integerp wholenump
boundp fboundp functionp symbolp
consp listp nlistp
atom null
;; math
1+ 1-
fceiling ffloor ftruncate fround float
abs
min max
;; comparators
> < >= <= /= =
eq eql equal
string-equal string=
;; numerical operators
+ - * %
;; misc
random

Some notable omissions and differences:

  • No division. Most likely incompatible with Scheme’s numeric tower.
  • Input is read with read-from-minibuffer, not read-string.
  • send-string-to-terminal is unusual to have, but most likely the primitive output function.
  • string-match is nice to have, but of limited use without match-string.
  • prin1-to-string exists, prin1 doesn’t.
  • print doesn’t add newlines and behaves like prin1 should.

To do anything outside of textbook exercises you’d need to define extra primitives. Guile’s module/language/elisp/boot.el shows how to apply a band-aid on some of the previous shortcomings:

(fset '/ (@ (guile) /))
(fset 'read-string 'read-from-minibuffer)
(fset 'prin1 (@ (guile) write))
(defun print (object) (prin1 object) (terpri))

You could write more of it to reach that goal of using Emacs Lisp as scripting language outside of Emacs, but need to write Scheme to get there. Why not just use Scheme? Why invent a new runtime? The effort would be comparable to what node.js did for Chrome’s JavaScript engine, except with a far weaker sales-pitch.

What does this mean for Guile Emacs?

What I’ve shown above is barely sufficient to bootstrap an Emacs on top of it. Guile Emacs requires a customized version of Guile and Emacs, then loads up the supporting Emacs Lisp files to do the rest. There are more incompatibilities, like called-interactively-p being stubbed out. Extending the presented rudimentary spec to contain actual tests would help with tracking progress and usability. It might even improve the overall quality of GNU Emacs itself, provided that the core developers are on board and believe in the idea. I’ve briefly searched emacs-devel for previous discussion on the topic, but only found bikeshedding about Guile Emacs itself, so anyone who feels strongly about the subject, feel free to start a discussion there!

With regards to Guile Emacs itself, the situation is trickier. The above repositories have not been touched for five years, with Robin Templeton being the sole contributor for five Google Summer of Code events. Even though the work is far from complete, it is impressive what a college student managed to do under supervision of Guile’s maintainer Andy Wingo and Ludovic Courtès. Further advancements require similarly motivated individuals to participate in the Guile community and become part of the effort, much like with other free software projects. It’s tempting to take a shortcut like donating to other developers, but unless they’ve figured out a way of converting that money into equivalent work, there will be little connection between what you give away and what they do in return. This again is a topic worth discussing, preferably with the people that can make a change.

[1]laurus did some research as well, you can find an interesting discussion on the #guile channel: http://logs.guix.gnu.org/guile/2020-05-16.log
[2]At least you could now solve SICP in Emacs Lisp with less footguns: You have bignums, lexical scoping by default and TCO!
[3]This isn’t exactly correct, what’s tested for is whether the symbol has its function/value slot bound which may contain other things, for example macros and keywords.
[4]Consider that people like to advocate for Guile Emacs with the argument that it will make for a faster Emacs. While this may hold true in the long term, it’s nowhere near close to that yet. Here’s hoping that Guile 3 will alleviate some of the pain…

by Vasilij Schneidermann at October 15, 2020 07:58 PM

October 08, 2020

GNU Guix

Childhurds and GNU/Hurd substitutes

A lot has happened since our Hello Hurd post beginning of April. No, not nearly as much as we joked on April 1st , but more than enough to share and be proud of.

Building a Hurd virtual machine

As some of you noticed, the previous hacks to build a Hurd virtual machine (VM) were removed and no longer work; using Guix you can now build a GNU/Hurd VM just like you would build a GNU/Linux VM:

guix system disk-image -t hurd-raw bare-hurd.tmpl

This cross-compiles all the relevant packages for GNU/Hurd—specifically the i586-pc-gnu triplet—and produces a VM image:

/gnu/store/n7jkfajw0fzp975hv0b9v18r9bbr961q-disk-image

You can build it and start it from your GNU/Linux machine with this command:

qemu-system-i386 -enable-kvm -m 512 -snapshot -hda \
  $(guix system disk-image -t hurd-raw bare-hurd.tmpl)

We are using this ready-made, minimal GNU/Hurd operating system description gnu/system/examples/bare-hurd.tmpl that looks suprisingly familiar:

(use-modules (gnu) (gnu system hurd) (guix utils))
(use-service-modules ssh)
(use-package-modules ssh)

(define %hurd-os
  (operating-system
    (inherit %hurd-default-operating-system)
    (bootloader (bootloader-configuration
                 (bootloader grub-minimal-bootloader)
                 (target "/dev/sdX")))
    (file-systems (cons (file-system
                          (device (file-system-label "my-root"))
                          (mount-point "/")
                          (type "ext2"))
                        %base-file-systems))
    (host-name "guixygnu")
    (timezone "Europe/Amsterdam")
    (packages (cons openssh-sans-x %base-packages/hurd))
    (services (cons (service openssh-service-type
                             (openssh-configuration
                              (openssh openssh-sans-x)
                              (use-pam? #f)
                              (port-number 2222)
                              (permit-root-login #t)
                              (allow-empty-passwords? #t)
                              (password-authentication? #t)))
               %base-services/hurd))))

%hurd-os

and it can be customized just like a GNU/Linux operating system description. The end result is a full-blown Guix System with the Shepherd managing system services and all that—finally we can run herd on the Hurd.

A lot of things had to be in place to support this, we worked on

counting some ~200 patches by ten people over six months; including generic cross-compilation fixes and support, and Hurd fixes and support.

Also, we finished the passive translator settings over extended attributes (xattrs) for the Hurd and for Linux.

You may notice that we are using the new disk-image command rather than the old vm-image. One of the big hurdles in producing a VM image was the way Guix produces VM images: it would run a target QEMU, e.g. qemu-arm. That does not work for the Hurd, as there is no qemu-hurd. Without going into the hairy details, when Ludo and Janneke were—three patch sets, 50 messages and 13 days later—almost ready to give up, Mathieu came to the rescue with his brand-new implementation of the disk-image command. At the time, Hurd work was done on the wip-hurd branch and the disk-image work on wip-disk-image. Soon after, Mathieu proposed an explosive mix of the two branches; we managed to create the first Hurd system that really felt like Guix System.

The new implementation of the disk-image command was followed by the introduction of an --image-type or -t option. This option allows to produce disk images targeting different supports. The hurd-raw and hurd-qcow2 image types, producing respectively a raw Hurd disk-image and a Hurd QCOW2 disk-image were introduced. They can be used this way:

guix system disk-image -t hurd-raw bare-hurd.tmpl
guix system disk-image -t hurd-qcow2 bare-hurd.tmpl

This mechanism providing much more flexibility in the Guix System image generation will be described in a future blog post.

We also offer downloads of continuously built (actually cross-built) Guix System on GNU/Hurd (~350 MiB).

Substitutes

While amazing to be able to just run the Hurd in a VM, development without substitutes is a real pain; a Guix system needs substitutes. How to go about that? We have a build farm but currently Hurd only runs on ancient hardware: not an option. We need a machine running Guix/Hurd to offload Hurd build jobs to.

The proposed solution was to automate the building and running of a Guix VM into a new Guix service: the so-called hurd-vm or childhurd service. We would add something like:

(service hurd-vm-service-type)

to a couple of our build nodes to add a hird of virtual Hurd machines to our build farm.

The Childhurd service

The Hurd—being based on a microkernel—has some beautiful built-in "virtualization" possibilites that have their own naming: neighborhurds, subhurds. Similarly, we are adding childhurds to this mix: a childhurd is a GNU/Hurd VM running on GNU/Linux and managed by Guix.

When you are running Guix System, building a Hurd VM manually is no longer necessary. Just add the hurd-vm service to your operating system description:

(service hurd-vm-service-type
         (hurd-vm-configuration
          (disk-size (* 12 (expt 2 30))) ;12GiB
          (memory-size 1024)))           ; 1GiB

and this will build a childhurd for you when you reconfigure your system. This childhurd can be stopped and started just like any other service:

herd stop hurd-vm
herd start childhurd

(childhurd is an alias for hurd-vm).

WARNING

This Hurd is fully operational

It is highly addictive.

Having shaved this yak, let’s not lose sight that our initial goal was to offload builds to those GNU/Hurd VMs. The childhurd service does all the heavy lifting. To offload from GNU/Linux to my childhurd, I added this to my /etc/guix/machines.scm:

 (build-machine
  (name "localhost")
  (systems (list "i586-gnu"))
  (host-key "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHZsrZ63zs+AhWbVJgYq6j1h2rgQGrWKCokpR2/Q/Jzy root@guixygnu")
  (port 10022)  ;the Hurd VM has SSH listening on that port
  (user "root")
  (private-key "/home/janneke/.ssh/id_rsa_childhurd"))

That can be used to transparently offload builds from GNU/Linux to the childhurd:

$ uname -o
GNU/Linux
$ guix build hello -s i586-gnu
The following derivation will be built:
   /gnu/store/jqdvjhcxxnbq370y8i2c973c9zfiqrgl-hello-2.10.drv
[…]
guix offload: sending 1 store item (12 MiB) to 'localhost'...
offloading '/gnu/store/jqdvjhcxxnbq370y8i2c973c9zfiqrgl-hello-2.10.drv' to 'localhost'...
[…]
/gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
$ file -L $(guix build hello -s i586-gnu)/bin/hello
/gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10/bin/hello: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /gnu/store/9vs3gkp6svam82zw7vjlml7iiarcs11c-glibc-2.31/lib/ld.so.1, for GNU/Hurd 0.0.0, not stripped

Hurrah!

Hurd substitutes

Last Friday we produced the very first GNU Hello substitute for the Hurd:

StorePath: /gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
URL: nar/gzip/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
Compression: gzip
FileSize: 61822
URL: nar/lzip/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
Compression: lzip
FileSize: 52887
NarHash: sha256:0g4k1kppjs5148ynm4zw4x1kpaby67npc3ws6s7y7hf0il1cgryk
NarSize: 204328
References: 803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10 9vs3gkp6svam82zw7vjlml7iiarcs11c-glibc-2.31 bwvd5338kfm0vsc4i9xvh48vdxr5ywrz-gcc-7.5.0-lib
System: i586-gnu
Deriver: jqdvjhcxxnbq370y8i2c973c9zfiqrgl-hello-2.10.drv
Signature: 1;berlin.guix.gnu.org;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjMDVGOEY5NjMxRUU5QzcxM0REQUNBRTYwNUNCNjJBNzlDNUY4NEVFQTIwMjc5OERBNTQ3NURCOUU2Q0FBRDMwMSMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDVBMzkzMTgwOUY1RkQyMTdGMDM4MUVDMTJEODYyNzIyOEYyNjJGRDA4MTcxQjREMzZBNEM0RjBBNjZEQkY4NSMpCiAgIChzICMwQTc2RjZGNENCOTMzQTczNzA4QkNGMzRGREExMzkyOTRGQTQxREQzQTUwQkEwOUE0ODRCQUQyOTA4MjQ5ODIxIykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjOEQxNTZGMjk1RDI0QjBEOUE4NkZBNTc0MUE4NDBGRjJEMjRGNjBGN0I2QzQxMzQ4MTRBRDU1NjI1OTcxQjM5NCMpCiAgICkKICApCiApCg==

For development, porting and fixing of packages, you can use a Childhurd configuration like this:

(use-modules (srfi srfi-1) (guix packages) (guix records))
(use-package-modules base compression file gawk gdb hurd less m4 package-management)

(define guix-packages
  (filter-map input->package
              (fold alist-delete (package-direct-inputs guix)
                    '("glibc-utf8-locales" "graphviz" "po4a"))))

(operating-system
  (inherit %hurd-vm-operating-system)
  (users ...)
  (packages
    (cons* diffutils file findutils gawk gdb-minimal git-minimal
           gnu-make grep gzip less m4 openssh-sans-x tar xz
           (append
            guix-packages
            (delete guile-3.0 %base-packages/hurd)))))

… which allows working from a Git clone:

15:51:05 janneke@dundal:~/src/guix/master [env]
$ ssh -A janneke@childhurd
Last login: Sat Oct  3 15:31:55 2020 from 10.0.2.2


  This is the GNU Hurd.  Welcome.

janneke@childhurd ~$ git config --global url."git+ssh://git.sv.gnu.org/srv/git/".insteadOf gnu:
janneke@childhurd ~$ git clone gnu:guix
Cloning into 'guix'...
remote: Counting objects: 394436, done.
remote: Compressing objects: 100% (85728/85728), done.
remote: Total 394436 (delta 309572), reused 392294 (delta 307893)
Receiving objects: 100% (394436/394436), 137.05 MiB | 1.18 MiB/s, done.
Resolving deltas: 100% (309572/309572), done.
Updating files: 100% (2199/2199), done.

but before we continue let's first check the weather:

janneke@childhurd ~$ guix weather
computing 11079 package derivations for i586-gnu...
looking for 11521 store items on https://ci.guix.gnu.org...
updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
https://ci.guix.gnu.org
  1.5% substitutes available (169 out of 11521)
  at least 443.3 MiB of nars (compressed)
  966.7 MiB on disk (uncompressed)
  0.012 seconds per request (142.2 seconds in total)
  81.0 requests per second
[..]

this gives an idea of how young this project is. Any day now, more Hurd substitutes will follow. Now let's configure and build Guix:

janneke@childhurd ~$ cd guix
janneke@childhurd ~/guix$ guix environment --bootstrap \
  --ad-hoc gcc-toolchain@7 libgcrypt zlib
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/gxq5flc8kwpn999dw9xxvldy9xfd0q2x-profile.drv
   /gnu/store/vfkjnwgl4ckyklrl2z4q8x2vnlrwwyfr-gcc-toolchain-7.5.0.drv
   /gnu/store/zh6snj49ayrpw24jn7whpzygj1fpy9cm-module-import-compiled.drv

66.3 MB will be downloaded
[..]
building profile with 3 packages...
janneke@childhurd ~/guix [env]$ ./bootstrap
[..]
janneke@childhurd ~/guix [env]$ ./configure --with-courage\
  --localstatedir=/var --sysconfdir=/etc
[..]
janneke@childhurd ~/guix [env]$ make
[..]
janneke@childhurd ~/guix [env]$ ./pre-inst-env guix build hello
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
0.1 MB will be downloaded:
   /gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
substituting /gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10...
downloading from https://ci.guix.gnu.org/nar/lzip/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10 ...
 hello-2.10  52KiB                    258KiB/s 00:00 [##################] 100.0%

/gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10

just like we are used to do…almost. We are using --bootstrap and a targeted --ad-hoc to avoid dependencies like libx11, python-minimal, and other packages that do not build yet.

Isolated build environments

To help achieve reproducible builds, Guix builds packages in isolated build environments: build environments contain nothing but the inputs explicitly declared in the package definition—not even /bin/sh. Build environments also lack network access. On GNU/Linux this is achieved by running builds in separate namespaces. Besides, the environment contains device nodes and “special” file systems that are usually expected to be available: /dev/null, /dev/pts, /dev/shm, /proc, the loopback networking device, and so on. (The exact contents are documented.)

On GNU/Linux, these special files and file systems are implemented by the kernel. Guix only cares about user-land software, meaning that these devices “leak” from the host kernel instead of being an explicit “input” of derivations, but that’s OK, that’s the deal: the kernel and hardware are considered outside of Guix’s control.

What about GNU/Hurd, though? In GNU/Hurd, /dev/null, a Linux-compatible /proc, the TCP/IP stack necessary to implement the loopback device, and even support for pipes are all implemented in user-land: writing to /dev/null amounts to talking to the /hurd/null service (or translator), operations on AF_INET sockets translate to remote procedure calls (RPCs) to /servers/socket/2, which the /hurd/pfinet program listens to, and so on.

That raises an interesting question: what should the build environment contain on GNU/Hurd? So far our GNU/Hurd builds were made in non-isolated environments; we have just started implementing support for isolated builds but we’ll have to answer that question first. If we stick to our approach—every piece of user-land software must be an explicit input of the build process—then code that implements TCP/IP, /dev/null, or even pipe should be an explicit input of any build process that needs those facilities.

This principled approach can push the notion of controlled, reproducible build environments to a whole new level. For example, we’ve had cases where the choice of the root file system—e.g., ext4 vs. Btrfs—has an observable effect on software behavior, leading to concrete issues such as test failures in one case and not in the other. On GNU/Hurd, build processes could run their own root file system, doing away with this kind of discrepancy.

On the other hand, there are practical issues that cannot be ignored: virtually all build processes need these facilities so they’ll need to be set up one way or another. Also, one could argue that things like /dev/null have a well-defined interface that’s set in stone and that, consequently, how they’re implemented does not matter at all. Can we say the same of the TCP/IP stack though? Maybe not. A line needs to be drawn somewhere.

We have yet to decide where to draw the line and to precisely define what the build environment contains on GNU/Hurd. These questions are closely related to bootstrapping issues we notably discussed at the 2019 Reproducible Builds Summit. Tricky, but exciting.

What's next?

In an earlier post we tried to answer the question “Why bother with the Hurd anyway?” An obvious question because it is all too easy to get discouraged, to downplay or underestimate the potential social impact of GNU and the Hurd.

We tried to make Hurd development as easy and as pleasant as we could. As you have seen, things start to work pretty nicely and there is still plenty of work to do in Guix. But in a way this is “merely packaging” the amazing work of others. Some of the real work that needs to be done and which is being discussed and is in progress right now includes:

All these tasks look daunting, and indeed that’s a lot of work ahead. But the development environment is certainly an advantage. Take an example: surely anyone who’s hacked on device drivers or file systems before would have loved to be able to GDB into the code, restart it, add breakpoints and so on—that’s exactly the experience that the Hurd offers. As for Guix, it will make it easy to test changes to the micro-kernel and to the Hurd servers, and that too has the potential to speed up development and make it a very nice experience.

Join #guix and #hurd on irc.freenode.net or the mailing lists and get involved!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

About the GNU Hurd

The GNU Hurd is the GNU project's replacement for the Unix kernel. It is a collection of servers that run on the Mach microkernel to implement file systems, network protocols, file access control, and other features that are implemented by the Unix kernel or similar kernels (such as Linux). More info.

The mission of the GNU Hurd project is to create a general-purpose kernel suitable for the GNU operating system, which is viable for everyday use, and gives users and programs as much control over their computing environment as possible.

by Jan (janneke) Nieuwenhuizen, Ludovic Courtès, Mathieu Othacehe at October 08, 2020 02:15 PM