AI Foundry lab · runbook

Add an account on the lab machines

One script creates a login on aifoundry1, aifoundry2 and aifoundry3, the shared ET-SoC-1 machines, and checks that the account can use the cards. Use it for your own account or for someone else's.

The commands below use these values. The key is optional, because Tailscale logins don't need one.

  1. Check that you can reach the machines

    You need to be on the AI Foundry tailnet, and root logins have to work for you. The first time, Tailscale prints a URL: open it to confirm it's you.

    ssh root@aifoundry1 hostname

    It should print aifoundry1.

  2. Save the script

    Paste this into a terminal on your own computer. It writes add-lab-user.sh to the current folder and makes it executable.

    add-lab-user.sh · 50 lines
    cat > add-lab-user.sh <<'ADD_LAB_USER'
#!/usr/bin/env bash
# Create an account for someone on every AI Foundry lab machine (aifoundry1, aifoundry2, ...).
# Usage: add-lab-user.sh <username> [<ssh-pubkey-file>] ["Full Name"]
# Needs `ssh root@aifoundryN` to work from this machine. Safe to re-run: an existing account is
# left alone and the key is only added if it is missing.
set -euo pipefail

usage() { echo "usage: $0 <username> [<ssh-pubkey-file>] [\"Full Name\"]" >&2; exit 1; }
user=${1:-} keyfile=${2:-} name=${3:-}
[[ $user =~ ^[a-z][a-z0-9_-]{0,31}$ ]] || usage

key=
if [[ -n $keyfile ]]; then
  key=$(head -n1 "$keyfile")
  if [[ ! $key =~ ^(ssh-|ecdsa-|sk-) ]] || ! ssh-keygen -lf - <<<"$key" >/dev/null 2>&1; then
    echo "$keyfile is not an SSH public key (pass the .pub file)" >&2; exit 1
  fi
fi

# Every aifoundryN on the tailnet, unless HOSTS="aifoundry1 aifoundry2" is set.
hosts=${HOSTS:-$(tailscale status 2>/dev/null | awk '$2 ~ /^aifoundry[0-9]+$/ {print $2}' | sort)}
hosts=${hosts:-aifoundry1 aifoundry2 aifoundry3}

rc=0
for h in $hosts; do
  echo "== $h"
  {
    printf 'user=%q key=%q name=%q\n' "$user" "$key" "$name"
    cat <<'EOF'
set -euo pipefail
if id "$user" >/dev/null 2>&1; then
  echo "account exists: $(id "$user")"
else
  adduser --quiet --disabled-password --comment "$name" --shell /bin/bash "$user" </dev/null
  echo "created: $(id "$user")"
fi
if [ -n "$key" ]; then
  # Write as the user, so a symlink in their home cannot redirect a root write.
  runuser -u "$user" -- sh -c 'umask 077; mkdir -p "$1/.ssh"; f="$1/.ssh/authorized_keys"; touch "$f"
    if grep -qxF "$2" "$f"; then echo "key already present"; else printf "%s\n" "$2" >>"$f"; echo "key added"; fi' \
    sh "$(getent passwd "$user" | cut -d: -f6)" "$key"
fi
for d in /dev/et*; do
  [ -e "$d" ] || continue
  if runuser -u "$user" -- test -r "$d" -a -w "$d"; then echo "$d: read/write ok"; else echo "$d: NO ACCESS"; fi
done
EOF
  } | ssh -o BatchMode=yes -o ConnectTimeout=15 "root@$h" bash -s || { echo "!! $h: failed" >&2; rc=1; }
done
exit $rc
ADD_LAB_USER
chmod +x add-lab-user.sh

  3. Run it

    Save their public key next to the script as alice.pub. On their computer it is usually ~/.ssh/id_ed25519.pub.

    ./add-lab-user.sh alice alice.pub 'Alice Example'

    It finds every aifoundryN machine on the tailnet, creates the account where it is missing, adds the key, and checks that the account can read and write each /dev/et* device. Running it again is safe. To do only some machines, put HOSTS="aifoundry2" in front of the command.

    Example output (your numbers will differ)
    == aifoundry1
created: uid=1010(alice) gid=1010(alice) groups=1010(alice),100(users)
key added
/dev/et0_mgmt: read/write ok
/dev/et0_ops: read/write ok
/dev/et1_mgmt: read/write ok
/dev/et1_ops: read/write ok
== aifoundry2
created: uid=1020(alice) gid=1020(alice) groups=1020(alice),100(users)
key added
/dev/et0_mgmt: read/write ok
/dev/et0_ops: read/write ok
== aifoundry3
created: uid=1005(alice) gid=1005(alice) groups=1005(alice),100(users)
key added
/dev/et0_mgmt: read/write ok
/dev/et0_ops: read/write ok

  4. Send them the login steps

    If they are not on the tailnet yet, a tailnet admin has to invite them first (Tailscale admin console, Users, Invite users). If the account is yours, these are your next steps.

    Message
    Your account on the AI Foundry lab machines is ready: alice on aifoundry1, aifoundry2 and aifoundry3.

1. If you are not on the AI Foundry tailnet yet, accept the Tailscale invite and install Tailscale.
2. Log in with: ssh alice@aifoundry1
   If Tailscale prints a URL, open it to confirm it's you. It asks again every so often.
3. The ET tools are in /opt/et/bin. To put them on your PATH, run this once on each machine:
   echo 'export PATH=/opt/et/bin:$PATH' >> ~/.bashrc

The machines

MachineDevicesDevice nodes
aifoundry12/dev/et0_mgmt /dev/et0_ops /dev/et1_mgmt /dev/et1_ops
aifoundry21/dev/et0_mgmt /dev/et0_ops
aifoundry31/dev/et0_mgmt /dev/et0_ops

All three run Ubuntu 24.04. The device nodes are crw-rw-rw- (mode 0666), so any account can use the cards without joining a group. The ET tools are in /opt/et/bin. Each machine has its own /home, so files don't carry over between machines.

How logins work

Logins go through Tailscale SSH. The tailnet's SSH rules decide who may connect, so a machine only needs the account to exist, and there are no passwords to hand out. The SSH key is a fallback for plain sshd on the lab's local network.

Giving someone sudo

New accounts have no password, so they can't use sudo. To give someone sudo on one machine, add them to the sudo group and set a starting password, which they can change later with passwd:

ssh -t root@aifoundry1 'usermod -aG sudo alice && passwd alice'

Without the script

Run these for each machine. adduser asks for the full name, and the other questions can stay empty. The second command is only needed if they sent a key.

for each machine
ssh -t root@aifoundry2 adduser --disabled-password alice
ssh alice@aifoundry2 'umask 077; mkdir -p ~/.ssh; cat >> ~/.ssh/authorized_keys' < alice.pub

If something fails

!! aifoundry2: failed
Run ssh root@aifoundry2 hostname to see why. Usually Tailscale wants you to confirm in the browser, or the machine is offline.
/dev/et0_ops: NO ACCESS
The account can't read and write that device. Check ls -l /dev/et* on that machine: the nodes should be crw-rw-rw-.
Tailscale refuses their login
They are on the tailnet, but its SSH rules don't cover them yet. A tailnet admin needs to add them to the SSH rules in the tailnet policy file.