All posts

I just received my Framework Desktop Motherboard, equipped with the AMD Ryzen AI Max+ 395 and 128 GB of unified memory. Time to get the massive GPT-OSS model with 120 billion parameters running on this beast!

The plan is to use llama.cpp to run GPT-OSS 120b. But first, we need to get everything set up.

Increasing VRAM Allocation

Before even installing the operating system, I increased the VRAM allocation from the default 0.5 GB to a whopping 96 GB, this can be done through the BIOS.

Choosing a Distribution

For the operating system, I opted for Ubuntu Server 24.04.3, the latest version at the time of writing. While ROCm officially supports a few distributions, I chose Ubuntu because it’s the most widely used.

After installing the OS, make sure it’s up-to-date:

❯ sudo apt update
❯ sudo apt upgrade

Upgrading the Kernel

The Ryzen AI Max+ 395 is officially supported starting with ROCm version 7.0.2. However, to run ROCm 7.0.2, we need kernel 6.14 or later. Ubuntu 24.04 ships with kernel 6.8 by default, but we can easily upgrade to the Hardware Enablement (HWE) stack with a single command:

❯ sudo apt install --install-recommends linux-generic-hwe-24.04

Installing ROCm

The following commands worked at the time of writing, but always check the sources below for the latest updates.

First, install the amdgpu-install utility:

❯ sudo apt update
❯ wget https://repo.radeon.com/amdgpu-install/7.0.2/ubuntu/noble/amdgpu-install_7.0.2.70002-1_all.deb
❯ sudo apt install ./amdgpu-install_7.0.2.70002-1_all.deb

Then, use the utility to install ROCm drivers with the appropriate use case:

❯ amdgpu-install -y --usecase=rocm --no-dkms

Compiling Llama.cpp

Compiling llama.cpp on Ubuntu is straightforward, thanks to the official amdgpu-install. Just clone the repository and run the build command!

❯ git clone https://github.com/ggml-org/llama.cpp.git llama.cpp
❯ cd llama.cpp
❯ HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" cmake -S . -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release && cmake --build build --config Release -- -j $(nproc)

Running the Model

Finally, start a server with the following command:

❯ ./build/bin/llama-server -hf ggml-org/gpt-oss-120b-GGUF -c 0 -fa on --jinja --no-mmap --n-gpu-layers 99

This will launch a server accessible on port 8080. If you're running on a local network, you may want to add --host 0.0.0.0. I noticed the model loads very slowly, but adding --no-mmap significantly improved load times. The -c 0 flag ensures the model loads with its maximum context length of 128k tokens!

Sources

• Ubuntu kernel lifecycle and enablement stack
• Install Ryzen Software for Linux with ROCm
• Build llama.cpp locally on GitHub
• ggml-org/gpt-oss-120b-GGUF on Hugging Face

This is more of a note for the future me. For a while, any non‑Latin fonts were not rendering on my ArchLinux install. Here's how I fixed it!

If you've played on large Minecraft networks, you may have noticed that sometimes, instead of being kicked to the main menu when a server crashes or restarts, you're quietly placed in an empty world. That "empty room" is what we often call a Limbo server.

In today's digital age, securing sensitive data is of utmost importance. One effective way to protect your data on a USB drive is by using Linux Unified Key Setup (LUKS) encryption. LUKS is a powerful and flexible encryption standard that ensures your data remains secure even if your USB drive falls into the wrong hands. In this guide, we'll walk you through the process of encrypting a USB drive using LUKS.

In this article I will cover how I created a Ceph cluster on a single node.

This article will cover how to install Ubuntu on the EMMC storage of the RK1, configure it in read-only mode and use the NVME as storage for Docker.

I recently received the Turing Pi 2.5 with 4 Turing RK1 modules, I wanted to build and maintain my own operating system images for the RK1s.

I never remember how to configure fstab properly and always end up breaking my system. Let's write a guide on how I configure my fstab so that I don't make the same mistakes twice!

After using Silverblue for six months, the package manager was starting to get really slow, so I decided to switch distro, Arch was my choice. The goal was to install Arch Linux with Full Disk Encryption and EFI boot. Here's how I did it.

Linux feels a bit sluggish on the VisionFive 2, so I wanted to install OpenBSD on my board since a RISC-V port is available.

The newly announced model by Mistral AI can show some really great results, let's see how we can run it locally!

For a long time I've been using the integrated password manager in my browser of choice. Starting with Google Chrome, Firefox and ended up on Brave.

I've been using Linux for years now, one of the main issue I am still getting after all these years is having the package manager fail during an update.