Recently I setup CentOS on the Raspberry pi 2. Now we’re going to be installing a modern Fedora 23 desktop on the Raspberry Pi 3. I’ve used the Pi 2 instructions from chisight’s blog post so original credit goes to him – I’ve modified them and added some things.
Getting Started
You’ll need a few things to get started, we’ll be using a Fedora Desktop ARMv7 spin. I’ll be using XFCE as it’s my desktop of choice but you can substitute this for one you like. Note that we’ll be using the Raspberry Pi Kernels here and not the upstream, latest Fedora Kernels – if you’re interested in the latter please check out David Todd’s blog here.
- Raspberry Pi 3 Model B (Quad Core 1.2GHz)
- Fedora XFCE or similar F23 image for ARMv7
- A decent USB hub or power source
- An SD card for installing Fedora
- Another Linux or OSX computer with an SD card reader
Download/Prep Fedora Image and SD Card
We’ll be doing an image-based approach here, but I’m told that U-Boot does support PXE now so kickstart may be an option in the future.
Identify your SD Card Device
In Fedora and EL-based distributions SD cards are usually mmcblk0. If you’re unsure run this command below before inserting your SD card, it never hurts to double check.
watch lsblk
Assuming /dev/mmcblk0 is your correct SD card you’ll download and write out the image. I’m using the XFCE Desktop spin image below, substitute as needed.
wget http://mirror.karneval.cz/pub/linux/fedora/linux/releases/23/Images/armhfp/Fedora-Xfce-armhfp-23-10-sda.raw.xz xzcat Fedora-Xfce-armhfp-23-10-sda.raw.xz | dd of=/dev/mmcblk0
Note: the above image location was a closer mirror to me in Europe. You may want to pick a better mirror location and adjust accordingly if download speeds are slow.
Partition Magic and Expansion
At this point your SD card will have the F23 image written to it, but to boot we’ll need a vfat primary partition in partition 1. We’ll want to expand the root partition as well since our SD card is probably larger than the Fedora image and we’ll need to swap XFS for EXT4 since it’s not supported.
echo -e "p\nt\n1\nb\nd\n3\nn\np\n3\n`fdisk -l /dev/mmcblk0|tail -1|tr -s ' '|cut -d' ' -f2`\n\np\nw\n"|fdisk /dev/mmcblk0; sync; partprobe
Now you’ll want to format the new primary partition as vfat.
mkfs.vfat /dev/mmcblk0p1
Lastly, you’ll want to check and resize the 3rd partition as copying an image into any larger disk will make the partition table assume the size of the image. We resized it in the previous step but we’ll need to issue the resize command to finalize things.
e2fsck -f /dev/mmcblk0p3 resize2fs /dev/mmcblk0p3
Copy Raspberry Pi Boot Files
You’re not done yet, you’ll need the Raspberry pi uboot and kernel images. We’re going to create a mounted directory structure and copy these files from the Raspberry Pi repository.
mkdir /mnt/sdcard mount /dev/mmcblk0p3 /mnt/sdcard mount /dev/mmcblk0p1 /mnt/sdcard/boot
Next we’re going to download the files, copy things in place and remove any extraneous files that we don’t need.
wget https://github.com/raspberrypi/firmware/archive/master.zip
unzip master.zip firmware-master/boot/* -d /mnt/sdcard/boot/ mv /mnt/sdcard/boot/firmware-master/boot/* /mnt/sdcard/boot/ rm -rf /mnt/sdcard/boot/firmware-master unzip master.zip firmware-master/modules/*-v7+/* -d /mnt/sdcard/lib/ mv /mnt/sdcard/lib/firmware-master/modules/*-v7+/ /mnt/sdcard/lib/modules/ rm -rf /mnt/sdcard/lib/firmware-master/
Edit fstab on Mounted Media
You’ll need to modify the /etc/fstab on the mounted SD card, first get the UUID of the first vfat partition we made earlier.
blkid -s UUID /dev/mmcblk0p1
Note the UUID, it should be a very short string, for it looked like:
UUID=016B-1F2F /boot vfat defaults,noatime 0 0
Edit your /mnt/sdcard/etc/fstab with the UUID like above, substituting for your UUID.
vi /mnt/sdcard/etc/fstab
Create Boot Lines
You’ll need a few other options passed to cmdline.txt to boot things properly. The following command should take care of it. Note: even if you used /dev/sdc or similiar instead of /dev/mmcblk0 above you’ll still want to leave this /dev/mmcblk0p3 like below, as this is how the Raspberry Pi will see it’s device.
echo "dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 \ root=/dev/mmcblk0p3 rootfstype=ext4 elevator=deadline rootwait"\ >/mnt/sdcard/boot/cmdline.txt
Force Full HD over HDMI
In the next few steps we’ll be doing as much prep work while our SD card is still mounted so when we boot up for the first time there’s nothing to do post-install.
cat <<EOF>/mnt/sdcard/boot/config.txt disable_overscan=1 hdmi_force_hotplug=1 hdmi_group=1 hdmi_mode=16 EOF
Setup Raspberry Pi Update
rpi-update is a useful utility that will sync the latest kernel and firmware/boot files to your Raspberry Pi. Let’s get this in place
wget -O /mnt/sdcard/usr/bin/rpi-update https://raw.githubusercontent.com/Hexxeh/rpi-update/master/rpi-update chmod +x /mnt/sdcard/usr/bin/rpi-update
NOTE: As of Fedora 25+ rpi-update doesn’t actually update the kernel (or possibly the firmware too) as Fedora ships the newer Fedora-created ARM kernels and firmware.
Setup Wifi Firmware & Modules
Currently the wifi does not work out of the box without some newer firmware, particularly the non-free firmware. Fix it by plopping these files into /lib/firmware. In future rpi updates this should be resolved.
First, grab the firmware files
wget -O /mnt/sdcard/lib/firmware/brcm https://github.com/RPi-Distro/firmware-nonfree/raw/master/brcm80211/brcm/brcmfmac43430-sdio.bin wget -O /mnt/sdcard/lib/firmware/brcm https://github.com/RPi-Distro/firmware-nonfree/raw/master/brcm80211/brcm/brcmfmac43430-sdio.txt
Setup the module to load on boot
printf "# load wifi\nbrcmfmac\n" > /mnt/sdcard/etc/modules-load.d/brcmfmac.conf
Set Root Password
This will set your root password, you can skip this step if your first boot will be hooked up to a monitor (as it will boot to run level 5 with the “first” boot screen allowing you to set things like account passwords, timezone, etc.). If you’re running headless or with a non-graphical image you’ll want to perform this.
read -s -p "Enter password: " pass echo -n root:$pass | chpasswd -c SHA512 -R /mnt/sdcard
Note: Some people have reported issues changing the password with this method, yxogenium in the comments has suggested a workaround involving editing the password directly in /etc/shadow and enabling ssh using QEMU to chroot on the partion.
Unmount SD Card
At this point you’re ready to unmount the SD card.
umount /mnt/sdcard/boot umount /mnt/sdcard sync
Now you’re ready to insert the SD card into your Raspberry Pi and boot it up. The next set of steps will take place once you gain access to your booted up Raspberry Pi.
If you’re using a Fedora Desktop spin or Workstation image you’ll boot into the Fedora firstboot screen where you can set things up like time, users, etc. Proceed to open a root terminal to continue the setup.
First Boot: Update Fedora & RPI
You’ll want to pull down the latest updates from both Fedora and the Raspberry Pi.
dnf update rpi-update
Fix: Sound Issues
If you have issues with sound you may need to perform this:
cat <<EOF>/etc/modules-load.d/snd_bcm2835.conf #load Raspberry Pi sound module snd_bcm2835 EOF
Now load the BCM chipset driver
modprobe snd_bcm2835
Fix: Firewall
Despite generally not caring for firewalld and the over-engineered approach to “zones” I’ve found it does not work reliably on ARMv7 in either CentOS or Fedora. Let’s remove it and move back to trusty old iptables.
systemctl stop firewalld systemctl mask firewalld
Install the iptables service.
dnf install iptables-services
Now set things up.
systemctl enable iptables service iptables save
Now edit your firewall rules like a sane human in /etc/sysconfig/iptables and when you’re done you can save them.
systemctl start iptables.service
Misc Packages
One downside of the ARMv7 architecture for now is some 3rd party packages won’t be available. You can build these from source most of the time however, but it doesn’t hurt to search places like rpmfind or other 3rd party sources if you trust them.
For example you might want to use the synergy program to share a mouse/keyboard with your fresh new ARMv7 Fedora 23 desktop.
wget funcamp.net/w/synergy-1.6.2-3.fc23.armv7hl.rpm rpm -ivh synergy-1.6.2-3.fc23.armv7hl.rpm
Issue: Bluetooth not Working
The only thing that didn’t work for me was bluetooth, but I didn’t spend much time on it. It’s still missing firmware similar to the wifi above as the device is not recognized on Fedora. I’ve posted on the forums to see if others had luck as well as IRC however other people report it working on Debian Jessie.
Update: Bluetooth is working now as of 2016-07-04 via updates provided by rpi-update.
modprobe btbcm systemctl start bluetooth
Known Issue: Wifi Stops Working with 4.9 Kernel
Update 2017-05-23 – rpi-update had pushed me to a 4.9 kernel and the broadcom wifi didn’t work without pulling down the firmware again. This seems like the same file that we originally pulled down starting out, where did it go buddy?
brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43430-sdio.txt failed with error -2
Let’s grab it again and reboot.
wget -O https://github.com/RPi-Distro/firmware-nonfree/raw/master/brcm80211/brcm/brcmfmac43430-sdio.txt /usr/lib/firmware/
Everything is as it should be now, strange. This bug may be related.
wlan0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether b8:27:eb:f0:e1:2e txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Known Issue: SD Card
I’ve seen an occasional issue on my host Linux machine where I formatted the SD card that it would cache the partition information of the SD card no matter if it was ejected and rescanned and only a reboot would show the correct geometry to continue. If any of the above commands fail try rebooting the machine with the SD card inserted before proceeding and retry them.
Also, I’ve managed to get my SD card into such a state that only the SD card firmware rewriter could supposedly fix it. I was able to insert it into a digital camera instead and a quick erase did the trick, again I had to reboot the host machine reading it for the partition table information to be read correctly.
Desktop Usage
Here’s my Fedora 23 desktop running on the Raspberry pi 3. I’ve been able to stay at around 500-600MB of memory (out of 1GB) usage running Firefox, Plank, a few terminals open and other miscellaneous things with synergy connected to my primary work desktop and x11vnc. It’s light and tight but the Pi 3 seems to handle average Linux desktop tasks well with a lightweight desktop environment.
A light weight browser like Midori (come with XFCE) might be ideal because modern Firefox is a bit of a memory hog after you’ve got a few tabs open with javascript. You also might want to touch up the fonts a little bit.
Mounting it .. Anywhere
Due to it’s small size and minuscule power usage you can leave the Raspberry Pi powered on pretty much all the time. You can also fit it pretty much anywhere. In haste I’ve duct-taped my Raspberry Pi 3 Fedora 23 desktop to the back of my desk. So sketchy looking!
hobo.house 
this post would be a lot more useful if you mention how to set up WiFi and BLE.. you know, the things that are actually different about the Pi3.
LikeLike
Hi Jarwah, that’s a good suggestion. I’ve updated the guide to cover setting up Wifi as I discovered it does not work out of the box after you mentioned it here (requires some non-free firmware). I wasn’t able to get bluetooth working yet.
LikeLike
Thank you for this post. It’s exactly what I needed. I’ve been away from Debian for several years and meanwhile I know Fedora inside and out. I want to play with my new Pi, not the OS.
LikeLiked by 1 person
How the hell are common people supposed to understand this post?
LikeLike
It seems pretty straightforward to me and others who’ve followed it. If you’re not sure about the Raspberry Pi or Linux this might not be the best article to dive into right way without some prior reading.
LikeLike
It doesn’t sound like you have much Linux experience. Most new projects will have detailed instructions much like what Will posted here. I suggest you read a few books and perhaps get comfortable with Linux on the x86 platform before diving into the small ARM platforms like the pi3.
If I am wrong and you feel adventurous and are not afraid of even the smallest amount of technical detail, then just give it a shot and if you get stuck I’m sure you can post another comment here with where you got hung up and others can help you.
Good luck in either case.
LikeLike
When I run the command to reformat the SD card with three partitions, it repartitions my card so that there is only one partition (/dev/sdb3 in my case) and removes the original (/dev/sdb1). Do you have any suggestions?
LikeLike
What command are you using to format the device? You’d need to modify the fdisk one liner and substitute your top-level block device for mmcblk0, for example you’d point to
e.g.
You might try this in individual fdisk steps also, but the above has worked for me and others.
One other strange thing I’ve seen with SD cards is sometimes the partition table isn’t fully recognized (cached perhaps) and even remove and re-inserting the SD card it won’t recognize the new partitions until I’ve rebooted the host machine. kpartx or partprobe didn’t seem to help.
LikeLike
Any chance you or someone who followed these instructions… could make an image out of the finished product and post it for others to download? I know the mSDcard sizes would vary but perhaps an 8GB image (compressed with xz) could be expanded post installed if desired.
LikeLike
Now load the BCM chipset driver
modprobe snd_bcm2835
why this is no bcm 2837
LikeLike
Looks like my Pi3 has the 2835
bcm2835_gpiomem 3040 0bcm2835_wdt 3225 0
LikeLike
I recommend just installing Fedberry (http://fedberry.org/) and calling it a day. I’m not found of their pre-installed software selection and the fact that they have too many services enabled by default… but all of that is easily customizable post-setup. I’m also impressed by the fact that they package their own kernel packages rather than using one provided by the Raspberry Pi foundation.
LikeLike
That’s always an option but I was after stock Fedora. Does Fedberry use the same yum repos and package stream as regular Fedora?
I’d still prefer to just go bog standard Fedora, but thanks for pointing this option out – other folks may prefer to go this way.
LikeLike
The don’t seem to modify much. This appears to be a list:
http://download.fedberry.org/releases/23/packages/stable/RPMS/
LikeLike
The Fedberry site says it’s specifically for RasPi 2B. Can we use it to run on Raspi 3B as well?
LikeLike
The chipsets don’t seem to differ much and it may work but I’ve stuck to vanilla Fedora for mine. May be best to ask on their forums to be sure.
LikeLike
Will, when trying to set the root password, I’m getting the following error (after the second command, echo….)
“configuration error – unknown item ‘PASS_MIN_LEN’ (notify administrator)”
Any idea what should I do here? Is there a guideline for the minimum length of password required?
LikeLike
Hey Picsnapr, I didn’t run into this issue but I use fairly complex passwords. Take a look at
/etc/login.defsand thePASS_MIN_LENvalue – you should either change this to be lower or change your password length to match/exceed it.LikeLiked by 1 person
Ahh, the PASS_MIN_LEN is set to 5 in this file, and I’m certainly giving it a longer password.
However, if you notice, the error says “unknown item ‘PASS_MIN_LEN’”. So it’s apparently not even looking at this file :(
LikeLike
Any luck testing complex password characters if you’re not using them, or poke around in /etc/security/pwquality.conf or PAM?
LikeLike
I ran into this same error and was able to get around it by using:
chroot /mnt/sdcard
passwd
LikeLiked by 1 person
Thanks for the instructions!!! After going through them though, I struggled with getting my RPi 3 to boot.
In my host system where I was getting the sdcard setup, my sdcard showed up as sdc, so I replaced mmcblk0 with sdc in ALL of the instructions. But I eventually realized that in the step where you are writing the boot lines to cmdline.txt, you actually want to keep the mmcblk0 reference because that is what the Pi will see the sdcard as. Once I fixed that, it booted perfectly. It’s probably just my lack of experience with Linux that really led to the problem, but you might consider adding a note in that step to address that scenario.
Just let me know if you need more clarification. Thanks again!!!
LikeLike
Thanks Adam, that does make sense, I appreciate you posting this bit – I’ve updated the guide to include this. As my SD cards always show as /dev/mmcblk0 devices I never ran into this so I’m sure it will help others in your situation.
LikeLike
Hi All,
I have tried as same way to install Fedora 23 arm image on pi 3 model B. But during booting I pi stuck in “random nonblocking pool is initialized”. Please provide suggestion.
MN
LikeLike
I’ve seen this message pop up once or twice in console STDOUT on my Raspberry Pi 2, but I never saw it again and had no further issues.
Please verify your Fedora image (you can check what the md5sum should be) or try a different one. There are some people that had this issue on the Raspberry Pi forums which was related to a bad disk image (or a faulty device). https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=100620
Hope this helps.
LikeLike
Thanks for your great work on this.
I have used this guide for a while, and it works perfect with both RPI2B and RPI3.
Im using the LXDE which i feel is a perfect desktop for the PI.
Just did Fedora 24 on them both, no problems.
I made a script with all those instructions, copied all external files to a directory, so any time i want to install i just run the script for the image i want, and it works flawless.
Except for setting the root password, which i dont care about since i can set that when the PI boots first time, into the last install screen of fedora. (set time, user, root password and network)
I left out the part with the firewall, which make all works as usual in fedora.
I also left out the part with “synergy” since everything just works, USB mouse and keyboard.
(using Logitech)
This way its the same feeling as if i am on my main Fedora computer.
Only thing i added was a line in the “Set HDMI resolution” section.
gpu_mem = 256
LikeLike
Hi Nucleus,
Nice! Would you be willing to share your script? I’ve had great luck with Will’s steps as well, but automating the setup process a bit more would be great for my use case.
Thanks!
LikeLike
Thank you for your very helpful guide … I couldn’t have gotten up and running without this. And though there are a bunch of steps, it’s pretty straightforward (especially about the fifth time! :-) ) My problem is with WiFi … but first, some background.
I took a slightly different route than your suggestion. I wanted to do some assembly-language programming in ARMv8, so I started with Kraxel’s 64-bit Fedora-24 image and followed your instructions. *Mostly* works. Xfce4 loads and functions fine, but Cinnamon and Gnome didn’t — many of the apps wouldn’t load. Had some trouble getting it to transfer the boot process to an old USB-connected hard drive, as I had done with Raspbian on another Pi, but eventually found the file /boot/extlinux/extlinux.conf and edited it to replace mmcblk0p3 with sda3 (my “/” partition on the hard drive), and that now works fine.
And, by the way, ARMv8 is a very interesting architecture. Very different from ARMv7 (breaks the programs I’d written before) but promises to be very fast in 64-bit mode when the compilers catch up.
I’ll gradually work through the problems and get it stabilized, I think.
BUT … Can’t get it to activate WiFi. Ethernet works but not wireless. lsmod shows
Module Size Used by
brcmfmac 233528 0
cfg80211 276501 1 brcmfmac
brcmutil 9345 1 brcmfmac
but I’m not sure those are the right modules.
Any suggestions where I might look to try to fix this one?
LikeLike
Hi David, I’m glad that the guide was useful for you. Here are the kernel modules I’m using
on Fedora23 / Raspberry Pi3:
bcm2835_gpiomem 3040 0
Module Size Used by
bnep 10340 2
xt_tcpudp 2269 2
nf_conntrack_ipv4 12912 2
nf_defrag_ipv4 1428 1 nf_conntrack_ipv4
xt_conntrack 3167 2
nf_conntrack 77460 2 xt_conntrack,nf_conntrack_ipv4
iptable_filter 1409 1
dm_mirror 13669 0
dm_region_hash 8193 1 dm_mirror
dm_log 9243 2 dm_region_hash,dm_mirror
dm_mod 92159 2 dm_log,dm_mirror
bcm2835_gpiomem 3040 0
bcm2835_wdt 3225 0
uio_pdrv_genirq 3164 0
uio 8000 1 uio_pdrv_genirq
sch_fq_codel 8513 3
brcmfmac 186343 0
brcmutil 5661 1 brcmfmac
cfg80211 427855 1 brcmfmac
hci_uart 17943 0
btbcm 5929 1 hci_uart
bluetooth 326105 7 bnep,btbcm,hci_uart
rfkill 16037 6 cfg80211,bluetooth
nfsd 246430 1
ip_tables 11637 1 iptable_filter
x_tables 12507 4 ip_tables,xt_tcpudp,xt_conntrack,iptable_filter
ipv6 347530 42
In particular, these might relate to wifi:
bcm2835_wdt 3225 0
brcmfmac 186343 0
brcmutil 5661 1 brcmfmac
cfg80211 427855 1 brcmfmac
Note: I did initially have to pull wifi firmware for this to work directly from the kernel sources, but this probably isn’t needed anymore. Have you also run the rpi-update command? Hope this helps.
LikeLike
Good Morning, Will (from Montana),
Thanks for the quick reply. It looks like I have a very different set of modules loaded … and not nearly as many as you do. I did pull the WiFi firmware to update it, but I’m not sure I got the right ones. I’ll try your reference.
In looking at dmesg output, I see that the WiFi module did try to load but died. But I also see some signs that there may be some access/authorization problems through SELinux that interfere with other modules. So I’ll start by loading firmware from the source you referenced, but I think I’ll need to go back and learn something about SELinux to see if it’s causing problems.
I’m still a little concerned that I’m running an Aarch64 OS but some of the modules may be Aarch32. It’s possible to run Aarch32 modules under the 64-bit OS (though not the other way around), but I’m not sure Fedora 24 was designed to do that. It *looks* like the updates are picking the modules that are right for Aarch64, but I’m worried that some that I installed manually might have been the wrong versions. I’ll go back to check.
And yes, I did the rpi-update and the dnf update right after the install, and I’m checking every day, since this is a pretty new system.
More after some experimenting.
LikeLike
Howdy David, I believe SELinux in enforcing mode is not supported yet on the Armv7l (that I use on the Pi 3), as I recall it’s recommended to actually disable it rather than put it in permissive mode as it wasn’t built into the rpi kernel at the time I have set it up. I’m not sure that’s your problem but something to try, let us know how it works out for you.
LikeLike
Hi, Will,
Thanks for the clue about SELinux. Disabled it. Didn’t make any difference for this particular problem, but I continue to disable it because I think it may have caused other problems. Checked to make sure I had updated the brcm firmware, and the files matched those from the repository. So I couldn’t get my FC24 install working with WiFi — and I seemed to have a number of other problems with it, too.
But I was pretty sure that the main source of my problems was that I had confused FC23 and FC24 and Aarch32 versions of Fedora. And I was concerned that FC24 might be just a bit too bleeding edge for me to be starting with. So I went back to Kraxel’s repository and got the 64-bit FC23 version (arm64-rpi3-f23-mainline-20160623), did the “dd” of that to a µSD card, did the e2fsck and resize2fs you showed, and installed it on my RPi3. Booted with no problems. Did the “dnf update”, “dnf install binutils tar wget”, got rpi-config and installed it as executable, did the rpi-update, rebooted, and there it was: wlan0 was present. Had to load some other programs to get it configured (ended up with Xfce4, which makes it easy), but it works fine.
So my conclusion from all this was that 1) I made it a lot harder than it is to install 64-bit Fedora on a RPi3 and 2) I didn’t understand the different versions and their incompatibilities and ended up confusing the system as I installed it. Especially if you’re installing onto a µSD card, it’s pretty straightforward. If you’re wanting to transfer the boot to a hard drive, it gets more complicated (particularly if you want more than 3 partitions, which is what the .raw.xz image wants to give you).
So I’m happily programming away in ARMv8 assembler. Some things don’t work — Firefox starts up but quickly comes crashing down. I’m a little disappointed that the gfortran compiler seems to generate 64-bit code that’s much worse than the Aarch32 code on Raspbian, but that gives me something to explore.
But the things I need seem to work OK.
Thanks for the tutorial and for the help … the tutorial got me started, and I’m pretty sure the SELinux issue was part of the problem.
LikeLike
Will, this is a bit long, and it’s for your information rather than as a posting. In fact, I started it as notes for myself in case I need to do the install again some time. But it might be useful to you if you decide to install 64-bit FC23/24 on a RPi3. — David
Umm. Formatting is pretty awful … it was a OneNote page, but my OneNote doesn’t format PDF output very well. The info is here but the layout isn’t quite what I wanted.
===========================================================================
Over the last week, I spend a great deal of time getting Fedora set up on the Raspberry Pi 3. Here’s how I did it.
First, motivation: I wanted to have access to a full 64-bit ARMv8 OS on the RPi3 to learn about the machine architecture through its assembly language. It’s not clear that Raspbian will ever go to 64 bit — they’ve said that. And it makes sense for them not to do so. Raspbian is based on ARMv7, and ARMv8 is a very different system. ARMv8 is a more complicated architecture; they’d have to support two major OS versions on two different architectures at least for the foreseeable future; their target audience of education and hobbiests don’t need the complication of the 64-bit architecture and OS. The ARMv8 made good sense as a next platform because it supports Aarch32 (as well as Aarch64), so they could use the platform with their 32-bit OS and apps and it would work just fine. But if Broadcom releases a chip with Cortex-A32 that is Aarch32 only but with the general features of Aarch32 under ARMv8 and compatibility with ARMv7 for Raspbian, that would be a smart future direction. So I didn’t want to hold my breath hoping for 64-bit Raspbian under ARMv8.
But in looking around at options, I found that 64-bit Fedora 23 and 24 had been released through the repository at Kraxel.org. No support, no blog, no communication about it at all, really, but you could download and install it.
So I tried FC24 64-bit first. I followed directions from Will Foster ( https://hobo.house/2016/03/13/installing-fedora-linux-on-the-raspberry-pi-3/ ) on installing Fedora on RPi3, but I think his instructions were tailored for the 32-bit install he was describing, so I ended up installing some 32-bit drivers. And I’m pretty sure I mixed in some FC23 drivers as I was flailing about trying to fix things. In any case it sorta worked, but WiFi never did. Fiddled with it, got it to reboot off the hard drive, etc., but it never really ran right. Cinnamon and Gnome desktops started up the desktop GUI but wouldn’t start apps (though Xfce4 did). So I finally gave up and installed the FC23 64-bit version from Kraxel and didn’t install drivers from other places. Worked (mostly), and WiFi worked as normal. Firefox starts but crashes. I stuck with Xfce4 as desktop since it seems to work. Copied the OS to hard drive and reboot from it now. Seems to work acceptably well.
I note that diatom running hxn.cdr takes twice as long as it does with Raspbian Aarch32 and seems to have numerical errors not present when the code is run on Raspbian. I attribute that to gfortran 5 compiler quality … may go looking at other FORTRAN compilers (there are several available on Fedora). It’s still early in the lifetime of ARMv8, and I wouldn’t think much work has yet gone into optimizing FORTRAN code for ARMv8 (though I’d think it would now be worth the effort to do so with the NEON capability built into the ARMv8 specs).
And perf doesn’t give much in the way of stats. Might get better over time, since that’s so processor dependent. But that makes it hard to validate improvements or degradations in performance of any coding changes I might make to improve compute times.
But for now, I’m able to assemble and execute ARMv8 assembly code, so I can poke around the machine to learn more about it. I’ll only install Fedora on one Pi right now (a RPi3 I’ve called “Pi-6”). If Raspbian comes out with a 64-bit version, I’ll convert to that. But I can do what I wanted to now.
Notes re the install process. This installs Fedora 23, 64-bit, on a Raspberry Pi 3.
1. Download FC23 64-bit from here: https://www.kraxel.org/repos/rpi2/images/ . I used the file https://www.kraxel.org/repos/rpi2/images/arm64-rpi3-f23-mainline-20160623.raw.xz What you get is a .xz zip file. You can unzip it with gunzip on the Mac to get a .raw file. If you rename it to .img and double click on it, OSX will open the vfat partition and mount it. It can’t do that with the ext4 partition since it doesn’t know about them. Or if you copy the image (or download it to) a Linux system, you can double-click and mount both partitions in order to poke around and see what’s in them. Dismount the partitions before the next step.
2. On the Mac or RPi, install a blank µSD at least 8GB in capacity; note the device name (something like /dev/disk2 on the Mac or /dev/sdb on RPi — but be sure you get the drive identity correct before you do this step or you may blow away your system!!), and then issue the command
dd bs=1m if=arm64-rpi3-f23-mainline-20160623.raw of=/dev/disk2
3. The result of this is that you will have 3 partitions on the µSD if you mount it on a RPi system:
a. /dev/sdb1: a vfat partition that will be “/boot” when you start up the RPi
b. /dev/sdb2: “/swap”
c. /dev/sdb3: an ext4 partition that will be “/” when you start up the RPi
The third partition is only about a GB in size, corresponding to the size in the distribution version. We need to expand it.
4. (may need to be on a RPi or other Linux system– don’t think OSX has the tools). Insert the µSD into a Linux system (e.g., Raspbian — doesn’t need to be an RPi3); don’t mount the partitions. Check the FS and then expand “/” on the µSD to fill the remaining drive (I’m assuming it’s /dev/sdb, but check its identity on your system or you may blow away your system!):
a. e2fsck /dev/sdb3
b. Resize2fs /dev/sdb3
5. Insert the µSD as the boot drive on a RPi3 and boot it up. It will boot up once, do some configuring, and then reboot itself. Don’t panic! Just be patient … and it’s fairly fast. It’ll quickly come back with a login prompt.
6. Log in as “root”, password “pi”.
7. As shipped, FC23 installs SELinux in “permissive” mode. Selinux may cause problems with operation of some apps — at least it looked to be implicated in some things not running because they didn’t have permissions to access or modify files. And Will Anderson (hobo) reported that SELinux is known to interfere with operations in Fedora. So on the first boot up, edit the configuration file to disable it: vi /etc/selinux/config and change “permissive” to “disabled”. Exit vi (“:wq”) and reboot. If you don’t know how to use vi, wait ’til you’ve installed a text editor you do know (eg, nano) and do it then.
8. Set the host name of your system:
echo MyHostName > /etc/hosts
(use your preferred host name in place of “MyHostName”)
9. A number of tools you’re likely to want to use aren’t installed in the distribution version. But that’s OK: you need to update software to stabilize the system and then you can download apps and get the system you want.
10. Connect your RPi to the Internet via the Ethernet port. If you’re skilled at getting WiFi configured via command line editing, go ahead and do that and see if you can get it to connect to the network, but the easiest way will likely be to plug the RPi3 into an Ethernet connection. If you do that, Fedora will tell you it found a link and made a connection. Some times it even told me the IP address (that may have been my FC24 install), but generally it doesn’t. And “ifconfig” isn’t there yet. But if you have a way of identifying the IP assigned to it (e.g., by looking at the DHCP assignments made by your router), then you can ssh into it from another system (e.g., ssh root@192.168.1.101, password is pi ). You may find it convenient to do much of the following work from another system, since you’ll spend a good deal of time waiting for downloads and installs to complete and you can do other things if you’re working from another system.
11. You’ll use “dnf” — a YUM extension — to do the updates and installs. First, update the OS distribution. Log in as root, password pi, either on the console or over the network. Issue
dnf update
12. Then, to get some essential tools installed,
dnf install binutils nano net-tools wget
dnf groupinstall “administration tools” “system tools”
If you’re an emacs user, you can dnf install emacs-nox at this point, too.
13. Reboot. Now many of the tools you might want are available. Your favorite editor should be there (or install it with dnf if it isn’t). “ifconfig” should be there and will show that you have an Ethernet connection and that wlan0 is there and functioning (but not configured and so not connected to an access point).
14. The Rpi uses firmware in the /boot drive to actually load the OS, and that gets updated occasionally, too. The tool to do that is called “rpi-update”. We have to download and install it, first, since it’s not part of the Fedora distribution:
wget -O /usr/bin/rpi-update https://raw.githubusercontent.com/Hexxeh/rpi-update/master/rpi-update
chmod +x /usr/bin/rpi-update
15. Now we can update the RPi firmware:
rpi-update
16. Reboot
17. Time to secure your system and set up a user account. Your preferences may vary, but at least log in as root, password=pi, and type
passwd
And then enter a new password for root.
I prefer to set up a group and account for myself using the same uid/gid I use on other systems, to make nfs connections less troublesome:
groupadd -g 502 hdtodd (your group id and group name as you prefer)
useradd -b /home -g hdtodd -G root -m -s /bin/bash -u 502 hdtodd
(again, use your preferences for gid, shell, uid, and username)
Type
passwd hdtodd (your name here)
And enter a password for your personal account.
And finally, I like to work in my own account but have easy access to su when I need it, so if you prefer to work that way:
visudo
And then space down to
root
and insert a following line with
hdtodd
(using your own username, of course) and exit (:wq). If you do this, you can now log in as yourself but issue a “sudo to do a specific administrative task or “sudo -s” whenever you need to jump into root .
18. There are now tools installed that would let you configure your WiFi connection from the command line, but you’re likely to want a GUI desktop interface anyway, so it may be easier to install that first. Choose the one you want. I found that Xfce4 works pretty well — lightweight but familiar look. You can see what’s available with the command:
dnf group list
Which will show a number of different application suites as well as desktop systems. Let’s say you would like to try Xfce. Issue the command:
dnf groupinstall “xfce desktop”
After the download and install had finished, reboot. It will come up with a graphical login screen. Login as yourself. For some reason, presumably distribution configuration, the system doesn’t start NetworkManager automatically initially, and as a result the WiFi configuration manager won’t start up. So issue the commands
systemctl enable NetworkManager
systemctl start NetworkManager
Now double-click the network icon in the upper right-hand toolbar. It’ll bring up your connection options. Ethernet should already be connected; Wifi won’t be. Go to the left side of that toolbar, click “Applications” to bring down the menu, click “system”, and select “network”. Now tell it you want to add a WiFi interface, wlan0 as the device, and fill in your SSID and authentication information and tell it to connect. At this point, you should be able to manage your network connections through the icon on the right side of the top toolbar.
19. If you prefer not to have a graphical login screen but want to stick with command line for most of your work, you need to relink your default from “graphical” to “multi-user” in /lib/systemd/system. Reboot and the system will come up with a command-line login prompt. Reverse it to go back to booting to a graphical login screen. You can accomplish the same thing through systemctl.
20. Now you can load programs to tailor the system to your own needs. Use dnf list | grep to identify a specific program that you’d like to install, then dnf install to actually install it. For larger packages that involve numerous programs, you can use dnf group list to identify other packages you’d like to install. If you want C and associated development tools, for example, issue the command.
dnf groupinstall “C Development Tools and Libraries”
21. If you want to transfer the boot process from the µSD to a USB-attached hard drive, the setup process isn’t terribly complicated, but there are too many variations and possible complications to give a single list of instructions. This assumes that you have some experience doing systems work in Linux — you know how to use fdisk, dd, and friends, for example, and you know that the results can be catastrophic if you screw up. The general idea is to tell the booting firmware, which is located on the µSD (/dev/mmcblk0) to redirect its booting process to a partition on the hard drive (/dev/sda, for example). In Raspbian, you do that in /boot/cmdline.txt by changing the reference to “/dev/mmcblk0p3” to point to “/dev/sda3” (for example). In Fedora (or maybe later versions of Raspbian … not sure where the change came from), you change that reference from mmcblk0p3 to sda3 in /boot/extlinux/extlinux.conf . But before you do that, you need to
a. fdisk /dev/sda (assuming that’s your new boot device) to create the partitions you want. I made a copy of /dev/mmcblk0p1 (the µSD /boot) to /dev/sda1 ; I made /dev/sda2 be “/swap”; I made /dev/sda3 be “/”; I made /dev/sda4 be “/home”; and I made /dev/sda5 be “/var”. Most likely that’s more complicated than most people would want. There’s no real reason to have /boot on the hard drive (Rpi only boots its firmware from the µSD, and having /boot point anywhere else causes other problems when you update firmware … the real boot partition, on mmcblk0, doesn’t get updated by rpi-update, for example!); and there’s no reason to split /home and /var as separate partitions (it was an exercise for me).
b. However you choose to set up your partitions, identify which one is going to be “/swap” (you probably really do want that to be on your hard drive!) and issue a
mkswap /dev/sda2 (for example)
c. Identify which partition is going to be “/”, and
mkfs -t ext4 /dev/sda3 (for example)
d. Mount the hard drive “/” partition. For example, mkdir /mnt/hdslash and then mount the partition with mount /dev/sda3 /mnt/hdslash . Copy the system from “/” on /dev/mmcblk0p3 to what will become the new “/” on /dev/sda3. But if you try to copy the live version of “/” (as the system is in operation), you’ll get lots of error messages and likely (since I’ve never let it complete) to be unsuccessful. This copying is best done by attaching the hard drive and your new FC23 µSD to a functioning Linux system and copying. This is the tricky part. If you get this wrong, you blow away your Linux OS or your new FC23 µSD drive. (How might I know that, you ask? Experience. Typing “/dev/sda” instead of “/dev/sdb” can be a fatal mistake). You need to be careful to identify your source and destination drives (they’re likely to be /dev/sda and /dev/sdb, but which is which? Better be sure!).
e. The last change you’ll need to make is in the /etc/fstab on the hard drive to identify /boot, /swap, and /. And confirm that the file /boot/extlinux/extlinux.conf (make sure it’s the one on /dev/mmcblk0p1) points to /dev/sda3 (for example) rather than /dev/mmcblk0p3.
f. Put your µSD card in your RPi3, attach and turn on your hard drive, and power up your RPi3. With any luck (or about the 5th time you try it), you’ll see the initial firmware boot off your µSD card, then see it load the OS from the hard drive.
g. Remember: if you identify in /etc/fstab that /boot is on your hard drive, any software updates (dnf or rpi-update) will only update that version, not the one you actually boot from on /dev/mmcblk0p1. After an update, you’ll need to copy the updated boot files back to that µSD partition. And you can get version skew between /boot and the OS files in / on that µSD, so you have to be careful to keep things in sync. I think I’ve just convinced myself not to do it that way any more: I’ll just leave /boot on /dev/mmcblk0p1 and not mount the /dev/sda1 as /boot any more.
h. There are numerous variations in this process that you might choose, depending on how you intend to use your system, the capacity of your hard drive, etc. One of the simplest would be to connect your hard drive to a Linux system and power it up; assuming it’s /dev/sda (CHECK THAT TWICE!), use the dd command and then the e2fsck and resize2fs commands from the beginning of these instructions to copy the repoistory distribution image to that hard drive. Edit the mmcblk0p1 version of /boot/extlinux/extlinux.conf to refer to /dev/sda3 and edit the /dev/sda3 version of /etc/fstab to mount /dev/sda2 as /swap and /dev/sda3 as /. You’ll end up with a small swap space, and you’ll need to update and install software on that new hard drive since you won’t have copied the software from your µSD, but it’ll be a much faster process and less error prone. I haven’t done it this way (yet), but I’ll try it on my next install and append to this if it’s as straightforward as I think it will be.
LikeLike
Thanks for thoughtful level of detail here David. I’m just going to put it in comments so others can benefit too. In order to rework the instructions I’d ideally need to be on the exact same hardware as you, the armv7 (1st gen? 32bit) Pi3 that I used seems to be differ slightly, but hopefully others can take the steps from the guide + any correlational notes you’ve provided here to be in business if they run into any of the issues you’ve kindly explained and were able to troubleshoot here.
LikeLike
Good Morning, Will,
My board says “Raspberry Pi 3 Model B vr. 2”. It’s possible that your board looks different from this, but if it’s a RPi3,I think it’s probably the same Broadcom SoC underneath. My chip says “Broadcom BCM2837RIFBG”.
The rest of this is speculation based on what I’ve read over the last few weeks, from number of Internet sources — some of whom might actually know what they’re talking about. [I’m not one of them!] :-) Some of the key references are the ARMv8 architecture reference manuals, from ARM. The 5634-page document labeled DDI0487A_i_armv8_arm.pdf is particularly informative. ;-)
I think there is only one model of Pi-3 — and the board, interestingly, is labeled Pi-3B. If you ask the Raspbian OS, it says it’s ARMv7, but that’s because the OS boots into AArch32 mode. It’s actually ARMv8 architecture (Cortex-A53 CPU) underneath, and that supports both 64-bit and 32-bit modes of operation (and actually, a restricted version of the 16-bit Thumb set, too, as I understand). (ARM’s nomenclature isn’t very helpful — figuring out what’s the Architecture, what’s the CPU model, what options from the Architecture are built into that CPU, and what the OS is actually using … pretty hard to untangle.)
With Raspbian, at least, and likely with Fedora 32bit, the OS boots up into 32-bit, Aarch32 mode. If the OS is in 32-bit mode, there’s no way to get to 64-bit instructions or registers, and if the OS asks the hardware, it is told that it’s an ARMv7 computer. And the OS reports itself as being ARMv7. Attempts to assemble 64-bit source code or run 64-bit binaries fail.
On the other hand, if you boot into 64-bit Aarch64 mode, you can support 32-bit (ARMv7) instruction set and memory addressing — though I’m not sure any OS does that at this point. The reason I went to Kraxel’s site is that I hoped that what were purported to be 64-bit versions really were real 64-bit versions. The OS (uname -a) reports that it’s 64-bit, and my 64-bit source code assembles, loads, and runs. So I think it really is a 64-bit system. Raspbian Jessie definitely is not.
Now, does a 64-bit OS really buy you anything as an end user? Not yet, as far as I can tell. The Aarch64 architecture is more systematic, with roughly double the number of registers, and requires that the NEON SIMD processor be part of the system. It has a much more sophisticated security system. But compilers lag behind, and my chemistry research programs take twice as long on FC23 as they do on Raspbian running on the same RPi3 hardware. In the long term, as compilers come to use that architecture better, I think code will run faster. But right now … not so much.
But I got into this because I wanted to learn ARMv8, and for that it works well. I’ve got other systems to do real-time data collection, run Firefox, etc. I wanted to learn ARMv8, and Fedora 23 on the RPi3 works (thanks to you!).
With any luck, I’ll have been able to attach a photo of my RPI3 board for you. It does say “Raspberry Pi 3 Model B vr. 2”, so it’s possible that your board looks different from this, but if it’s a RPi3 I think it’s probably the same Broadcom SoC underneath. My chip says “Broadcom BCM2837RIFBG”.
I’m still figuring my way through all this, so I may have all this wrong. If you have the same chip, I think you have an RPI3. It masquerades as an ARMv7 until you load an OS that activates it as an ARMv8, 64-bit system. While I’m not recommending it for production, if your chip matches mine in labeling and you have a spare µSD card, download Kraxel’s 64-bit FC23 and install it and see if I got it right.
Here’s my photo …
Oh, well, that didn’t work. I filled in the identifying information above so you can check your board and Broadcom chip (I needed a magnifying glass, and I needed to look at it at the right angle). I can email you a copy of the photo if you’d like one.
David
LikeLike
Will, I see I that I had left a few things incomplete in my explanation about installing 64-bit Fedora onto a Raspberry Pi 3. This one’s important.
On item 17, security, I had written with the intention of going back to fill in, then forgot to. I wrote:
“And finally, I like to work in my own account but have easy access to su when I need it, so if you prefer to work that way:
visudo
And then space down to
root
and insert a following line with
hdtodd ”
That should be:
visudo
then space down to
## Allow root to run any commands anywhere
root ALL=(ALL) ALL
go to the next line (blank), type “i” to begin inserting, and type in
yourusername ALL=(ALL) ALL
press the ESC key to end the insert, then type “:wq” to save the file and exit.
LikeLike
So I’ve followed largely the same process from Kraxel’s blog but after getting it up and running a rpi-update borks the boot process. Seemingly around “switching to vc4drmfb from simple”, anyone encountered this reckon it’s a 32 bit driver not playing nicely with 64 bits? Is it possible to try nodemodeset in raspberry pi somehow? Pretty new to pis although not to Linux obviously. ;&)
LikeLike
Hey Philip, I’ve not hit this issue on my 2 (CentOS) or 3 (Fedora23) with latest rpi-update. If it doesn’t boot at all to even run1 you might try mounting/chroot the sdcard and adding nomodeset to a grub lines via a new initrd.
LikeLike
Philip, I haven’t seen this one either. While the F23 install on Pi is a little quirky (e.g., Firefox won’t run), I’ve updated with rpi-update and had no problems afterward.
LikeLike
Yeah, my bad it’s the dnf update that borked my Pi, sorry for the delay I’m oncall for work ATM and unfortunately $work doesn’t officially involve messing with Pis. ;^). Will attempt the various instructions today/over the weekend whilst I’m stuck indoors and update.
LikeLike
WARNING: Latest dnf update borks the boot process!
Philip, I hadn’t seen this problem, but I thought I’d see if I could provoke it. :-) Sure enough, I was successful. :-(
I just did a dnf update (didn’t get to the rpi-update), and when I rebooted in preparation for doing the rpi-update, I got the same hang you did at “fb: switching to vc4drmfb from simple”. So the system was working fine until the latest update (which I mistakenly thought might fix Firefox and IP packet out-of-sequence warnings, but instead it broke the system).
I didn’t write them all down, but for the record, some of the updated module versions are as follows (that is, this is a partial list of things you might NOT want to let the system update right now, until this gets fixed):
kernel-modules 4.6.5-200
kernel-main 4.7-1
firmware rpi3-boot-20160810-fcf0618
firefox 48.0-5 fc23
I’m not sure what to do now, with what was a working system. If I figure out how to restore to a working version, I’ll post. Otherwise, I’ll rebuild the system — after waiting for another update to fix whatever’s broken.
LikeLike
The process outlined here in this guide and the Raspberry Pi 2 guide both use the raspberry pi kernels and shouldn’t be using the system distribution Fedora kernels (though they do get installed). Currently that’s locked at 4.4.17-v7+ and provided via rpi-update.
I can’t speak for any issues if you deviate from rpi-update as I’ve not tested that. I can verify there are no issues on either the Pi 2 or 3 using the rpi-update provided kernel/firmware.
Did dnf update try and replace the current rpi kernel with the Fedora one? I’ve not experienced this if so.
LikeLike
Philip, I’ve been experimenting with this and found a trick that might get you up and running quickly. I think you’re right that it’s something to do with the vc4 driver, but I haven’t pinpointed it. But I went back and uncommented the extlinux/extlinux.conf file to ENABLE the “label kernel-4.7.0-1-main” section by removing the leading “#”, then added to the bottom of the config.txt file the two lines
gpu_mem=128
dtoverlay=vc4-kms-v3d
and rebooted. It booted OK, gave a dump of a polkit error, and then let me log in. Xwindow wouldn’t start up — haven’t investigated cause but it’s probably 4.7.0/vc4 related since it was working under 4.6.5.
So if you can insert your µSD card in another computer and mount that boot partition (/dev/mmcblk0p1), which you should be able to do since it comes up as an MSDOS partition, then you can try the edits above to see if they get you to a bootable system.
I’m hoping that the next update will fix the issues around vc4 startup … until then, I still have a working system.
LikeLike
@Dave: You and I are following the exact same path in lock step now; I also had the vc4 stuff booting, I think the overlay was the only critical component, although I also have:
avoid_warnings=2 # VPU shouldn’t smash our display setup.
due to the fact that Raspberry Pi cube apparently screws with the VC4 driver. I also have cma=256M@256M appended to my kernel cmdline, and I have no clue how this stands with regards to the gpu_mem parameter in the world of VC4
LikeLike
Policy Kit crash also evident; I have not yet established what is causing it
LikeLike
@SirSpudd, thanks for making the connection! Just to recap what I’ve got runnning:
— 64-bit FC23, AArch64, installed from kraxel’s repository using Will’s guide above
— rpi-update applied current to yesterday, reporting that it updated to 4.4.17+
— dnf update applied current to yesterday, wanting to boot kernel-4.7.0-1-main,
and while that sorta works, with the gpu-mem and dtoverlay edits to config.txt,
it still has some problems
— with that edit to /boot/extlinux/extlinux.conf to comment out the 4.7.0-1 section
I standardly boot into 4.6.5-3; that version is pretty stable, though there are
routine warnings about out-of-sequence WiFi packets and other small nuisances.
Am I right that I’m running Raspbian firmware (4.4.17+) with Fedora Linux kernel (4.6.5)? I just fiddled around ’til I got things to work. I’m surprised it worked at all, if that’s where I really got to. I haven’t edited any source code or recompiled any modules: everything I have installed came from the original install from kraxel’s repository or via dnf update or rpi-update. Dumb luck.
Also, I’m not sure I quite understand the edits you’ve made:
“@Dave: You and I are following the exact same path in lock step now; I also had the vc4 stuff booting, I think the overlay was the only critical component, ”
By “vc4 stuff booting”, do you mean the “dtoverlay=vc4_kms_v3d” edit to config.txt, which results in the system not hanging at the vc4 transition step? If so, finding that reference was just a stroke of luck for me, but it did seem to be the critical element in getting the system to boot with 4.7.0-1.
“although I also have:
avoid_warnings=2 # VPU shouldn’t smash our display setup.
due to the fact that Raspberry Pi cube apparently screws with the VC4 driver.”
I don’t have that line in config.txt; is there some warning that I haven’t noticed that this would disable? Should I only add it if I see that warning?
“I also have cma=256M@256M appended to my kernel cmdline, and I have no clue how this stands with regards to the gpu_mem parameter in the world of VC4”
Did this solve a problem, or is it something you just routinely add. I don’t know what it does, but maybe I don’t need it if I’m not triggering the problem you’ve seen.
I dropped this a couple of weeks ago as I worked on another project and then traveled a bit, and I was hoping that some of the problems would get resolved by updates in the interim. I’ll put a little more time into it now, especially working on the hard-drive boot setup, to see if I can get a smoothly-working system.
Let’s keep in touch! — David
LikeLike
@SirSpudd,
“Policy Kit crash also evident; I have not yet established what is causing it”
I only see it if I boot the 4.7.0-1 kernel, not if I boot 4.6.5-3. In both cases, I have rpi-updated to 4.4.17+, so it seems to be dependent on kernel but not on firmware.
LikeLiked by 1 person
David/Phil,
I managed to establish the cause of the PolicyKit crash; it is the same thing causing the Firefox crash and the Qt Quick runtime crash (which is why I investigated it!). The 4.7 kernel is configured with CONFIG_ARM64_VA_BITS_48; the virtual address space is bumped from 39 bits to 48, which is a problem since many developers make use of those end bits for packing data. I grabbed the source code for that exact kernel, unset that option and both PolicyKit and Qt started working perfectly. (This is a known issue in the Firefox camp, so it should also remedy that)
Video of my Aarch64 fedora image running a personal app here: https://www.youtube.com/watch?v=mRHDhYVYq7A and my blog discussing all the changes I had to make is in the comments section
LikeLike
SirSpudd, thanks very much for working through the 4.7.0 issues. I’ve gone and looked for CONFIG_ARM64_VA_BITS_48 but can’t find a kernel configuration file for 4.7.0 — just for 4.6.6. (And in 4.6.6, it’s set at 42 bits, which may explain why it never got off the ground either.)
Is this something we could fix on our own, without recompiling the kernel with the parameter set to 39, anyway? It would seem that that parameter is baked into the kernel and would need to be changed in the compilation rather than in the config file afterward. So I’d think we’d wait for Kraxel to update his repository.
If this is something we can change without recompiling, would you please post where to look for that setting and how to change it?
Thanks, again, for finding this!
LikeLike
@David:
I cover that in the aforementioned blog post:
http://chaos-reins.com/2016-08-20-qt-pi3-fedora-aarch64/
—
dnf download –source kernel-main (on Fedora image)
Copy this to a build machine
Extracted the rpm (using rpmextract.sh on Arch)
Unpacked the source
Copied the standing config from /proc/config.gz to .config
make ARCH=arm CROSS_COMPILE=/opt/arm-sirspuddarch-linux-gnueabihf/bin/arm-sirspuddarch-linux-gnueabihf- bcm2835_defconfig
make ARCH=arm64 CROSS_COMPILE=/opt/aarch64-rpi3-linux-gnueabi/bin/aarch64-rpi3-linux-gnueabi- menuconfig
Disabled CONFIG_ARM64_VA_BITS_48
Deployed, updated extlinux.conf
—
Where you clearly need to have the toolchains in question. I can’t guarantee that this will be fixed upstream anytime in the near future.
LikeLike
@SirSpudd: Well, I’ve done the kernel build 4 times now, getting more proficient and consistent each time, but the OS still won’t load at boot. (Current error is “bad Linux ARM64 Image magic”)
I’ll document for others what I’ve done in case anyone else travels down this path. Again, I’m doing the builds on an RPI-3 running Fedora 24 aarch64.
. Followed your instructions to download the source
. Since I’m doing this on a RPi-3, I didn’t need to copy to a build machine, but I did create a directory for it; did all my work (except installs) as myself, not root
. Retrieved rpmextract.sh from a repository; needed to “dnf install bsdtar” to get it to work
. Unpacked the source with rpmextract.sh and with tar
. Your guide didn’t mention it, but I COULD NOT patch the code … looks like the patches were already installed, but I’m a little uncomfortable about this part since the patches mostly affect the device tree, which I think is not being created correctly (see below)
. In ~/linux.4.7 (the source root), copied the standing config from /proc/config.gz (“cp /proc/config.gz . ; gunzip config.gz; mv config .config”)
. edited the .config file to make sure VA_48 was disabled and VA_39 was enabled; also set the local suffix to be me (“-2-hdt”) so I could identify the resulting products of the make
. “make -j 4” to make the kernel; there have been some features added, so the oldconfig phase of the make asks a series of questions; in most cases I let it select default; exceptions were ARM64 crypto instructions and a few other things like that; compiled and linked with no errors
. “make -j 4 modules”; compiled and linked with no errors
. “sudo make modules_install” and “sudo make install”; no errors in either case
I noticed that the install process doesn’t create the 4.7.0-2-hdt directory under /boot/dtbs, so I just copied the 4.7.0.1 files to that directory. That may not work, but it doesn’t seem to get far enough yet to choke on that anyway. This is one of those things I need to explore a bit more … likely some setting in the config process that isn’t right.
BUT: I also noted that the install process creates a “vmlinuz…” file in /boot, and I don’t know that the boot loader is prepared to handle that. So I copied the ~/linux.4.7/vmlinux file to be /boot/vmlinux-4.7.0-2-hdt. I tried booting with extlinux.conf pointing to first the vmlinuz file and then to the vmlinux… file; didn’t work in either case.
The error message I’m getting suggests that the vmlinux image isn’t right; some googling brought up references to “fastboot”. I’ll pursue that when I get back to this, but I’ve got some traveling coming up, so it’ll be a few weeks before I get back to this. And maybe by then there will be a new Kraxel version to work from (and maybe this VA_48 issue will have been resolved).
Again, thanks for the guidance that got me this far!
David
LikeLike
@SirSpudd: Thanks for the pointer to the Image file. Looks like that might be what the installer copied to /boot/vmlinuz-4.7.0-2-hdt (same size … didn’t do a diff). But I copied the Image file over to /boot as vmlinux-4.7.0-2-hdt; found the dts directory in the build tree and copied its contents over to /boot/dtbs/4.7.0-2-hdt/; noticed that the dtsb/4.7.0-2-hdt/ directory didn’t have the bcm2837-rpi-3-b.dtb so copied that from the 4.7.0-1 directory; edited the extlinux.conf file to make sure it pointed to the right things; and rebooted. Hung at “starting kernel …”. *May* be because I hacked in that bcm dtb file, but as I recall, it hung in earlier attempts if I didn’t have it in the version-specific dtbs directory.
I’ve got a (mostly) working FC23 µSD and a (mostly) working FC24 µSD. I just need to be careful about doing “dnf update” on them. I’m sure I’ll tinker with this a bit more as other bright ideas come to me, but I won’t be able to give it focused attention for a couple of weeks. More then.
Again, thanks for the pointers and the advice: above and beyond. :-)
David
LikeLike
Top of the morning David!
The bad ARM64 magic is due to you using the vmlinuz file; you are meant to be using the new Image format, and the associated file is under the ./arch/ARM64/boot directory.
I have updated the chaos reins blog accordingly :)
LikeLike
@ SirSpudd, Thanks for the link to chaos-reins.com … I apologize for missing it, but I hadn’t realized the connection when I saw it earlier.
For others who follow this, I’ll note that when I appended “cma=256@256M” to the command line in extlinux.conf, the system hung on boot at the “fb: switching to vc4drmfb …” step again. So I removed that parameter from the command line, and it boots just fine into 4.7.0-1.
It’s been over 10 years since I made a Fedora kernel, but I’m giving it a try. I’m compiling on the Pi-3 and it’s taking a while [ :-) ], plus I forgot the -j 4 option (though it might fail if I do try it). I think I botched it already, though. As the tail of your instructions, you have:
. Copied the standing config from /proc/config.gz to .config
. make ARCH=arm CROSS_COMPILE=/opt/arm-sirspuddarch-linux-gnueabihf/bin/arm-sirspuddarch-linux-gnueabihf- bcm2835_defconfig
. make ARCH=arm64 CROSS_COMPILE=/opt/aarch64-rpi3-linux-gnueabi/bin/aarch64-rpi3-linux-gnueabi- menuconfig
. Disabled CONFIG_ARM64_VA_BITS_48
. Deployed, updated extlinux.conf
I see that the .config file (the one copied from /proc/config.gz and unzipped) has:
# CONFIG_ARM64_VA_BITS_39 is not set
CONFIG_ARM64_VA_BITS_48=y
CONFIG_ARM64_VA_BITS=48
So won’t that compile everything with 48-bit VA? Isn’t that a compilation setting rather than a boot-time parameter? Should I instead edit the .config, before doing the “make ARCH= …” to say:
# CONFIG_ARM64_VA_BITS_48 is not set
CONFIG_ARM64_VA_BITS_39=y
CONFIG_ARM64_VA_BITS=39
That is, shouldn’t that bullet in your instruction come before the make commands? And am I right that I want VA_BITS=39, or should it be some other value?
And since I’m compiling on the Pi-3, I presume that I don’t need CROSS_COMPILE set; the commands would be
make ARCH=arm [do I really want to do this if I’m generating Aarch64 version?]
make ARCH=arm64
Again, thanks for working through this and sharing how you got it going. I’m running 4.7.0-1 now, though it’s still failing in some things (e.g., polkit-related) because of the VA_BITS issue you found.
David
PS: recompiling with the parameter changes that I think I needed to make; remembered “make -j 4” this time and it’s moving much faster. ARCH=arm fails to compile. Not sure I needed it, but that’s not a good sign.
PPS: took 48 min to compile arm64 kernel. No errors, but it wouldn’t load at boot time. More research (and experience) needed at my end.
LikeLike
Hey Will.. just FYI.. at the how to, where you say: Fedora XFCE or similar F23 image for ARMv7, you actually link to the KDE image :D
LikeLike
Oh, that’s ok – that will work too haha. Thanks, I updated it.
LikeLike
Will, in response to your comments (in quotes)
“The process outlined here in this guide and the Raspberry Pi 2 guide both use the raspberry pi kernels and shouldn’t be using the system distribution Fedora kernels (though they do get installed). Currently that’s locked at 4.4.17-v7+ and provided via rpi-update.”
I just did the “dnf update”, and it may have pulled code from kraxel. So that’s probably what caused the problem. Again, I’m running FC23-aarch64 on a Pi 3, downloaded from the kraxel.org repository.
I *was* able to take a step back and restore a working system by mounting my boot volume on my Mac and editing the extlinux/extlinux.conf file to comment out the section “label kernel-4.7.0-1-main” so that it boots using the section “label kernel-4.6.5-3-main” (that is, regress one update step).
I then did the rpi-update and it updated to 4.4.17-v7+, and that rebooted fine. So the problem really is in the “dnf update” that pulled in kernel 4.7.0 from somewhere.
“I can’t speak for any issues if you deviate from rpi-update as I’ve not tested that. I can verify there are no issues on either the Pi 2 or 3 using the rpi-update provided kernel/firmware.”
Yes, that’s what I see, too. I’m convinced that my problem was a result of my deviations from your guide.
“Did dnf update try and replace the current rpi kernel with the Fedora one? I’ve not experienced this if so.”
Yes, I believe that’s what happened. I’m not sure how I could go back to document that, but that seems to be the problem (I do know that some of the dnf update-d modules were from kraxel, as I monitored the downloads). And since I do the “dnf update” fairly regularly (hoping that some of the nuisance problems will be fixed), I’m not sure how I could have avoided that.
But I confess that I’m still confused about what updates come from where and what the incompatibilities might be. My failed update (and Philip’s install problem) probably result from that.
LikeLike
Dave: Thanks for the heads up about rolling back via editing ext-linux; discovering this kind of thing by yourself is no fun. Also thanks for the heads up about rpi-update; I knew /opt/vc was AWOL but did not know I was meant to remedy that. The Pi is moving to using a vanilla kernel, which is probably why this dude is starting to respect the upstream aarch64 bit kernels. I could also not fix the move to vc4drmfb, and suspect config.txt might play into that since I had to adjust it to get VC4 to work on Arch Linux.
All in all, very cool. Only disadvantage of aarch64 Fedora 24 over armv7 Arch is the sd-card IO, which is incredibly painful
LikeLike
Donald, yes, I’m pleased with having gotten the 64-bit OS working on the Pi-3, even if it’s a little flaky. I wanted to do some AArch64 assembly programming, and at least that part has worked quite well for me on the FC23 AArch64 OS. Most of the time, I find performance on the µSD to be surprisingly good; occasionally it’s surprisingly bad (network performance over WiFi is sporadic, and it’s still reporting out-of-sequence packets far too often); some times it just fails entirely (odd Firefox crashes and SSH sessions terminated abruptly).
I haven’t yet tried, but will shortly, to have the Pi 3 boot (not reboot) off a USB-attached hard drive on which I had previously installed FC23 AArch64. If I understand correctly, having already set the Pi 3 to boot off USB if it doesn’t find a µSD, it should boot up from USB drive automatically if it sees the bootcode.bin file under /boot. I’ll post something once I’ve tried it. I did set it up on my other Pi-3, running Raspbian, and it worked. I’m hoping it’ll work for Fedora 64-bit, too, and that should make for an interesting system.
LikeLike
Well, OK, I tried to get the Pi-3 to boot FC24-Aarch64 directly off the USB drive, but I’ve been unsuccessful. I installed 4.6.2-4 from Kraxel’s repository (arm64, 20160623 version) onto a new USB drive with Will’s instructions above, rebooted to the USB drive from SD, did the dnf update and the rpi-update (which ultimately get to kernel 4.4.18-v7+ and Linux 4.7.0). Used Raspbian to set program_usb_boot_mode=1 and program_usb_boot_timeout=1 in the OTP. Booted FC24 and did the BRANCH=next rpi-update (and tried both with and without that new code). I tried various combinations of firmware and updated systems. No luck getting the Pi-3 to boot directly off the USB drive without a µSD to start the process. I’m not surprised, but I was hopeful. I’m able to reboot from µSD to FC24-Aarch64 on the USB drive (a Lexar 64GB, which was noted on the MSB site (https://www.raspberrypi.org/blog/pi-3-booting-part-i-usb-mass-storage-boot/) among comments as having worked for Raspbian. Most apps work fine; WiFi works; Firefox dies trying to start up.
Along the way, I did get 4.7.0 to boot (for a while … did some rpi-update that forced me to go back to 4.6.2-4). But I think I have too many versions of firmware/kernel/OS conflated to have a stable system.
I’ll go work on some other projects for a while before coming back to see if new releases help any.
Oh, I used both the 17KB version of bootcode.bin and the 50KB version (the one from github mentioned in the web site comments above). Neither one worked for direct USB booting.
LikeLike
@Will Foster, @SirSpudd,
Travels delayed, raining in Yellowstone, so I went back to work on getting 64-bit FC24 working on RPI-3B. Did a successful compile/install of 4.7.3 with VA_BITS=39 (thanks, SirSpudd!), and while it eliminated the outright crashes of polkit associated with memory addressing, a number of things that rely upon polkit for authentication fail, with polkit saying something about:
PolicyKit daemon disconnected from the bus.
We are no longer a registered authentication agent.
I downloaded polkit source and recompiled under the 4.7.3 VA_BITS=39 environment but continued to get the same errors. Any suggestions about how to resolve this? SirSpudd, are you running xfce? If you type “systemctl default”, do you get a result?
I’m writing up the procedure for installing 64-bit FC24 on the RPI-3, shamelessly borrowing from (and acknowledging) Will’s work here. I’ve also started a page on compiling FC24 on the RPI-3, again borrowing from (and acknowledging) SirSpudd’s work (above). Here’s the warning (below) at the beginning of my page. Is this what you’re seeing, @SirSpudd?
I’ve got a little more editing to do and then I’ll pass the URL along in a subsequent post. I’d rather not have started those pages on my own blog site (on google), but I didn’t know where else to go to be able to format in a reasonable way. Open to suggestions from either of you. –David
===========================================================================
64-bit Fedora on Raspberry Pi-3 — ongoing issues
As of 2016.09.12, there are problems with updates to the Fedora 24 Raspberry Pi distribution after the initial install that limit functionality.
1. The 20160623 distribution (step 1 below) installs kernel 4.6.2-4-main. That system functions reasonably well: WiFi works, xfce GUI desktop functions, etc. Firefox crashes before it can bring up a window. There may be problems in other applications, but that’s the one I’ve found.
2. If you do a “dnf update”, as of 2016.09.12, you’ll update to kernel 4.7.3-1-main. That update breaks a number of things (as do previous updates I had tried earlier). The most critical issue is that it breaks “polkit”, if you have that installed. And you have that installed if you use “systemctl”, “xfce desktop”, or “administration tools” (those are the ones I know about). SirSpudd (reference above) found that the problem was caused when the upgraded kernel was compiled with VA_BITS=48 instead of VA_BITS=39. Indeed, if you recompile and install the kernel with the corrected setting for VA_BITS, “polkit” no longer crashes with memory addressing problems. But other components continue to fail, and they point to “polkit”. “systemctl default” fails; xfce4 simply hangs the system (and trying to boot with the graphical interface set as the default hangs without giving a prompt). Those are the ones I know about. If you get to that point, put your boot µSD into another system (Mac, Linux) and edit the /boot/extlinux/extlinux.conf file to comment out (#’s in column 1) the 4.7.3-1-main boot entry — let it default back to 4.6.2-4-main — and reinstall that µSD in your RPi-3B to reboot it. Your system will boot up again in FC24 using kernel 4.6.2-4-main. As of this date, that’s as far as I’ve gotten in trying to set up a stable, upgradeable FC24 RPI system.
LikeLike
I should have said that I did *not* need to do the WiFi install in Will’s instructions — the WiFi worked “out of the box” after the install, dnf update, and rpi-update. But I did have to manually “systemctl enable NetManager” and then use the GUI interface to configure the wlan0 interface.
LikeLike
@Will, @SirSpudd: here’s my first attempt at documenting the installation process and problems for getting 64-bit Fedora 24 running on the Raspberry Pi-3B.
http://fc24-rpi.blogspot.com/
Details for building the kernel on the RPi3 to follow shortly. Corrections, comments, suggestion welcome: I don’t blog, so I’m not sure this will come out right. –David
LikeLike
@hdtodd – Awesome, thank you for sharing this!
LikeLike
@SirSpudd, @Will:
“dnf update” now updates to 4.7.3-2, which was built with VA_BITS=39. Fixes the polkit memory addressing problem, but other components (systemctl, xfce) continue to fail in ways that point to security access being denied by polkit. I’ll re-install 4.6.2, verify that xfce works, and then update on a clean machine to see if xfce continues to work under a clean update to 4.7.3-2.
LikeLike
@SirSpudd, @Will:
Added a page on building the 64-bit Fedora kernel on the Raspberry Pi-3B:
http://fc24-rpi.blogspot.com/
Still pretty rough; comments & suggestions welcome.
–David
LikeLike
@Will, @SirSpudd,
I’ve run the experiment and found that updates to the 20160623 distribution repository images at kraxel.org break the GUI desktop (at least): I installed clean FC24 system from the repository to get kernel 4.6.2; installed xfce and it works; didn’t tinker with any configuration parameters; did “dnf update” to get kernel 4.7.4; xfce now broken. polkit memory-addressing crashes were corrected by the latest updates, but xfce issues not fixed. After the update (which also updates firmware), WiFi is a bit flaky — drops connections, packets out of sequence reported as warnings.
I’ve revised my blogs at http://fc24-rpi.blogspot.com/ to reflect this experience: advise editing extlinux.conf to give a boot menu; don’t “dnf update” unless you really need to for some reason (4.6.2 is most stable); if you do update to install a new kernel, keep 4.6.2 as one of the stable kernels in case you want to use a GUI desktop.
I’ll report back and update the blog when we get an update that fixes some of these issues. Meanwhile, it *is* possible to get a 64-bit Fedora 24 running nicely on the Raspberry Pi, even if not everything works.
–David
LikeLike
Hey David, I’ve put a link to your blog prominently in the beginning of the guide. Thanks for all your work around getting the latest upstream Fedora kernels working – I know it’s a work-in-progress but it’s improving rapidly.
LikeLike
Thanks, Will. I noticed that that thread was getting a bit long, mostly because I was getting too verbose. ;-)
I sent Kraxel my results a couple of days ago and had hoped that the next update (4.7.4) would fix it, but it didn’t. He’d been very responsive to earlier notes but not to this last one — I suspect either it’s not easy to track this down, and I’d expect that efforts might be better directed at a next major release, hoping that’ll fix it.
Anyway, we’re on the road for a week starting Friday. I’ll tear down equipment Thursday, but until then I’ll continue monitoring and will update that blog site if the problems are fixed by a subsequent release.
LikeLike
I love this guide, and I get a kick out of lines like this:
echo -e “p\nt\n1\nb\nd\n3\nn\np\n3\n`fdisk -l /dev/mmcblk0|tail -1|tr -s ‘ ‘|cut -d’ ‘ -f2`\n\np\nw\n”|fdisk /dev/mmcblk0; sync; partprobe
Thanks for putting this post together. I’m going to be getting my new RPi3 soon and I intend to set it up using exclusively Ansible playbooks, so that in theory, the work of setting up one of these could be massively reduced. I’ll keep you posted.
LikeLike
I had issues chrooting and changing the password, I eventually managed to edit the password directly in /etc/shadow and to enable ssh using QEMU to chroot on the partion. I wrote a short markdown note to document the process: https://gist.github.com/Yxogenium/801dbd4cf5d9e5151be1e7e60e5f7653.
Thank you for the thorough primer!
LikeLike
Hey yxogenium, thanks for posting your QEMU workaround. You’re the first person I’ve seen that has hit this but I’ve included your Github gist in the guide.
LikeLike