How I got a XenHVM kernel and xen-tools working in FreeNAS

[freeqeed]

Did you try reinstalling ports? > portsnap fetch extract

Following your post I tried it but without joy. I then created a new FreeBSD 9.2 machine from scratch and run "portsnap fetch extract" before installing the build tools. Nonetheless I encountered exactly the same error. Then I build FreeNAS by issuing the "make NO_CHECKSUM=yes release" command to circumvent the checksum error but then got the following error:

Code:

extracting for ataidle-2.7.2 unrecognised archive format

I'm starting to wonder whether FreeBSD introduced a problem with "ataidle" between now and then and this problem is not really related to the Xen settings.
Below is the link to my GitHub repository where you can see what changes I applied https://github.com/topler/freenas-9.2-xen/compare

I presume my problem is related to my build environment rather to the Xen Server related changes since a normal build without those Xen Server related changes gives the same error.

If someone fancy, you could also run the build by using this repository on https://github.com/topler/freenas-9.2-xen. If you build environment doesn't have this glitch I have, I believe it should go through just fine.

Code:

git clone -b xenserver https://github.com/topler/freenas-9.2-xen.git

 

[freeqeed]

I have finally been able to compile successfully after I used an Amazon EC2 instance provided by FreeBSD. :)

Please see the below GitHub repository for a comprehensive guide.
https://github.com/topler/freenas-9.2-xen

I hope this will be of help.

 

[coldice]

I have finally been able to compile successfully after I used an Amazon EC2 instance provided by FreeBSD. :)

Please see the below GitHub repository for a comprehensive guide.
https://github.com/topler/freenas-9.2-xen

I hope this will be of help.

Thanks, but my build failed:

Code:

+ export UNAME_m=amd64
+ export UNAME_p=amd64
+ export UNAME_r=9.2-RELEASE-p3
+ uname -v
+ uname -p
+ uname -r
+ sed -e s/amd64/amd64/ -e s/9.2-RELEASE-p3/9.2-RELEASE-p3/g
+ export 'UNAME_v=FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013    root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC '
+ awk '/\#define.*__FreeBSD_version/ { print $3 }' /home/freenas-9.2-xen/FreeBSD/src/sys/sys/param.h
+ export OSVERSION=902502
+ export PATH=:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
+ chroot /home/freenas-9.2-xen/os-base/amd64/_.w /bin/sh -exc 'cd /usr/ports/packages/All;pkg_add -F xen-tools-4.1.3_3.tbz'
+ cd /usr/ports/packages/All
+ pkg_add -F xen-tools-4.1.3_3.tbz
pkg_add: could not find package python27-2.7.5_3 !
pkg_add: could not find package python2-2_1 !
pkg_add: could not find package python-2.7_1,2 !

I removed xen-tools & xe-guest-utilities from os-base, tried to build again...
Then it failed again:

Code:

=> bsnmp-ucd-0.4.0.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch http://bsnmp-ucd.googlecode.com/files/bsnmp-ucd-0.4.0.tar.gz
bsnmp-ucd-0.4.0.tar.gz                                  29 kB    0  Bps
fetch: transfer timed out
fetch: bsnmp-ucd-0.4.0.tar.gz appears to be truncated: 29696/30011 bytes
=> Attempting to fetch ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/bsnmp-ucd-0.4.0.tar.gz
fetch: ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/bsnmp-ucd-0.4.0.tar.gz: File unavailable (e.g., file not found, no access)
=> Couldn't fetch it - please try to retrieve this
=> port manually into /usr/ports/distfiles/ and try again.

How can I resolve these two errors?


EDIT:
Clean install
Using following commands:

Code:

$ portsnap update
$ portsnap extract
$ portsnap fetch extract
$ pkg_add -r git
$ pkg_add -r cdrtools
$ pkg_add -r pxz
$ pkg_add -r screen

Not a single problem afterwards.

EDIT2:
Still getting:

Code:

 
===>  Cleaning for bsnmp-ucd-0.4.0
===>  License BSD accepted by the user
===> Fetching all distfiles required by bsnmp-ucd-0.4.0 for building
===>  Extracting for bsnmp-ucd-0.4.0
=> SHA256 Checksum mismatch for bsnmp-ucd-0.4.0.tar.gz.
===>  Refetch for 1 more times files: bsnmp-ucd-0.4.0.tar.gz
===>  License BSD accepted by the user
=> bsnmp-ucd-0.4.0.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch http://bsnmp-ucd.googlecode.com/files/bsnmp-ucd-0.4.0.tar.gz
bsnmp-ucd-0.4.0.tar.gz                                  29 kB    0  Bps
fetch: transfer timed out
fetch: bsnmp-ucd-0.4.0.tar.gz appears to be truncated: 29696/30011 bytes
=> Attempting to fetch ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/bsnmp-ucd-0.4.0.tar.gz
fetch: ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/bsnmp-ucd-0.4.0.tar.gz: File unavailable (e.g., file not found, no access)
=> Couldn't fetch it - please try to retrieve this
=> port manually into /usr/ports/distfiles/ and try again.
*** [do-fetch] Error code 1
 
Stop in /usr/ports_dir/net-mgmt/bsnmp-ucd.
*** [checksum] Error code 1
 
Stop in /usr/ports_dir/net-mgmt/bsnmp-ucd.

HOW TO FIX?

 

[freeqeed]

I had some wired issues related to some ports for which I never got to the bottom of it. Only after using the highlighted Amazon EC2 images, I could successfully deploy by following the steps mentioned.
Are you using those Amazon EC2 instances?

Go to -> http://www.daemonology.net/freebsd-on-ec2/ -> then use one of the machines mentioned in row "FreeBSD 9.2-RELEASE For 64-bit m1, m2, c1, and t1 instances"

As of the error you are getting, the port at location ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/bsnmp-ucd-0.4.0.tar.gz does not exist. When you look up ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/ you will that only older versions of that port are located there. I cannot tell whether this particular version of the port was in existence at the time of my compilation or whether it was removed between now and then. Methinks that you might use a different FreeBSD 9.2 instance and thus run into this issues.

 

[Xlot]

Has anyone managed to build Freenas with 9.2.1.x with xentools support for Xenserver?

I can build with just Xen HVM drivers and it works successfully (I can run up a VM using the pvhvm drivers and with the xvda and xbd block devices), but if I try to build with xen-tools, I get nowhere quickly..

So far I've had to add the following to ./freenas/nanobsd/os-base

add_port devel/gmake
add_port sysutils/xen-tools
add_port sysutils/xe-guest-utilities
add_port lang/perl5.16
add_port devel/p5-Locale-gettext

(gmake, perl and p5-Locale are all needed to meet dependencies for the xen packages).

 

[Xlot]

Has anyone managed to build Freenas with 9.2.1.x with xentools support for Xenserver?

The error I'm getting is this, when trying to build the port "sysutils/xen-tools"

Code:

cc -O2 -pipe -fno-strict-aliasing -O2 -fomit-frame-pointer -m64 -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wno-unused-value -Wdeclaration-after-statement  -D__XEN_TOOLS__ -MMD -MF .xen-detect.d  -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE  -Werror -I /usr/workdir/usr/ports_dir/sysutils/xen-tools/work/xen-4.1.3/tools/misc/../../tools/python/xen/lowlevel/xc -I /usr/workdir/usr/ports_dir/sysutils/xen-tools/work/xen-4.1.3/tools/misc/../../tools/libxc -I /usr/workdir/usr/ports_dir/sysutils/xen-tools/work/xen-4.1.3/tools/misc/../../tools/include  -Wl,--no-as-needed amd64 xen-detect.c  -o xen-detect
getconf: no such configuration parameter `LFS_CFLAGS'
getconf: no such configuration parameter `LFS_LDFLAGS'
cc: amd64: No such file or directory

The error is due to the "amd64" argument in in the cc command string (line 1 ) above - however I don't know enough about the freenas build scripts to figure out how to fix it.

The original post refers to a similar issue, but the freenas build scripts have changed since it was written and my amateur hacking has failed to find/fix the issue.

When I tried the build with only these changes, I got an error during the build of xen-tools. I 'fixed' the problem by hacking nanobsd/common to remove any mention of TARGET_ARCH from the do_add_port function. Specifically the command on Line 236 is now:

Can anyone help?

 

[Ringerl]

I've compiled https://github.com/freenas/freenas/releases/tag/9.3-STABLE-201501241715 with PVHVM support (without xen-tools!) as detailed by Xlot and godbolt - results available (without guarantees)

https://j.mp/freenasPVHVM

Haven't tested it yet - so please use with caution!

 

[cyberjock]

Honestly, this somewhat scares me.. not sure how the update system will handle this (going from this custom build to some future build based off the standard builds).

It would make a fun experiment for those willing to try, but I'm very hesitant to try to use it in any capacity at all because updates for FreeNAS are so frequent and updates would no doubt fubar your custom build.

 

[Xlot]

Honestly, this somewhat scares me.. not sure how the update system will handle this (going from this custom build to some future build based off the standard builds).

Until Freenas moves to FreeBSD 10, the standard downloadable build updates will not work for Xen environments. However, the update process and functionality in the FreeNAS GUI works just fine using an update file built using the process to add Xen support (i.e. has a correctly xen-supporting GUI_Upgrade.txz file with its own matching sha256 hash). If you're using PVHVM disk/network drivers for near-native IO performance under Xen, then using the standard upgrade downloads will break disk and network access and its a headache to recover from.

Obviously using FreeNAS under Xen is entirely unsupported - but for those of us running in Xen-based virtualised environments, its not like we have many options :) For me, the race is on between Freenas and nas4free - the first one to move to FreeBSD 10 with its kernel-support for Xen will pretty much corner the market, which includes Amazon EC2 (its Xen based).

Cloud-based ZFS storage (with snapshots, backups, etc.) is a very nice tool for supporting cloud-based infrastructure. Between virtualised FreeNAS and pfSense, my infrastructure needs are taken care of, and I use a mix of Linux (Centos, Fedora, Ubuntu server) and Windows (xp/7/8.1) all of which use varying mixes of nfs, cifs and iscsi from my freenas instance. Yes, all that functionality can be provided without using Freenas, but having a GUI to manage it all simplifies things greatly.

The most important point however, is that this is an entirely unsupported combination of FreeNAS and xen - I wouldn't run mission-critical stuff on it period. Its a "use at your own risk" configuration.

 

[Phlogi]

I agree with the last posts, but still think it's a good setup if you know what you are doing. I also did my own build with xen support and then went through the troubles of installing it on xenserver. In case anyone is looking for a click and run version, you can download an Upgrade File or a fresh and clean .xva (VM Import) appliance if you want - just let me know.

If there is further interest, I'm able to provide regular updates.

 

[Ringerl]

Phlogi, have you succeeded building with xen-tools?

 

[Phlogi]

Well I did not change the build itself at all to include additional packages yet, if that's what you mean?

However I got a xen enabled freeNAS build working. On that one you can "simply" pkg install xe-guest-utilities if you need that. Or do you want xen-tools to use freeNAS as a dom0 - I doubt that :)

 

[Ringerl]

Hi Phlogi! pkg install is a non starter on 9.3:

Code:

root@freenas] ~# pkg upgrade
Updating local repository catalogue...
pkg: file:///usr/ports/packages/meta.txz: No such file or directory
pkg: repository local has no meta file, using default settings
pkg: file:///usr/ports/packages/digests.txz: No such file or directory
pkg: Unable to update repository local

 

[Phlogi]

Hi Ringerl

I could post the steps to get pkg working, but in my case it destroyed some other packages while installing xe-guest-utilities. However you can just reboot and the changes are gone anyway. If you feel adventurous, have look at the file: /usr/local/etc/pkg/repos/FreeBSD.conf

Cheers

Hi Phlogi! pkg install is a non starter on 9.3:

Code:

root@freenas] ~# pkg upgrade
Updating local repository catalogue...
pkg: file:///usr/ports/packages/meta.txz: No such file or directory
pkg: repository local has no meta file, using default settings
pkg: file:///usr/ports/packages/digests.txz: No such file or directory
pkg: Unable to update repository local

 

[Phlogi]

I found a way:

1. Go into any jail where pkg is installed / or use the activated pkg on root carefully
2. pkg fetch xe-guest-utilities xen-guest-tools
3. copy over the txz files to some xen-packages folder that persists after reboot
4. pkg add -f xe-guest-utilities-6.2.0_1.txz xen-guest-tools-4.2.2_1.txz
5. start the service: echo "xenguest_enable=YES" >> /etc/rc.conf && /usr/local/etc/rc.d/xenguest start

The last two steps could be put into a task that is run on every boot.

Cheers

 

[Phlogi]

Just to let you know, I'm running my own FreeNAS xen build for some time now without any issues. I also added the guest utilities to the packages, so the only thing I need to do as a boot task is:

Code:

/usr/local/etc/rc.d/xenguest onestart

Also I did a recent build and just used the GUI update file, which worked perfectly.

Here are some more details:

Code:

root@storage ~ # dmesg | grep xen
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 30 at device 3.0 on pci0
xenstore0: <XenStore> on xenpci0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xenbusb_back0: <Xen Backend Devices> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
xbd0: 6144MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
Trying to mount root from zfs:freenas-boot/ROOT/FreeNAS-FreeNAS-9.3-STABLE-xen-201512041013 []...
root@storage ~ # uname -a
FreeBSD storage 9.3-RELEASE-p28 FreeBSD 9.3-RELEASE-p28 #1 r288272+f229c79: Fri Dec  4 11:15:10 CET 2015  root@freeBSD9.3:/mnt/secondDisk/freenas/objs/os-base/amd64/mnt/secondDisk/freenas/FreeBSD/src/sys/FREENAS.amd64  amd64
root@storage ~ # xen-detect
Running in HVM context on Xen v4.6.

PS: This is on xenserver Dundee Beta 1, which also works flawlessy and much faster (boot time, i/o performance) than 6.5.

 

[Phlogi]

The latest stable 9.10 version has "native" xen support, I just found this in /build/profiles/freenas9/kernel/FREENAS.amd64:

# Xen HVM Guest Optimizations
# NOTE: XENHVM depends on xenpci. They must be added or removed together.
options XENHVM # Xen HVM kernel infrastructure
device xenpci # Xen HVM Hypervisor services driver

This means we don't need our own builds anymore to run it in on xenserver. Anyone tested this already?

 

[Phlogi]

Great news:

• The FreeNAS-9.10-STABLE-201603252134.iso installs on xenserver dundee (VM type other) without any issue.
• Boots without any tweaks at all!
• xe-tools can be added easily:
  
  Code:

  [root@freenas] ~# pkg add -M -f xen-guest-tools-4.2.2_1.txz
  Installing xen-guest-tools-4.2.2_1...
  Extracting xen-guest-tools-4.2.2_1: 100%
  [root@freenas] ~# pkg add -M -f xe-guest-utilities-6.2.0_1.txz
  Installing xe-guest-utilities-6.2.0_1...
  Extracting xe-guest-utilities-6.2.0_1: 100%
  [root@freenas] ~# /usr/local/etc/rc.d/xenguest onestart
  Starting xenguest. 

;)

 

[cyberjock]

Great news:

• The FreeNAS-9.10-STABLE-201603252134.iso installs on xenserver dundee (VM type other) without any issue.
• Boots without any tweaks at all!
• xe-tools can be added easily:
  
  Code:

  [root@freenas] ~# pkg add -M -f xen-guest-tools-4.2.2_1.txz
  Installing xen-guest-tools-4.2.2_1...
  Extracting xen-guest-tools-4.2.2_1: 100%
  [root@freenas] ~# pkg add -M -f xe-guest-utilities-6.2.0_1.txz
  Installing xe-guest-utilities-6.2.0_1...
  Extracting xe-guest-utilities-6.2.0_1: 100%
  [root@freenas] ~# /usr/local/etc/rc.d/xenguest onestart
  Starting xenguest. 

;)

Just a word of warning that you are in unsupported territory by trying to install packages yourself. We've warned God and Country about not doing package installs this way, so do this at your own peril.

 

[jgreco]

We've warned God

I don't recall that... ;-)