Automating Dell iDRAC Administration with Racadm

Tuxdell_logoManaging more than a handful of Dell servers becomes cumbersome using the iDRAC web interface.  This is especially painful with several hundred servers or more.  Luckily Dell provides CLI tools you can use with XML templates to automate provisioning and administration for you.

Why use XML Templates?
We tie in a bunch of other Open Source tools like Foreman, Ansible and Puppet to provide a seamless lifecycle and management workflow.  Dell has proprietary software to manage iDRACs but using the innate XML template functionality within the iDRAC better fits our provisioning workflow with CLI tools and orchestration.  Lastly, being able to revision control things into an SCM like git is important for us.

Racadm and Dell Automation
You can also just do a lot using ssh and racadm commands and we’ll cover the most common examples that are useful to us.  You should take a look at other racadm guides for a more exhaustive example of racadm administration.

Requirements:
You’ll need a few basic things before you proceed:

  • An EL5, EL6, EL7-based Linux host or VM
  • Access to the internet and access to the iDRAC interface you want to manage
  • A NFS server export (for storing/serving XML templates)

Download the iDRAC CLI Repo
Repo instructions are taken from Dells site here

wget -q -O - http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash

Piping anything into bash is generally a bad idea, but this a vetted method.
This will setup a few extra yum-specific plugins needed, and a few dependencies.

Downloading GPG key: http://linux.dell.com/repo/hardware/latest/RPM-GPG-KEY-dell
    Importing key into RPM.
Downloading GPG key: http://linux.dell.com/repo/hardware/latest/RPM-GPG-KEY-libsmbios
    Importing key into RPM.
Write repository configuration
Downloading repository RPM
Installing repository rpm: http://linux.dell.com/repo/hardware/latest/platform_independent/rh60_64/prereq/dell-omsa-repository-2-5.noarch.rpm
Installing yum plugins for system id
Loaded plugins: product-id, security, subscription-manager, versionlock

Dependencies installed:

================================================================================
 Package                Arch      Version              Repository          Size
================================================================================
Installing:
 yum-dellsysid          x86_64    2.2.27-4.4.1.el7     dell-omsa-indep     16 k
Installing for dependencies:
 python-smbios          x86_64    2.2.27-4.4.1.el7     dell-omsa-indep     69 k
 smbios-utils-python    x86_64    2.2.27-4.4.1.el7     dell-omsa-indep     63 k

Transaction Summary
================================================================================
Install  1 Package (+2 Dependent packages)

Install iDRAC CLI tools
This will pull in smbios-utils-bin, srvadmin-argtable2, srvadmin-omilcore

yum install srvadmin-idracadm openssl-devel

How to Use the iDRAC CLI
There are a handful of very useful ways to use the iDRAC CLI, almost any action you’d normally have to do in a browser or remote OOB Java viewer you can do with the tool.  It can also parse and read XML templates to configure the iDRAC.

  • 10.1.8.30:/srv/configs is our local NFS (RW) share in these examples.

Managing iDRAC XML Templates
iDRAC automation centers around manipulation of XML templates which comprise the configurations and settings of both the iDRAC and the attached peripherals (PERC cards, etc).

Changes will revolve around pulling, editing, pushing templates.
It’s also useful to revision control these if you like via git, but entirely optional.

Pulling iDRAC Template
This is useful for safekeeping before editing and also to modify and use for pushing a similar configuration across many many systems.

idracadm -r 10.1.8.59 -u root -p PASSWORD get -t xml -f host.xml -l 10.1.8.30:/srv/configs

Now you should have an XML file called host.xml or whatever you called it sitting in the /srv/configs NFS share.  It’s a good idea to backup all of your configs this way before doing any major modifications.

Pushing iDRAC Template
You can now edit the XML file to change or add attributes.  You can then push this XML file to your iDRAC for application by substituting get with set above.

idracadm -r 10.1.8.59 -u root -p PASSWORD set -t xml -f host-edit.xml -l 10.1.8.30:/srv/configs

Example:  Adding new local iDRAC Users
In this example you can easily add new users to any number of iDRACS.  Note the Users.4 below and be sure to increment this for future additions (or track it somewhere).  We simply appended Users.4 with a new user which will auto-create on the DRAC end.  Future users would use Users.5 and so forth, iterating upwards.

<!-- <SystemConfiguration> -->
<SystemConfiguration Model="PowerEdge R620" ServiceTag="3H5XXXX" TimeStamp="Wed Dec 11 11:38:15 2013">
<!--Exported configuration does not contain FC-HBA data as the presence of the appropriate cards could not be detected!-->
<Component FQDD="iDRAC.Embedded.1">
 <Attribute Name="IPMILan.1#Enable">Disabled</Attribute>
 <Attribute Name="IPMILan.1#PrivLimit">Administrator</Attribute>
 <Attribute Name="IPMILan.1#EncryptionKey">0000000000000000000000000000000000000000</Attribute>
 <Attribute Name="IPMILan.1#CommunityName">public</Attribute>
 <Attribute Name="IPMILan.1#AlertEnable">Disabled</Attribute>
 <Attribute Name="Users.4#UserName">qe-scale</Attribute>
 <Attribute Name="Users.4#Password">********</Attribute>
 <Attribute Name="Users.4#Privilege">507</Attribute>
 <Attribute Name="Users.4#IpmiLanPrivilege">Operator</Attribute>
 <Attribute Name="Users.4#IpmiSerialPrivilege">Operator</Attribute>
 <Attribute Name="Users.4#Enable">Enabled</Attribute>
 <Attribute Name="Users.4#SolEnable">Enabled</Attribute>
 <Attribute Name="Users.4#ProtocolEnable">Enabled</Attribute>
 <Attribute Name="Users.4#AuthenticationProtocol">SHA</Attribute>
 <Attribute Name="Users.4#PrivacyProtocol">AES</Attribute>
</Component>
</SystemConfiguration>

Example:  Carving out RAID Disks
You can also carve out RAID disks with the XML templates.  The easiest way to do this is to set one machine up manually, pull the configuration down and then push it to other iDRACs.  They should reboot into the PERC menu and do the right thing.

Example: Changing the Boot Order of NICs
Sometimes it’s useful to switch the order of what NICs will PXE first, you can do this with an XML template.  This is useful in environments where a lot of automated bare-metal provisioning might be needed.

  • .1.1.1 = first NIC
  • .1.2.1 = second NIC

You can view the current boot order from the iDRAC via racadm commands

ssh mgmt-file01.example.com racadm get BIOS.BiosBootSettings.BootSeq

Note: If you add new network interfaces you’ll need to reboot the iDRAC for them to show up in this listing so they can be manipulated.

You can modify this order in the XML below:

 <Attribute Name="BiosBootSeq">NIC.Integrated.1-2-1, HardDisk.List.1-1, NIC.Integrated.1-1-1</Attribute>

Viewing The current Job ID
It’s easy to make mistakes in XML, so you will want a way to view the status of a job.
You can view the status of any job by looking at it’s JID or Job ID.

idracadm -r 10.1.8.59 -u root -p PASSWORD jobqueue view -i JID_404441080579
---------------------------- JOB ------------------------- 
[Job ID=JID_404441080579]
Job Name=Export: System configuration XML file
Status=Completed
Start Time=[Not Applicable]
Expiration Time=[Not Applicable]
Message=[SYS043: Successfully exported system configuration XML file.]
----------------------------------------------------------

Viewing All Prior Jobs
You can simply list the status of all jobs run also.

idracadm -r 10.1.8.59 -u root -p PASSWORD jobqueue view

-------------------------JOB QUEUE------------------------ 
[Job ID=JID_365611035667]
Job Name=Configure: Import system configuration XML file
Status=Completed
Start Time=[Not Applicable]
Expiration Time=[Not Applicable]
Message=[SYS053: Successfully imported and applied system configuration XML file.]
----------------------------------------------------------
[Job ID=JID_254163911329]
Job Name=Export: System configuration XML file
Status=Completed
Start Time=[Not Applicable]
Expiration Time=[Not Applicable]
Message=[SYS043: Successfully exported system configuration XML file.]
----------------------------------------------------------

Other Useful iDRAC Commands
You can reboot a bare-metal server remotely (useful for automating large-scale systems provisioning) via the following command:

idracadm -r 10.1.8.59 -u root -p PASSWORD serveraction powercycle

Rebooting a remote iDRAC.

idracadm -r 10.1.8.59 -u root -p PASSWORD racreset soft

You can also reboot a local iDRAC directly from the CLI of the server after querying it for information.  This is useful in case of OOB failure/troubleshooting:

/opt/dell/srvadmin/sbin/srvadmin-services.sh start
/opt/dell/srvadmin/sbin/racadm getsysinfo 2>&1 | less
/opt/dell/srvadmin/sbin/racadm racreset

Updating the lifecycle controller firmware via racadm.

You can use an anon FTP server with the extracted payload/firmimg file.  This can be extracted from the .EXE as it’s really just a zip archive.  The example below is an iDRAC update with the 2.41.40.40 version renamed from payload/firmimg.d7

ssh mgmt-c10-h07-r730xd.example.com racadm fwupdate -f 10.12.64.163 ftp user -d pub/2.41.40.40.d7

Caveat – iDRAC username Length
We found a bug where if you exceed the username character limit of 16 characters the iDRAC will hard lock, requiring a manual hands-on reset or worse.  We’ve filed this issue with Dell but aren’t sure if it’s fixed or not.  Avoid usernames greater than 16 characters.

Caveat – iDRAC IP vs. FQDN
In later versions of the iDRAC firmware there’s a bug in DNS lookups that fail when using a FQDN with racadm to push commands.  Use the IP address to work around this.  We’ve traced this down via GDB and strace it and seems to fail getaddr() (we’ve also notified Dell).

Fix Hung iDRAC Tasks
Occassionally the iDRAC may get hung up on job requests not completing and piling up, rebooting will not fix this.  The following command may work to unwedge them.

racadm -r drac.ip.address -u root -p password jobqueue delete -i JID_CLEARALL_FORCE

Then wait about 120 seconds and reboot the iDRAC again

racadm –r drac.ip.address –u root –p password racreset

Usage and Extension
We utilize the racadm CLI tools with a simple shell for loop to quickly provision multiple racks with only a few commands.  We tie this into Foreman for full lifecycle deployment and management and tools like Ansible and Puppet to provide configuration management.

Using Ansible to Manage Dell Servers via Racadm
You can do a lot with Ansible, here’s some Ansible playbooks we use to do things like toggle NIC interface boot order or also change RAID configuration remotely and other things.  Feel free to use/borrow these as examples and build upon them further.

Automating Everything
When deploying new servers you can tie all this together.  Everyones infrastructure is different but this is how we do it:

  • Obtain MAC address information beforehand of any new servers and iDRAC
  • Prep racadm commands with proper templating to fire off when servers are racked
  • Prep your DHCPD entries for the iDRAC interfaces so they come online when powered on
  • Prep the Foreman Hammer CLI commands to create host entries, set to build
  • Once servers are powered on, run your deployment script
    • racadm commands to create accounts, settings, carve disks etc.
    • Foreman Hammer CLI commands to provision bare-metal hosts
  • Optionally, automatically populate a Wiki based on the new servers

This set of examples is by no means exhaustive, experiment and see what else you can do with it.  We’ve found almost everything you normally had to do within a browser or Java OOB can be done by referencing parameters in the iDRAC XML template.  I take no responsibility for any damage that may be done.  Please comment if you’ve got new use cases we can add.

About Will Foster

hobo devop/sysadmin.
This entry was posted in open source, sysadmin and tagged , , , , , , , , , , , , , . Bookmark the permalink.

8 Responses to Automating Dell iDRAC Administration with Racadm

  1. Thank you for the post about Automate Dell iDRAC Administration with XML, you helped me a lot.

    Like

  2. chris mcgrail says:

    Great article, thanks… How can you push the XML file to multiple iDRACs?

    Like

    • Will Foster says:

      Great article, thanks… How can you push the XML file to multiple iDRACs?

      Thanks Chris. If you want to push to multiple idracs you can do it sequentially via a for loop (assuming the XML files are generic enough to perform the same action).

      e.g.

      for x in `cat /tmp/list-of-drac-ip-addresses.txt`; do idracadm -r $x -u root -p PASSWORD set -t xml -f host-edit.xml -l 10.1.8.30:/srv/configs; done

      10.1.8.30 would be an NFS server where you have the config available and /tmp/list-of-drac-ip-addresses.txt would be a list of iDRAC IP addresses (one per line) of all the iDRACs that you want to hit.

      Like

  3. Matt Lauer says:

    This is great. What if the servers you racked, all need firmware updates? Is that possible too via the XML config files? If not, I’m curious how you handle firmware updates. With this elegant solution, I’m sure you’re not wasting countless hours standing in front of a BIOS screen and waiting for the Lifecycle controller to load :)

    Like

    • Will Foster says:

      Hey, I’m glad you found it useful. XML configs are more for applying IPMI-level changes or settings and not really the best way to apply firmware and BIOS updates however.

      The easiest way to handle BIOS and firmware updates is just ssh to racadm and call it.

      This is what we do, and run a big loop for all our systems of the same type:

      e.g. for BIOS (download the .EXE file – it’s just a zip archive)

      * make a list of iDRAC / interfaces and provide an NFS mount (10.12.64.161 in our case) containing the .EXE BIOS update:

      for host in $(cat /tmp/r620-sys); do ssh root@$host racadm update -f BIOS_NGYCX_WN64_2.7.0.EXE -l 10.12.64.161:/ srv/configs/R630 --reboot; done

      For Firmware the same thing, but we use an anon vsftpd server (you can substitute for nfs like above). You also want to extract the firmware payload file (firmimg.d7) instead of pointing it at the .EXE archive (Dell provided .EXE are just really zip files)

      for host in $(cat /tmp/630-sys); do ssh mgmt-$host racadm fwupdate -f 10.12.64.163 ftp user -d pub/firmimg.d7 ; done

      Like

      • Matt Lauer says:

        Excellent! I am going to try this out on the 18 Dell systems (Dell R620 and R720s w/ iDRAC8) that just arrived yesterday. Do you also have a trick to first getting your desired IPs configured on the DRACs? If I power up 18 systems, they all have the default 192.168.0.120 IPs factory set on all the DRACs. If they are all set to DHCP clients by default, then I could harvest the MAC address and IP for each one from our DHCP server table.

        Like

      • Will Foster says:

        Yeah it’s irritating that Dell iDRAC ship with a 192.168.0.x scheme and not DHCP by default, our way around this is to have our Dell VAR reconfigure the iDRAC from the front panel to DHCP before we get the systems and just send us the Mac addresses ahead of time, that way we can pre-populate them into Foreman (what we use for DHCP/PXE/DNS/TFTP/Provisioning).

        There’s no shortcuts here besides plugging in a system into the same management switch your iDRACs go into and give it a 192.168.0.x scheme with no gateway on one of the interfaces, then you can tunnel through it to access all the iDRAC interfaces (or use racadm commands over ssh) to change them to DHCP.

        You can scrape the mac addresses from the cam tables (show ethernet switching table on JunOS for us) on the management switches as well in a pinch once it generates some network activity. Hope this helps.

        Like

Have a Squat, Leave a Reply ..

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.