Automate Dell iDRAC Administration with XML

Tuxdell_logoManaging more than a handful of Dell servers becomes cumbersome using the iDRAC web interface.  This is especially painful with several hundred servers or more.  Luckily Dell provides CLI tools you can use with XML templates to automate provisioning and administration for you.

Why use XML Templates?
We tie in a bunch of other Open Source tools like Foreman, Ansible and Puppet to provide a seamless lifecycle and management workflow.  Dell has proprietary software to manage iDRACs but using the innate XML template functionality within the iDRAC better fits our provisioning workflow with CLI tools and orchestration.  Lastly, being able to revision control things into an SCM like git is important for us.

Requirements:
You’ll need a few basic things before you proceed:

  • An EL5, EL6, EL7-based Linux host or VM
  • Access to the internet and access to the iDRAC interface you want to manage
  • A NFS server export (for storing/serving XML templates)

Download the iDRAC CLI Repo
Repo instructions are taken from Dells site here

wget -q -O - http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash

Piping anything into bash is generally a bad idea, but this a vetted method.
This will setup a few extra yum-specific plugins needed, and a few dependencies.

Downloading GPG key: http://linux.dell.com/repo/hardware/latest/RPM-GPG-KEY-dell
    Importing key into RPM.
Downloading GPG key: http://linux.dell.com/repo/hardware/latest/RPM-GPG-KEY-libsmbios
    Importing key into RPM.
Write repository configuration
Downloading repository RPM
Installing repository rpm: http://linux.dell.com/repo/hardware/latest/platform_independent/rh60_64/prereq/dell-omsa-repository-2-5.noarch.rpm
Installing yum plugins for system id
Loaded plugins: product-id, security, subscription-manager, versionlock

Dependencies installed:

================================================================================
 Package                Arch      Version              Repository          Size
================================================================================
Installing:
 yum-dellsysid          x86_64    2.2.27-4.4.1.el7     dell-omsa-indep     16 k
Installing for dependencies:
 python-smbios          x86_64    2.2.27-4.4.1.el7     dell-omsa-indep     69 k
 smbios-utils-python    x86_64    2.2.27-4.4.1.el7     dell-omsa-indep     63 k

Transaction Summary
================================================================================
Install  1 Package (+2 Dependent packages)

Install iDRAC CLI tools
This will pull in smbios-utils-bin, srvadmin-argtable2, srvadmin-omilcore

yum install srvadmin-idracadm openssl-devel

How to Use the iDRAC CLI
There are a handful of very useful ways to use the iDRAC CLI, almost any action you’d normally have to do in a browser or remote OOB Java viewer you can do with the tool.  It can also parse and read XML templates to configure the iDRAC.

  • 10.1.8.30:/srv/configs is our local NFS (RW) share in these examples.

Managing iDRAC XML Templates
iDRAC automation centers around manipulation of XML templates which comprise the configurations and settings of both the iDRAC and the attached peripherals (PERC cards, etc).

Changes will revolve around pulling, editing, pushing templates.
It’s also useful to revision control these if you like via git, but entirely optional.

Pulling iDRAC Template
This is useful for safekeeping before editing and also to modify and use for pushing a similar configuration across many many systems.

idracadm -r 10.1.8.59 -u root -p PASSWORD get -t xml -f host.xml -l 10.1.8.30:/srv/configs

Now you should have an XML file called host.xml or whatever you called it sitting in the /srv/configs NFS share.  It’s a good idea to backup all of your configs this way before doing any major modifications.

Pushing iDRAC Template
You can now edit the XML file to change or add attributes.  You can then push this XML file to your iDRAC for application by substituting get with set above.

idracadm -r 10.1.8.59 -u root -p PASSWORD set -t xml -f host-edit.xml -l 10.1.8.30:/srv/configs

Example:  Adding new local iDRAC Users
In this example you can easily add new users to any number of iDRACS.  Note the Users.4 below and be sure to increment this for future additions (or track it somewhere).  We simply appended Users.4 with a new user which will auto-create on the DRAC end.  Future users would use Users.5 and so forth, iterating upwards.

<!-- <SystemConfiguration> -->
<SystemConfiguration Model="PowerEdge R620" ServiceTag="3H5XXXX" TimeStamp="Wed Dec 11 11:38:15 2013">
<!--Exported configuration does not contain FC-HBA data as the presence of the appropriate cards could not be detected!-->
<Component FQDD="iDRAC.Embedded.1">
 <Attribute Name="IPMILan.1#Enable">Disabled</Attribute>
 <Attribute Name="IPMILan.1#PrivLimit">Administrator</Attribute>
 <Attribute Name="IPMILan.1#EncryptionKey">0000000000000000000000000000000000000000</Attribute>
 <Attribute Name="IPMILan.1#CommunityName">public</Attribute>
 <Attribute Name="IPMILan.1#AlertEnable">Disabled</Attribute>
 <Attribute Name="Users.4#UserName">qe-scale</Attribute>
 <Attribute Name="Users.4#Password">********</Attribute>
 <Attribute Name="Users.4#Privilege">507</Attribute>
 <Attribute Name="Users.4#IpmiLanPrivilege">Operator</Attribute>
 <Attribute Name="Users.4#IpmiSerialPrivilege">Operator</Attribute>
 <Attribute Name="Users.4#Enable">Enabled</Attribute>
 <Attribute Name="Users.4#SolEnable">Enabled</Attribute>
 <Attribute Name="Users.4#ProtocolEnable">Enabled</Attribute>
 <Attribute Name="Users.4#AuthenticationProtocol">SHA</Attribute>
 <Attribute Name="Users.4#PrivacyProtocol">AES</Attribute>
</Component>
</SystemConfiguration>

Example:  Carving out RAID Disks
You can also carve out RAID disks with the XML templates.  The easiest way to do this is to set one machine up manually, pull the configuration down and then push it to other iDRACs.  They should reboot into the PERC menu and do the right thing.

Example: Changing the Boot Order of NICs
Sometimes it’s useful to switch the order of what NICs will PXE first, you can do this with an XML template.  This is useful in environments where a lot of automated bare-metal provisioning might be needed.

  • .1.1.1 = first NIC
  • .1.2.1 = second NIC

You can view the current boot order from the iDRAC via racadm commands

ssh mgmt-file01.example.com racadm get BIOS.BiosBootSettings.BootSeq

Note: If you add new network interfaces you’ll need to reboot the iDRAC for them to show up in this listing so they can be manipulated.

You can modify this order in the XML below:

 <Attribute Name="BiosBootSeq">NIC.Integrated.1-2-1, HardDisk.List.1-1, NIC.Integrated.1-1-1</Attribute>

Viewing The current Job ID
It’s easy to make mistakes in XML, so you will want a way to view the status of a job.
You can view the status of any job by looking at it’s JID or Job ID.

idracadm -r 10.1.8.59 -u root -p PASSWORD jobqueue view -i JID_404441080579
---------------------------- JOB ------------------------- 
[Job ID=JID_404441080579]
Job Name=Export: System configuration XML file
Status=Completed
Start Time=[Not Applicable]
Expiration Time=[Not Applicable]
Message=[SYS043: Successfully exported system configuration XML file.]
----------------------------------------------------------

Viewing All Prior Jobs
You can simply list the status of all jobs run also.

idracadm -r 10.1.8.59 -u root -p PASSWORD jobqueue view

-------------------------JOB QUEUE------------------------ 
[Job ID=JID_365611035667]
Job Name=Configure: Import system configuration XML file
Status=Completed
Start Time=[Not Applicable]
Expiration Time=[Not Applicable]
Message=[SYS053: Successfully imported and applied system configuration XML file.]
----------------------------------------------------------
[Job ID=JID_254163911329]
Job Name=Export: System configuration XML file
Status=Completed
Start Time=[Not Applicable]
Expiration Time=[Not Applicable]
Message=[SYS043: Successfully exported system configuration XML file.]
----------------------------------------------------------

Other Useful iDRAC Commands
You can reboot a bare-metal server remotely (useful for automating large-scale systems provisioning) via the following command:

idracadm -r 10.1.8.59 -u root -p PASSWORD serveraction powercycle

Rebooting a remote iDRAC.

idracadm -r 10.1.8.59 -u root -p PASSWORD racreset soft

You can also reboot a local iDRAC directly from the CLI of the server after querying it for information.  This is useful in case of OOB failure/troubleshooting:

/opt/dell/srvadmin/sbin/srvadmin-services.sh start
/opt/dell/srvadmin/sbin/racadm getsysinfo 2>&1 | less
/opt/dell/srvadmin/sbin/racadm racreset

Updating the lifecycle controller firmware via racadm.

You can use an anon FTP server with the extracted payload/firmimg file.  This can be extracted from the .EXE as it’s really just a zip archive.  The example below is an iDRAC update with the 2.41.40.40 version renamed from payload/firmimg.d7

ssh mgmt-c10-h07-r730xd.example.com racadm fwupdate -f 10.12.64.163 ftp user -d pub/2.41.40.40.d7

Caveat – iDRAC username Length
We found a bug where if you exceed the username character limit of 16 characters the iDRAC will hard lock, requiring a manual hands-on reset or worse.  We’ve filed this issue with Dell but aren’t sure if it’s fixed or not.  Avoid usernames greater than 16 characters.

Caveat – iDRAC IP vs. FQDN
In later versions of the iDRAC firmware there’s a bug in DNS lookups that fail when using a FQDN with racadm to push commands.  Use the IP address to work around this.  We’ve traced this down via GDB and strace it and seems to fail getaddr() (we’ve also notified Dell).

Fix Hung iDRAC Tasks
Occassionally the iDRAC may get hung up on job requests not completing and piling up, rebooting will not fix this.  The following command may work to unwedge them.

racadm -r drac.ip.address -u root -p password jobqueue delete -i JID_CLEARALL_FORCE

Then wait about 120 seconds and reboot the iDRAC again

racadm –r drac.ip.address –u root –p password racreset

Usage and Extension
We utilize the racadm CLI tools with a simple shell for loop to quickly provision multiple racks with only a few commands.  We tie this into Foreman for full lifecycle deployment and management and tools like Ansible and Puppet to provide configuration management.

Automating Everything
When deploying new servers you can tie all this together.  Everyones infrastructure is different but this is how we do it:

  • Obtain MAC address information beforehand of any new servers and iDRAC
  • Prep racadm commands with proper templating to fire off when servers are racked
  • Prep your DHCPD entries for the iDRAC interfaces so they come online when powered on
  • Prep the Foreman Hammer CLI commands to create host entries, set to build
  • Once servers are powered on, run your deployment script
    • racadm commands to create accounts, settings, carve disks etc.
    • Foreman Hammer CLI commands to provision bare-metal hosts
  • Optionally, automatically populate a Wiki based on the new servers

This set of examples is by no means exhaustive, experiment and see what else you can do with it.  We’ve found almost everything you normally had to do within a browser or Java OOB can be done by referencing parameters in the iDRAC XML template.  I take no responsibility for any damage that may be done.  Please comment if you’ve got new use cases we can add.

About Will Foster

hobo devop/sysadmin, all-around nice guy.
This entry was posted in open source, sysadmin and tagged , , , , , , . Bookmark the permalink.

4 Responses to Automate Dell iDRAC Administration with XML

  1. Thank you for the post about Automate Dell iDRAC Administration with XML, you helped me a lot.

    Like

  2. chris mcgrail says:

    Great article, thanks… How can you push the XML file to multiple iDRACs?

    Like

    • Will Foster says:

      Great article, thanks… How can you push the XML file to multiple iDRACs?

      Thanks Chris. If you want to push to multiple idracs you can do it sequentially via a for loop (assuming the XML files are generic enough to perform the same action).

      e.g.

      for x in `cat /tmp/list-of-drac-ip-addresses.txt`; do idracadm -r $x -u root -p PASSWORD set -t xml -f host-edit.xml -l 10.1.8.30:/srv/configs; done

      10.1.8.30 would be an NFS server where you have the config available and /tmp/list-of-drac-ip-addresses.txt would be a list of iDRAC IP addresses (one per line) of all the iDRACs that you want to hit.

      Like

Have a Squat, Leave a Reply ..

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s