My essential Ubuntu applications

A few times recently i’ve had to think about the essential applications i use on my desktop.  The latest was Anthony Burke’s tweet, but the recent churn in the Linux desktop world and my unhappiness with Unity means that i need to be prepared for moving away from Ubuntu when Unity becomes the only option.  (I’m currently on Ubuntu 10.04 LTS “lucid lynx”, so this hopefully will be some time away.)  So here’s my list of desktop essentials, mostly for my benefit, but hopefully of use to others.  Many of these are simply Ubuntu’s default applications for their respective tasks and are provided in the base install.

Everyday essentials

Most of these applications are open all the time on my laptop:

  • Mozilla Firefox – recently updated to the fast-release versions on 10.04 LTS, which offers some great speed improvements.
  • Mozilla Thunderbird – I’ve been a Netscape/Thunderbird user for more than 10 years now, and i still can’t understand why people put up with a less-capable email client.
  • Pidgin – instant messaging for IRC, XMPP (Jabber, Google Talk), and Twitter (via tweet.im)
  • Amarok 1.4 – music player.  I use the older version because it pulls down my podcasts automatically, rescans my library automatically, and i can just type “198” into the search box and get a great selection of music from the 1980s, instead of having to write match queries (which the new version of Amarok seems to require).
  • Workrave – rest break software for preventing RSI
  • evince – the default PDF reader; it’s rare that i don’t have 2 or 3 PDF files open for reference, sometimes more like 20 or 30
  • Google Desktop – search for all of those local PDFs filling up my hard disk
  • OpenOffice – newer versions of Ubuntu & Debian have moved to LibreOffice now, but 10.04 LTS still uses the Sun/Oracle version
  • Tomboy – simple desktop notes
  • Getting Things Gnome (gtg) – my much-ignored todo list
  • icewm – I replace the default GNOME 2 window manager with icewm, which is a very simple, fast, customisable window manager
  • J-Pilot – my password database dates back to my PalmOS days, so J-Pilot is my password manager even though i don’t use PalmOS devices any more.  People starting fresh would more likely find KeePass or something similar a better choice.
  • By far the most-used applications on my laptop are gnome-terminal, ssh, vim, git, bash, and the suite of Linux/Unix shell script utilities.  I do most of my coding in vim, and rarely have less than 10-15 shell sessions open to different servers or network devices.  Most of my important work happens on the servers, not on the desktop.

Other applications

These are less frequently-used, and aren’t open all the time:

  • Liferea – RSS feed reader
  • Opera – i use this to keep my finance-related web sites separate from my main browser
  • Google Picasa – photo organiser which offers automatic sync to Picasa Web, and simple export to Facebook & Gallery; runs as a Windows application under WINE
  • baobab (Disk Usage Analyser) – great for tracking down where i need to trim back on my disk usage; its exploded pie charts are outstanding (Users of other operating systems might like to check out JDiskReport, which offers the same type of chart.)
  • Unison – file synchronisation
  • gns3/dynamips – Cisco IOS emulator
  • dia – diagram editor; much more rudimentary than Microsoft Visio, but still very usable.  I mostly work with pencil & paper when it comes to network diagrams and the like anyway.
  • GnuCash – double-entry accounting system; all of my business accounting happens here
  • Adobe Acrobat Reader 9 – for some reason, my bank produces PDFs that evince can’t read but Acrobat Reader can
  • Eclipse – IDE for Java coding
  • Audacity – sound editor
  • grip – old-school CD ripper which gives complete control over MP3/OGG encoding options


Source: libertysys.com.au

Initial thoughts on the HP A5500-EI switch

Background

I first came across HP’s A5500 switches when i started looking at configuring VRRP and distributed trunking to provide routing redundancy for the core of a client’s campus network using two ProCurve 5400 switches. I found that i could get a new pair of A5500s for around the same price as the software license upgrade on the 5400s (multicast routing, distributed trunking, and OSPF cost extra), and they came with a much broader range of features. HP’s IRF is far preferable to VRRP and distributed trunking from the perspective of administrative complexity, since the IRF stack appears as a single switch to hosts and other switches.

The 5400 is a chassis-based switch, and replacing it with the stackable 5500 would not normally be ideal for a core switch setup, but we had very low port density and bandwidth requirements in our core, so two 24-port 5500s were sufficient. The 5400s were moved to the distribution layer along with a pair of 2824s used primarily as copper-to-fibre converters. Each distribution switch has four 1 Gbps links to the core (two to each switch) as an LACP trunk, as does each of our four VMware ESX servers.

I’ve been using ProCurve networking for about 9 years, working with it directly as a network admin for the past 5. Most of my networking experience has been with E series switches, so i will write mostly from the perspective of a ProCurve user. I’ve learned about these swiches mostly from reading the manuals and testing features as we migrated our core from standalone 5400s to an IRF stack of 5500s, but Wayne Jewell at Layer127 also provided some helpful assistance with ACLs and OSPF.

This review will probably degenerate into note form before too long, since i’ve been sitting on a draft of this post for far too long already, and i want to get it out the door. 🙂

Packaging and hardware

The information in this review is based on A5500-EI switches running Comware 5.2, revision 2208. The switches were shipped on firmware 2202; finding, downloading, and upgrading the firmware to 2208 (via Xmodem over the console cable) was an easy and straightforward process. (Cisco, take note!) After i set up the IRF stack, the second switch automatically downloaded the firmware from the first switch, installed it, and rebooted as an IRF stack member.

The switches are branded H3C rather than HP. I assume this will change as they do new manufacturing runs. Hardware quality of the units themselves seems excellent – they continue HP’s excellent track record with the ProCurve platform, and have inherited its lifetime warranty.

Perplexingly, port numbering on the switches starts from the bottom left rather than the top left, which is rather disconcerting to those of us expecting a traditional layout. I assume this is due to the switch’s Chinese heritage. I’ve intentionally designed the port configuration on our stack to minimise the risk of errors related to this, but i’m pretty sure it will cause operational problems sooner or later. Let’s hope that HP produces region-specific versions of these switches which allow customers to choose top-to-bottom port numbering.

Main features

Features i’ve tried (in alphabetical order):

  • ACLs
  • DHCP relay
  • IRF
  • LACP (bridge aggregation)
  • NTP
  • OSPF
  • PIM (dense mode)
  • RIP
  • UDP helpers

All of the above work as expected and without hiccup. When i first connected the IRF stack as the network core, it was a disappointingly quiet maintenance window; there were no major problems to deal with.

One perplexing issue which hasn’t been resolved yet on our 5500s is that 20% of ping packets to any IP interface on the switches are lost. There have been no issues with performance of the switching and routing, so it seems to be just a management plane issue. (See further comments on CLI responsiveness below.)

User interface

All of my access to the 5500 has been at the CLI via console cable or ssh session. I haven’t bothered configuring the web interface, nor looking at HP’s IMC management software (which i hear is excellent). Most of the rest of this review will relate to CLI features.

Command syntax

Much of the Comware command syntax seems to use alternative vocabulary from the command sets on most other switches and routers as a deliberate technique to make it seem different from Cisco. However, this difference seems to be only skin-deep, and most of the commands follow an almost identical structure to Cisco. (This is particularly evident in interface context.) I don’t have enough experience with 3Com equipment to comment on whether this was the case with earlier Comware releases or is a feature of only the post-Huawei switches.

‘display this’ is an extremely useful command when you’re configuring anything which has its own context (e.g. OSPF). It gives you immediate feedback on what you’ve just configured, without introducing noise or requiring pipes.

‘hotkey’ is a way to create shortcuts for common commands. I’ve defined:

  • Ctrl-O as ‘display ip rout | exclude InLoop0’ (mnemonic: ‘rOutes’)
  • Ctrl-T as ‘display this’ (mnemonic: ‘This’)
  • Ctrl-U as ‘return’ (mnemonic: ‘Up one level’)

‘command-alias’ is great for making things a little more familiar. Examples of common commands which might be aliased are:

  • show – display
  • no – undo
  • exit – quit
  • end – return

The default output of many diagnostic commands (e.g. ‘display vlan’) is unneccessarily verbose by default. I much prefer the ProVision approach, which gives brief tabular data by default and requires additional parameters to extract details.

Interacting with the CLI

Having access to basic pipe facilities (‘| begin/include/exclude’) seems like a minor issue, but after using it for a few days, going back to switches without this feature becomes increasingly frustrating. (I understand this has also been added on recent 5400 firmware versions.) I long for a real ‘less’ command which can page forwards and backwards in command output. (I hear Cisco has a real ‘grep’ command on their latest IOS; hopefully this is a sign of better things to come.)

The 5500 doesn’t seem to detect terminal length like the 5400 does. The 5400 detects changes in terminal size even during the same session, whereas the 5500 seems fixed at 25 lines.

Ctrl-C doesn’t cancel command input, only running commands (well, some of them – it doesn’t work with ‘display ntp-service trace’). One has to substitute Ctrl-E followed by Ctrl-X or Ctrl-A followed by Ctrl-Y instead. Ctrl-A is a pain for me to use because i’m usually connected either through minicom or GNU screen. I long for a real bash shell (which supports vi editing mode) on a switch.

Note to self: investigate Vyatta and Arista on this point.

The 5500 is somewhat more sluggish to respond to commands at CLI than the 5400, both when connected via the console cable, and when using ssh over gigabit Ethernet. This may be due to a relatively low-powered CPU being used for management, but that would explain poor ssh keystroke latency without explaining the same experience over a 115,200 baud console cable. Most of HP’s low-end ProCurve switches (e.g. 2824, 2610) are more responsive than the 5500.

The 5500 CLI also requires that the backspace character be set to Ctrl-H instead of the usual delete, and i could find no way to change it in the CLI (i.e. there’s no equivalent to stty on Linux/Unix). This is more troublesome for those of us who routinely switch between Comware and ProVision switches. A facility in the CLIs to allow a consistent backspace setting to be used would be highly desirable. (Alternatively, they could just allow both Ctrl-H and DEL on both platforms and then everyone could forget about it. I suspect this is what ProVision already does.)

VLAN setup

Configuration of ports and VLANs is the most Cisco-styled part of the CLI and is much harder with Comware than ProVision. I personally think that dealing with port modes is the most cumbersome part of configuration on all of the non-ProCurve switches i’ve worked with, and i very much hope HP retrofits the VLAN parts of the ProVision CLI to these switches.

Changing a hybrid port to a trunk port (or vice versa) requires setting it to an access port first. I can’t conceive of any technical reason for this being exposed in the CLI; if it is a technical requirement, it should be done automatically by the underlying OS.

On that note, is anyone out there using hybrid ports in a production environment? I can think of only one reason for using them (protocol-based VLANs), and i can’t see why anyone would want to use that for anything more than a fun experiment.

ProVision has ‘show run vlan’ on recent firmwares, which shows only the VLAN definition parts of the config. There seems to be no equivalent on Comware, even though there are similar parameters for other subsystems (e.g. ‘display current-configuration configuration acl-adv’).

VLAN descriptions can only be 32 characters long – this is not enough to give a useful description. Note that this is not the (short) display name, which is set separately.

ACLs

ACLs use negated rather than conventional netmasks – this seems rather counter-intuitive to those of us who have worked mostly with host-based firewalls, but once you are familiar with it, it isn’t too hard. Anyone who has worked a lot with OSPF on Cisco devices should find it no problem.

Non-contiguous netmasks may be used with ACLs, which is a useful feature if you use a standard numbering system for gateway addresses, server IP ranges, and the like.

The ‘hardware-count’ feature is disabled by default on all ACL entries. Since the switch automatically reverts to software counting for any ACLs which do not support hardware counting (e.g. SNMP access to the management plane), i cannot conceive of a reason why it should not be enabled in all cases.

Documentation

The H3C documentation will be unfamiliar to people who are expecting ProCurve’s rigidly-defined format, but it is mostly of a high standard. It wastes a lot of space repeating how to get to the required view for most commands and shows some signs of backtranslation into English (also present in the CLI), but i have only encountered only one error so far (the documentation of ‘display interface brief’ as ‘display brief interface’), which was corrected in a subsequent version of the document. HP networking’s documentation site seems to lag behind the current firmware version; i found the documents updated for 2208 on H3C.com months before they appeared on HP.com.

HP networking have produced a resource which has made my conversion from ProVision to Comware much more straightforward: the HP Networking and Cisco CLI Reference Guide, which can be found at HP Networking’s training resources page. This document compares the Comware, ProVision, and IOS versions of many common commands, focusing on practical day-to-day differences and providing lots of examples. It is an indispensable reference for network engineers switching between these platforms, and has even been a useful resource in helping me become more familiar with Cisco’s CLI.

The Future

After working with Comware for a few months, i have come to appreciate its feature-richness compared with ProVision. I have really enjoyed learning the Comware platform, and it is understandable that HP have been strongly focusing on it since the 3Com acquisition. I’ve only really begun to scratch the surface of their security features, which would be really useful at the access layer. I expect that i’ll be recommending these switches to clients a lot more than ProVision-based ones in the future.

IRF supports a ring topology with up to 9 switches on the 5500s, so before long we’ll probably aim to upgrade our campus backbone fibre ring to 10 Gbps with 5500 switches at each point on the ring, providing complete switching and routing redundancy.


Source: libertysys.com.au

A strange rrdtool error; Linux conntrack documentation

Last week i made some fairly significant changes on a client’s production firewall/routing cluster during our maintenance window.  The next morning there were reports of file server drives not connecting correctly and inaccessible web sites.  Because all wireless-to-wired and Internet traffic goes through this cluster, the firewall changes were the obvious culprit.  Looking at the logs it turned out we had run out of space in the connection tracking table:

May 26 08:55:05 corella1 kernel: ip_conntrack: table full, dropping packet.
May 26 08:55:13 corella1 kernel: ip_conntrack: table full, dropping packet.
May 26 08:55:15 corella1 kernel: ip_conntrack: table full, dropping packet.

I checked the counters in /proc/sys/net/ipv4/netfilter/, upped the limit for net.ipv4.netfilter.ip_conntrack_max in /etc/sysctl.conf to 4 times its previous value, and loaded the new value into /proc.

Then i started to hack up a few little scripts to monitor and graph ip_conntrack_count against ip_conntrack_max using rrdtool. I’ve used rrdtool a little before, so i thought it would be pretty straightforward.  I created my RRD file and started updating it every minute with the latest counters from netfilter.  However, as soon as i tried to graph it i got the error

ERROR: parameter ‘cnt’ does not represent a number in line AREA:cnt#00FF00:countn

A search of Google brought up a lot of hits which contained the same text but were not relevant – most of them were errors in not specifying the variable correctly.  However, i came across one very similar problem: https://lists.oetiker.ch/pipermail/rrd-users/2007-November/013277.html

Unfortunately, this post on the rrdtool users mailing list had no responses, so i was down to solving it myself.  It took me some time before i realised that both the original poster of that message and myself had made exactly the same elementary mistake: forgetting to include a filename for the graph output.  This rudimentary error is not picked up by rrdtool’s command line parser (at least not as at version 1.2.12 on SUSE Linux Enterprise Server), resulting in a very confusing error message.

So then i had a working rrd graph on my firewall, which seems to have settled down nicely.  You can find the current (very rough) state of the scripts at https://github.com/paulgear/puppet/tree/2b5363a3fbc1e73d5d88158e93ab5d879910173b/modules/netfilter/files.

At the moment i’m only graphing the connection tracking count vs. its maximum (see the graph below).  Note the interesting minor variation on the graph from the max value that isn’t actually changing.  This seems to be due to rrdtool’s consolidation of data points – the change to a solid line was effected by truncating the date to an exact multiple of the step interval that the rrd was set up with (in this case, 60 seconds).

Sample conntrack rrd graph

After getting this working, i wondered whether there were other conntrack values i should be checking (the ip_conntrack_tcp_be_liberal and ip_conntrack_tcp_loose sounded particularly interesting) so i started going looking for documentation on the files in /proc/sys/net/ipv4/netfilter/. Initial searches came up with very little. The best description i could find of them was at http://netfilter.linux-kernel.at/documentation/pomlist/pom-extra.html#tcp-window-tracking, but i must admit that i crave more detail.  If anyone can point me to a better reference, or suggest which conntrack items really need monitoring, please drop me a line.

(Incidentally, i’ve discovered that collectd has a netfilter conntrack plugin, so i will probably not develop the scripts i created any further, but will try to adapt that plugin to my needs.)

Attachment Size
ipconntrack-day.png 35.95 KB


Source: libertysys.com.au

When (Windows) software updates go awry

One of my clients had some very interesting Internet traffic statistics last week.  We came in Thursday morning and found that overnight we had downloaded over 700 GB of data from our ISP (UQ SchoolsNet).

Traffic graph from last week

When we looked through our proxy server logs we found that 538 GB of the total came from a single PC attempting to download a single URL for Adobe Acrobat Reader 9.2 updates.  Fortunately, we’re on an Internet plan which is capped rather than charged for excess traffic, and more fortunately still, our ISP hosts an Akamai mirror, which is where the URL for the updates resolved to.  So, no harm done.

What this reinforced to me was that allowing direct access to the Internet by PCs is rather irresponsible, both from a bandwidth utilization perspective and a cost perspective.  (And that doesn’t even take into account what legal ramifications there might have been if it had been a BitTorrent client rather than a misconfigured/buggy software update client.)

Attachment Size
internet-traffic.png 20.37 KB
internet-traffic-2.png 60.4 KB


Source: libertysys.com.au