Tonight i was working on getting a client’s Zimbra SSL configuration up to scratch, and found it somewhat difficult to get our server to make Qualys’ SSL Labs scanner happy. I was working from the following Zimbra wiki pages:
It seems that as of Zimbra 8 (possibly before that?) there is no longer any need to configure jetty – everything seems to go through nginx as an SSL reverse proxy.I tried several different combinations and still kept getting insecure ciphers in the Qualys scan results until i stumbled across this nginx forum post and these certificate installation instructions. Between them i managed to glean that:
!aNULL is necessary to disable unauthenticated ciphers like TLS_DH_anon_WITH_AES_128_CBC_SHA and TLS_ECDH_anon_WITH_RC4_128_SHA (the latter is particularly infrequent in Google’s search results).
So the commands i ended up with for Zimbra were:zmprov modifyConfig zimbraReverseProxySSLCiphers '!ADH:!eNULL:!aNULL:!DHE-RSA-AES256-SHA:!SSLv2:!MD5:RC4:HIGH'zmmailboxdctl restartThis was enough to get us an “A” rating in Qualys’ eyes.
Prompted by a request from staff at a client’s head office, a couple of days ago i posed this question to a couple of the mailing lists i’m on: what is your size limit on individual email messages?
I was blown away by the speed, quantity, and quality of the responses i received from the AusNOG and SAGE-AU communities. Within an hour i had some hard data and a useful recommendation to take to my client.
I’ve published the statistics and the raw figures to separate sheetes in the same Google Docs workbook; a few explanatory comments about the results are necessary:
A number of responses indicated two values, often broken down by receive/send or internal/external criteria (with the latter being the smaller). This is indicated as “Tier 1” vs. “Tier 2” in the raw results. I’ve used the “Tier 1” figure to calculate the results.
Answers which were ambiguous or indicated no limit were not included in the calculations, nor was one answer of 5 GB, since it skewed the results unrealistically.
I’d say that anyone using something in the range of 10 – 50 MB could consider themselves reasonably “normal”; both those figures are within one standard deviation of the mean.
Here are some of the more interesting comments i received, along with the size they indicated. In most cases, these are direct quotes, but i’ve edited them for spelling, clarity, and punctuation where necessary. I’ve highlighted two responses that i found striking, given their closeness to the actual results. (I also suggest reading the AusNOG discussion – boththreads – some excellent points were made.)
If people need to send more than that, email is the wrong answer.
We’ve found in the past increasing above 15 MB resulted in a large number of bounce backs for organizations rejecting messages that were too large being sent to them. The biggest issue we have is explaining this to our customers and them believing it. Mainly because they don’t understand that a simple 8 MB JPEG can blow out to 20-25 MB because of mime encoding etc. We try our best to advise them of this, we do get quite a lot of arguing and feedback requesting we increase it anyway. However, slowly they’re realizing: when their large messages start bouncing back they ask us to set the limit back to what it was before.
I imagine a general consensus will be 25 MB upper limit due to Google Apps.
Most of my clients have gone Google Apps.
Our general view is that if a limitation is lower than what a customer gets on gmail (which is currently 25 MB) and related free services, then you will need to support at least that limit. A limit of 30 MB doesn’t have to be in place long before user actively notice that the limits are typically elsewhere, and start talking about how good their system is. Non-technical high-ups will struggle with paying for a business service that offers less than their personal accounts.
Microsoft did a risk assessment for us and noted that having large message sizes and large mailbox sizes (10G to 60G) is a high risk.
… and we still get complaints.
People still run into [our limit. We] had ‘someone’s IT guy’ tell us the ‘industry standard’ was 10 MB. I expect you’re getting a wide range of answers, and that there really isn’t an ‘industry standard’.
Unfortunately, I still get called every time an email bounces due to remote size limits.
We didn’t see any notable impact because of this change [to 100 MB], no delays, additional load or problems caused by the larger emails. Note: These clients had 20, 50, 100meg or faster Internet pipes.
I’m actually looking at reducing email size limits to force users into using technologies designed for file sharing and governance – Sharepoint, Skydrive Pro, etc. Reducing limits to 5 MB has all sorts of flow on effects: not even talking about freeing up link bandwidth, Exchange store sizes, etc. I’ve found that email enables poor habits. Emailing a 10 MB doc to the user 2 rooms down via a hosted exchange? Floods the link twice, plus stores the attachment in your local OST, the recipients local OST, and two copies in the exchange store. Now, modify it, and send it back. Ouch.
If I had to pick a single size that’s used, it would probably be 20 MB – but there’s no end of variation. 10 MB is common, although mainly for historic reasons, and the number of people with such a low limit is dropping. 25 MB and even 50 MB aren’t uncommon. 100 MB is rare, but out there – mainly in situations where mail is being sent to a specific recipient and they have also upped their [overall] limit. I’ve even seen one company who wanted their limit set to 1 GB…
I can not express enough the frustration in a customer saying they want to send a bigger email and wanting us to up our limit, explaining the internet is just too hard a task sometimes. In one specific case it was an 11 MB email, the customer response was “It’s only an extra 1 MB can you just let it through this once”, so I pointed him to an SMTP with no limit on it; next day he is forwarding a bounce back from the receiving end who blocked him based on size.
For those who are interested in the decision: my client and myself were both previously part of the “10 MB is the industry standard” camp, but found the argument about gmail compatibility compelling, and have decided to increase to 25 MB, much to the delight of the staff member pushing for the change.
Disclaimer: I am not a statistician; this is not a scientifically- or statistically-valid survey; all online polls are inherently bogus due to the respondents self-selecting; i have no idea whether this sample is statistically significant or valid; i did not attempt to authenticate or validate the responses in any way; YMMV; no warranties expressed or implied, etc.
This post is the story of my first practical look at Junos on Juniper EX-series switches.
One day last December, Skeeve Stevens from eintellego opened a can of worms by offering a deal on Juniper equipment to all network engineers on the AusNOG mailing list. I had been looking for an excuse to try out Juniper switches for a while; my office switch, an HP ProCurve 3400cl (J4905A), was very loud, and the EX2200-C was the only affordable fully-managed Gigabit fanless switch which i could find from a major networking vendor. So i ordered one and waited patiently. I expected it to take 6 weeks or so to arrive (since apparently all the cool kids ordered one too), so i was pleasantly surprised when it arrived the day after Boxing Day. (For those non-Commonwealth inhabitants reading this page, Boxing Day is the day after Christmas Day; it’s always a holiday here in Australia.) Over the Christmas/New Year break, i spent three days learning and playing with Junos. I don’t claim this as a fair, balanced, or reasonable review – it’s simply a recounting of my experiences.
I wanted to get a good general idea of the Junos platform and set it up as my main office switch, moving the HP to my test lab (which is only turned on when i’m using it). So the basic task i set myself was to learn enough Junos to replicate the functions of my ProCurve as completely as possible. I knew ahead of time that it wasn’t going to be a 100% fit – the 3400cl was a great value switch for its day (and still is, in many ways), and had a lot of features included out of the box (like OSPF and PIM-DM multicast routing). Junos requires an additional license to implement OSPF, and it wasn’t discounted (buying it would nearly have tripled the cost of the switch), so i knew that i’d be eliminating that part of my config. My other OSPF devices are VMs running Debian Linux with Quagga, and Cisco 1700 & 1800 series routers running IOS, so i simply connected them to the same VLANs and phased out the ProCurve as a router.
My first few impressions of the hardware were positive:
The switch has a solid metal case with a nice heavy feel to it.
It was well packed, and came with a very well-packed rack mount kit (perhaps a little too well-packed):
It also came with a rather longer-than-expected, but solid, power cord retainer:
It sports wall-mounting holes as well as a rack-mount kit (hat tip to Chris Lathem for pointing this out):
The biggest turn-off for me on the hardware side was the realisation that Juniper switches use zero-based port numbers. Seriously? Being an old C programmer, i think of this as “proper” from a programming perspective, but try telling an end-user (troubleshooting an issue in a branch office over the phone with tech support) that port 1 is the second port not the first. I can’t think of any reason other than compatibility with “the way they’ve always done it” on their routing platforms, and i consider this self-indulgent behaviour on an equipment manufacturer’s part. Other vendors manage to start unit or chassis numbers at zero without being bloody-minded about starting port numbers at zero (although HP’s Comware switches start their port numbers at the bottom left rather than the top left, which is almost as frustrating).
After my inspection of the hardware, i moved on to plugging it in and inspecting the software. Bootup took a bit longer than I expected (the system uses a 1.2 GHz Marvell ARM CPU – the same as in my QNAP TS-219P), but it looked reassuringly Unixy: Junos uses a FreeBSD-based management plane. After seeing the switch booted through to a login prompt, i logged in as root and had a poke around. Typical Unix commands like ls, df, and ps all had their usual functions, and one runs “cli” to get into the Junos configuration system. I worked for the first day or so from the serial console.
I noticed some disconcerting dates in the initial copyright notice:
“This product includes FreeBSD software developed by the University of California, Berkeley, … 1979, … 1994. “
Does this mean that 1994 is the most recent date they synchronised with the FreeBSD project? I hope this is not true, and that Juniper are actively engaging with the Open Source project upon which their code is based.
It has been a very long time since i’ve seen a Unix distribution old enough to contain a gated implementation, and i sincerely hope that no vestiges of this code remain.
During boot an alarm light appeared on the front panel. I was afraid i had a hardware fault and would have to return my shiny new switch! After some investigation in the relevant manuals, i found the command to view the active alarms (“show chassis alarms”) and found that it was due to the management interface being down. I already had a management VLAN in my network, and i was expecting this switch to host it. I investigated the management LAN (at this point a shout-out to the helpful folks in #juniper is due) and found that it was not VLAN-capable at all. It’s more like a NIC for the switch’s management plane. So it’s a true out-of-band network, and can’t be bridged onto any part of the data plane. I quickly decided not to use it (and disabled the alarm with “set chassis alarm management-ethernet link-down ignore”). I can understand why the management port is provided, but i’d much prefer that it could be bridged (in software, if necessary) onto the relevant VLAN for installations that aren’t large enough to have a dedicated physical out-of-band management network. I also think enabling alarms for the management LAN being disconnected in the factory default configuration violates the principle of least surprise.
Software Updates and Support
When commissioning a new switch, router, server, storage array, etc., one of the first things i do is look for the latest firmware/software to ensure i’m up-to-date with security patches. This proved to be a bit of an issue for Junos. As a newly-arrived customer, i didn’t know what to expect; HP offer all software updates for their switches on their public web and FTP sites; with Cisco it’s a bit more involved, requiring a login to their site, and with routers you need to know the right things to say to TAC if you don’t have a support contract (with switches they’re more generous – could this have anything to do with the existence of GNS3?).
With Juniper, it seems one must have a login on their site that is specifically granted access to register devices. I had already registered a user account on their site in order to download the documentation and view their certification materials, and i was somewhat perplexed and frustrated to find that i couldn’t even register my switch until Juniper had changed my account to have “real customer” status. Their support staff were first class, though, and got it sorted out pretty quickly.
My reading of documentation began with Juniper’s “Day One” guides, some of which i had downloaded prior to receiving the switch. These guides range from the 66-page “Exploring the Junos CLI” to the massive “Complete Software Guide for Junos OS for EX Series Ethernet Switches, Release 11.1”, weighing in at 4628 pages. Suffice it to say i only used the latter as a reference. If you’ve used Vyatta at all, you’ll find the former guide a fairly dull read. A quick skim to see the differences between Junos and Vyatta was all that was necessary for me. (Vyatta is a deliberate clone of the applicable parts of Junos’ interface.) “Configuring Junos Basics” was a useful guide to the initial setup of the switch, some parts of which have made it into my Network Engineer’s Rosetta Stone – initial setup wiki page. At 84 pages, it’s pretty digestible in a short CLI session. I used the free PDF format downloads, but most of the Day One guides are available for purchase in print format also. (Curiously, they cost US$0.99 to purchase in Kindle format, even though Kindle can read PDFs just fine.) The Day One guides (and the follow-on series “This Week”) are generally excellent in quality, and formatted for easy comprehension, although i thought that the headings could have been clearer in some cases. (“This Week: Hardening Junos Devices” devices is the one that i made mention of in my notes.)
My checklist for features to convert from ProCurve to Juniper looked like this:
Initial setup: user logins, hostname, management IP address, default gateway, DNS, NTP, time zone
Essentials: VLANs, Rapid Spanning Tree Protocol (RSTP), LACP trunk to VM server, port labels, syslog, SNMP, ssh public keys
Hardening: login inactivity timer, login banner, disable unused services (including telnet and web management), management ACLs, eliminate LLDP data leakage on Internet interfaces, Spanning Tree BPDU protection/BPDU filtering
Nice-to-have: QoS on VLANs, port mirroring, IGMP querier, DHCP relay
Initial setup was pretty straightforward, especially given the aforementioned Day One guide. Having worked with Vyatta for a few months, i found it easy to adjust to the Junos CLI style. It’s a simple and elegant configuration system which made instant sense to me the first time i used it, and Junos worked very much as i expected. When logged in via ssh, responsiveness of the EX2200-C at the CLI (in terms of character echo delay) is noticeably snappier than the ProCurve 3400cl it replaced (and many other similar devices).
Junos has a number of powerful configuration features. The ones i came to appreciate most were configuration groups (i used this to set up SNMP) and interface ranges (which i used to set up my VLAN assignments). The latter is very powerful: a range’s name can be used in place of an interface name almost anywhere, and it lives in the configuration as a real entity, as opposed to the temporary nature of the equivalent on IOS. (Recent HP Comware versions allow a port list to be permanently associated with a name, but store the configuration in the individual ports rather than the interface range itself.)
Configuration is very slow to commit on the EX2200-C compared with Vyatta on an Intel server (understandable, given the low-end CPU); “commit | display detail” set my mind at rest that it was doing useful things, so i just got used to waiting for my commits to complete. When working in configuration mode, using “run <operational command>” works similarly to IOS’ “do <operational command>” with the added bonus that tab and ? completion works. This was very helpful when i found that “run show interfaces brief” is not very brief, and “run show interfaces terse” is probably more like what i expected of briefness.
There were a number of features in the configuration process that struck me as sub-optimal, or at least unpleasantly surprising:
There are separate NTP servers during boot and normal operation, with separate configurations for each. (Presumably under the covers the former are used as arguments to ntpdate during boot whereas the latter are used to configure ntpd.) This strikes me as a rather obscure feature which probably resulted from a large government department insisting on its inclusion in a large RFP. The very strange thing about this is that only one boot NTP server is allowed, whereas multiple NTP servers are allowed during normal operation.
The Junos shell interprets the pipe character (vertical bar) in a counter-intuitive fashion: the pipe is a well-established mechanism on Unix/Linux, Windows, IOS and Comware (amongst others) that sends the output of the first command to the input of the second command, which is a filter. Junos seems to use it as a generic way to affect the display of the output of the first command, which would more regularly be done by flags or additional arguments on other operating systems (e.g. “ls -l” on Unix/Linux, “dir /b” in DOS/Windows, “show ip interfaces brief” on IOS). This is another departure from the principle of least surprise, and i’m glad that Vyatta didn’t follow Junos in this respect (thus “show | compare” in Junos is simply “compare” in Vyatta).
The Junos best practice (i can’t remember which guide i read this in) is to set the IP address of lo0.0 to 127.0.0.1/32 rather than 127.0.0.1/8, as it is on Linux and Windows. This is more likely an issue when using Junos for routing rather than switching, but i still couldn’t see a reason for it.
Entering multi-line banners is annoying: one must use n within the string instead of just pressing enter and finishing the string on the next line.
VLAN setup is a little tricky. I initially tweeted: “#Juniper#EX2200-C impression #9: #Junos makes more sense as a router OS than a switch OS. VLAN setup is almost as painful as #Mikrotik.” But after spending more time setting up VLANs on Mikrotik recently, i can now formally rescind that remark. 🙂 VLANs are configured in two different ways: under the interface (or interface range) and under the VLAN definition. Juniper recommends using the former for trunk (tagged) interfaces, and the latter for access (untagged only) interfaces. Because i often play around with my VLAN assignments, i ignored this advice and defined them all as if they were trunks.
Junos includes a concept of an “edge” port, but i couldn’t find a clear definition of it in the manuals i was working with, and as soon as i tried to apply BPDU filtering using “bpdu-block-on-edge” in interface context, i discovered its inflexibility, and discarded it in favour of “set ethernet-switching-options bpdu-block interface NAME”.
Almost everything i had on my checklist to convert from my 3400cl was done at the end of my three-day lab. The exact mechanisms for each feature (on both the Juniper EX2200-C and the HP 3400cl it replaced) can be found in the config files attached to this post. A few features seemed to be missing or deficient (although this could be due to my poor research skills rather than a feature gap):
There is seemingly no equivalent of IOS’ “switchport trunk allowed vlan remove ID”. I couldn’t find a way to set up a VLAN trunk such that all VLANs except a particular list were allowed.
There’s also seemingly no equivalent of ProCurve’s “lldp admin-status PORTNUMBER RxOnly” (for allowing receive but not send of LLDP info); Julien Goodwinsuggested that ACLs could be used to fix this; i simply chose to disable LLDP entirely on my ISP uplinks.
More perplexingly still, there’s seemingly no equivalent of IOS’ “no ip routing”. Seems very strange that there is a Junos kernel option to drop all TCP packets (i can’t think why anyone would want to turn on this option, since it would prevent remote management via ssh), but no option to turn off IP routing. Workarounds suggested by the IRC crowd include:
blocking traffic between vlans by policy
using multiple routing instances and putting each VLAN in its own instance
create only L2 VLANs with no L3 interface (which is the option i chose in this case)
There’s no way to enable all interfaces for sFlow – it appears one must add each interface individually. And further, there’s no SNMP write access at all, so i couldn’t just configure authentication and leave the heavy lifting to sFlowTrend.
Probably the feature i spent the most time investigating and discussing on IRC (hat tip especially to Kurt Bales) was VLAN QoS priorities. Junos’ QoS capabilities are extensive, but configuring them is not as straightforward (some would say “simplistic”) a matter as ProCurve makes it (vlan X; qos priority N). Because my home office switch is never congested, i decided it wasn’t worth spending the time to learn this part of Junos until later.
Junos is a mostly-logical network OS with an elegant configuration system and excellent freely-available documentation. The quality of the EX2200-C hardware is first rate, and i suspect this only increases in subsequent models in the series. I’m glad i took the time to get familiar with the basics of Junos, and i’ll probably try to be certified to their entry-level standard if time permits. I don’t think Juniper’s offering is unique enough to cause me to choose them ahead of Cisco or HP (the latter’s Comware switches are still my go-to choice), but they’re certainly an excellent choice for quality managed switches, especially for those organisations with an investment in Juniper’s routing platform.
I’ve been doing a lot of work with Vyatta of late, and one of the great things about having a router based on Debian is that you can combine Cisco-/Juniper-style CLI with Linux goodies. One of my favourite Linux utilities is watch, which just runs a command repeatedly and shows you its output. So to watch a mail queue empty after you’ve fixed a downstream mail server, one might run:
watch "mailq | tail"
On Vyatta, this comes in handy if one wants to watch a route table or something similar. However, by default, the CLI will not allow you to run “show” commands directly, because they’re implemented internally by vbash, the Vyatta version of the well-known Linux/Unix shell, bash. So, for example, the following will not work:
watch "show ip ospf database"
watch "vbash -c 'show ip ospf database'"
The trick is to use the -i flag to vbash, which tells it to assume that it’s an interactive shell, like so:
watch "vbash -ic 'show ip ospf database'"
I’m not sure why Vyatta felt it necessary to require this, since the only conceivable reason one would run vbash instead of bash is to get access to the Vyatta extensions, but this is an easy and painless workaround. (I’ve also documented this at the Vyatta forum thread that talks about it, since Google still points there for a number of searches – hopefully they’ll update the links soon.)
Note also the double quotes are necessary to tell watch to run send the entire command to its own internal shell. If you have lots of $ variables and the like, this will quickly turn into quote and backslash hell, so keep it simple, or put your commands in a script file.
I discovered a bug with BackupPC’s error reporting. This has hit me more than once (evidenced by the deja vu i experienced when debugging the problem), but i mustn’t have written down the solution previously. A quick Google (by which i mean “skimming the first two screens of hits”) doesn’t show any obvious signs of people having the same issue, so i thought i’d document it here for search engine posterity.
The basic issue is that backups to certain systems fail, and the diagnostics shown in the web interface look like this:
Last status is state “idle” (no ping) as of 11/2 14:00.
Last error is “no ping response”.
Pings to laptop1 have failed 39 consecutive times.
However, no ping response is not the problem. If i login as the backuppc user on my backup server, it is able to both ping and ssh to the host in question just fine.
Digging deeper in the logs, i found this in /var/lib/backupppc/log/LOG:
2012-11-02 14:52:05 laptop1: mkdir /var/lib/backuppc/trash: Permission denied at /usr/share/backuppc/lib/BackupPC/Lib.pm line 629
I fixed this by chowning /var/lib/backuppc to the backuppc user, and the backup proceeded as normal. So it seems that backuppc will not do the right thing without a trash directory in place, and if it doesn’t have permissions to create it, it gives a misleading error message.
In my case, this happened because my removable drive for backups died and i replaced it with a new one without fully recreating the directory structure as required by backuppc. So i guess it’s my fault, but a more helpful error message would have been good.