Chombo's Blog

I’m not going to beat around the bush here: what’s with the constant bickering between web hosts? It seems that choosing a web host is like choosing a religion – what gives guys? We’re all friends here.

We should all throw a big web-host party and have a few drinks together – like the crew from AusNOG do! We have such a ball every Tuesday and foster an attitude of comradeship. I often find myself being helped by competitors and helping them in return. This is how things are supposed to be!

If you own a hosting company and want to meet up with the Chombo crew in Brisbane, flip us an email. Your people call my people and we’ll do beer.

Imagine a computer room air conditioning system that switches between cooling types based on environmental conditions, using ambient air when wet-bulb temperature is below a predefined threshold, substantially reducing the amount of electricity used to cool the datacentre facility – well, I imagined exactly that a few months ago and our datacentre greening strategy is well on the way to implementation (well, after we iron out a few little bugs and I stop electrocuting myself!).

There are probably a few of you asking yourselves why on earth I’d build something custom when there are many off-the-shelf environmental monitoring and free cooling packages commercially available. The answer to this is twofold: cost and flexibility. Our datacentre is unique in its construction and is specifically designed for medium density content delivery and telecommunications infrastructure for SONET’s ISP business. Whilst most off the shelf systems are customizable enough to allow for equipment of this type, I wanted more granular control over triggers for cooling state changes.

The system is based on a few discreet components:

1. Power control for the evaporation coolers
2. Infrared logic control for the refrigerated air conditioners
3. Temperature and humidity sensors for environmental monitoring (both inside and outside)
4. Embedded Linux controller system to perform the business logic for the application, as well as providing environmental monitoring feedback to the network operations centre.

I’ll give a little insight into how the whole shebang is supposed to work.

AC Power Control

Being a relative newcomer to AC power control (I am not an electrical engineer and my experience with power systems so far has been predominantly DC for rural power systems), power control for the evaporative cooler remains a challenge. There are a few complexities with AC power control – especially with reactive loads – which I am still grasping – thankfully I’ve got an electrical engineer friend who has been helping me sort through the technicalities of our implementation.

I settled for a Braemar evaportative cooler and was especially diligent to ensure that the unit was single phase, as timing TRIAC states between multiple phases is a complex art, requiring high accuracy oscillator circuits to ensure synchronization with the grid. I am settling on using thyristors because of their relative ease of implementation for single phase circuits and their lack of requirement for a snubber circuit for reactive loads (I think – I’m sure an engineer reader can provide a more in-depth description of what exactly goes on here).

I am controlling gating using at ATMEL micro-controller and DAC interfaced by RS232 to a Linux controller box which is responsible for controlling cooling states and passing environmental information back to the NOC. This also gives us the flexibility to pass information in whatever format we like (we are not reliant on SNMP). Whilst this isn’t in production at the moment, when it does we will most likely use SOAP to talk directly to our status display panels in the NOC and in my home office – information will also be saved into a MySQL database for trends analysis, which will allow us to make more informed choices in regards to continuous improvement strategies.

One of the other benefits of this setup is the ability to change gating frequency – this means we can change the air volume based on load and cooling requirements in the facility.

Controlling The Aircons

We’re using commercial splits to cool our facility. Before you get on your high horse, explaining the importance of using properly sized CRACs for load, consider the both the unusually large amounts of control we have over the compressors as well as the amount of time invested in analyzing and sizing the exact requirements to provide adequite cooling to the facility.

Ironically, despite the closed architecture of most split systems, these are actually the easiest of all the components to control. Each setting on the remote control is a separate infrared command, making the control logic for these devices exceptionally easy. Because of the use of our own independent environmental monitoring, we have no need (or desire) to read state information from the air conditioners.

We are using IR windows to pass infrared commands to the splits to change their states based on requirements.

Environmental Monitoring

Did you know they make temperature/humidity sensors with RS232 output? We didn’t until we started using them, and they are fantastic. Reading state information is as easy as logging state updates from the tty into a database and reading them out as required. This also gives us the ability to read trend reports on temperature settings and make changes to our setup as required.

I’ll be monitoring temperature and humidity both inside and outside the datacentre facility, which (with some maths-fu) allows us to determine the potential wet-bulb temperature of the evaporative coolers and activate the appropriate cooling type based on the environmental conditions.

We’re somewhat unlucky in Brisbane, as the humid climate does not allow for evaporative cooling the predominance of the year. Fortunately, our ambient temperatures are some of the most predicable in Australia which allows us to take some liberties with our evaporative strategy.

There are many times where ambient temperature is low enough to cool the facility, but humidity is far too high to allow full evaporative cooling of the facility. By passing commands to the refrigerated A/Cs, we can get them to perform dehumidification without any refrigeration (in conjunction with the evap coolers) for a fraction of the power usage of just running refrigerated A/C (something currently unachieveable with any datacentre environmental control system I have come across). Our use of hot aisle containment allows us to get away with this.

Controlling it All

Debian is God’s gift to rapid deployment. That’s all that needs to be said here. Its flexibility regarding reading from/writing to tty’s is what has allowed us to develop a control system of this complexity so rapidly.

Extensive use of PHP/shell scripting helps too, as well as MySQL. While it may seem counter-intuitive to use a scripting language for this, rapid time-to-market is more important than processor cycles in this regard, and it is no doubt cheaper just to throw more processing power at the problem than it is to optimize. If I was to sell this as a product I’d probably take a more serious approach to this but at the moment it’s purely an R&D experiment.

Future Improvements

Where to go from here – well, a good start will be implementing CT metering for power, so we can get feedback on exactly how much power it takes to cool the facility. This will allow us to make decisions within the business logic regarding how we value efficiency vs. hot/cold aisle temperature and give us some REAL metrics to give when making decisions to cut carbon emissions.

This will also allow us to make proactive changes to cooling during power outages: the evaporative cooler uses so little power that it is feasable to run it off the UPSes. During outages, this means we can sacrifice cold aisle temperature for increased generator/UPS runtime, proactively avoiding power outages within the facility.

There are also obvious improvements, like introducing redundancy for monitoring systems and separating the business logic from the software itself (at the moment a lot of this is contained in hotchpotch PHP scripts, cron jobs and other nasties – this is TRULY in beta). We’ll also need to develop some reports (probably in Crystal Reports) to allow us to get near real-time feedback on our greening strategies and give us highly granular control over our cooling and power usage.

So, we ran out of capacity in our datacentre today. It’s not that we don’t have enough power, or space – our growth over the past few months (both Chombo and SONET) has meant that we’ve actually run out of carriage to our datacentre!

Our DC is served by an Optus SDH feed, terminated on an Alcatel-Lucent STM-1 ethernet demarcation unit. We’re lucky – because the data is carried via an STM-16 and then demuxed into STM-1 for us, we get the protection of Optus’ optical ring and microwave transmission path between  Rochedale and the city, without having to pay for additional fibre into the building.

Now, as good as our STM-1 is (we’ve never had an outage), we’re not satisfied by having “all our eggs in one basket”. Whilst SDH is venerable, a ring failure could prove catastrophic for us. Because of our location, the only other optical carrier in the area is Telstra – whilst we have no issues with Telstra Ethernet/IP services (we’ve found the quality and price of their products to be fantastic), the cable paths for their ethernet services are exactly the same as Optus – which subjects us to potential backhoe attenution (ie, idiots with shovels).

As most of you probably know, Chombo’s parent entity, SONET, owns substantial microwave infrastructure throughout Brisbane. We’ve been using this as a customer access network, to deliver advanced services such as IaaS, VoIP, WAN and Internet via our own microwave network – we call this wirelessIP. We’re currently putting infrastructure in place which will allow us to double our capacity to our datacentre and double our backhaul and resiliency for our wireless customers – as well as installing some long-haul radio routes to get to the Sunshine Coast and Lismore.

Our radio network is not as direct as fibre, but provides us with an independently controlled, resilient network whose infrastructure we own entirely – this allows us to continue to service our constantly growing client base and dramatically reduce our operational expenditure by making intelligent, informed decisions on strategic capital expenditure.

Sometimes it’s worthwhile to take your time and enjoy the scenic route

Optus' Fibre Termination

There's an E1 in the building which provides clock for this archaic old thing.

Lots of fibre! We terminate managed WAN and an OC-48 ring generator in here too.

Our lowly server room/datacentre facility. It's nice and cool in summer.

The views from the top of the Cootha repeater site are amazing!

It's a long way down!

10KM away and 600m above the Brisbane CBD is the frenzel zone of the new backhaul link, providing hundreds of megabits of protected ethernet trunking.

Starting a business is a lot like starting a family. Months of speculation as to whether it’s a “good idea” are proceeded by years of hard work, late nights and ridiculous amounts of risk – but you do it anyway, because ultimately, the reward far outweighs the temporary inconvenience. At least in theory.

As the father to a 14-month-old daughter, I find myself constantly drawing parallels between my role as a father and my role as the manager of Chombo/SONET. There is no doubt that there is no greater joy than watching your child achieve milestones and slowly grow into their own autonomous entity – the same can be said for my business ventures.

Every parent wants the best for their children. For a plethora of different reasons (which I will discuss in a later blog post), earlier this year Andrew and I made an executive decision to sell Chombo to Refresh Domains Pty Ltd. We approached Refresh, as we had close ties with their Managing Director through relationships with both SONET and personally with myself. Their approach seemed sound and it appeared that our goals and visions for the growth of Chombo were significantly aligned – it was my honest belief that the cash investment Refresh were planning on making, combined with their supposed long term vision would yield the best outcome for Chombo’s customers.

As operational control of the business was handed over, the differences between the approaches of Refresh and SONET became evermore apparent – many of these are exemplified in some of the posts which Michael made on this blog earlier this year. Whilst I have always strongly believed in organic growth and conservative investment (especially in such a fragile economy), Refresh focused on growth acceleration, including the investment of significant amounts of money in both web and radio advertising and the introduction of telephone support and unsustainable pricing: this kind of “pump” methodology has been something we’ve always attempted to avoid, as it was never our intention to sell Chombo. The construction of our purpose-built datacentre facility and continued re-investment of profit into the business is a testament to our ongoing desire for continuous improvement.

There is no denying that the “pump” strategy is a fantastic way to build volume. The thought of building the business for fast sale is very alluring, especially when considering the potential returns on a short-term investment, but this does not change the fact that there is no way to sustainably offer the services we have offered previously, with the level of service our customers are used to: ultimately, our customers are the reason we exist and a failure to show continued respect for their trust and patronage is to invite doubt into their minds as to our intentions.

Within two months, Chombo came dangerously close to insolvency. We used our contractual exit clause to regain control of the assets of the business and have managed to successfully halt and reverse the debt spiral for which it was destined.

Whilst there is no denying that the past few months have been challenging for all involved, we’ve reconciled our position, hired more staff and made continued investment into the business – our weekly reconciliations look healthy once again and we’ll turn a profit by the end of this month (not including the amortization of the bail-out – that’ll be resolved by the end of October without accounting for growth).

Earnest Hemingway quoted: “never confuse movement with action”. We’re moving on, collecting the pieces and quietly expanding our internal business systems and exposing more of our business process to our customers. We have learned a valuable lesson from this experience – one we will never make again.

I’d like to take one more moment to thank everyone for your unwavering support during these turbulent times – as always, your understanding and patronage remains an inspiration to myself and my team and for that we thank you.

Please welcome the newest member of our team, we call him Robbie. You can speak to Robbie by calling us on 1300 678 699, and he’ll automatically answer your call 24 hours a day, 7 days a week. He doesn’t get much time off, mainly because he’s our new IVR recorded by superstar Robbie McGregor.

We recently had our IVR set up as part of our new Customer Service model aimed at increasing the level of service provided to customers and building on the experience people have had with Chombo in the past with our email support. We decided things should be awesome from the moment the call is answered, so we decided to have the calls answered by someone who would be recognisable to everyone (even if you don’t know him by name!).

When you call you’ll be presented with varying options depending on when you call. If you call during our trading hours of 8.30am to 5.30pm AEDT, Monday to Friday you’ll be presented with an options for Sales, Support, and Accounts. If you call after hours, you’ll be told you’re being put through to voicemail but given the option to Press 1 if you need urgent Technical Support. This is only for critical or urgent technical queries, if it’s not urgent please just send through an email.

The best part of the whole thing, is that you’ll never wait on hold for more than 3 minutes. We don’t believe we should waste your day with terrible music. If IVR works out that you’d wait for more than 3 minutes, you’re redirected to voicemail and you retain your position in the queue. If you’re really in a hurry, as soon as you get put on hold you can just press 1 and be put through to voicemail as well (I’d suggest waiting 10 seconds first in case we aren’t busy).

Hopefully this makes everyone’s lives a little easier and just a little bit cooler as well, now that we’ve got Robbie on board.

The transition from SONET’s Chombo to my Chombo has been difficult in a few different ways. It’s been difficult for me because running a hosting company is not something I’ve done before (I think this is a good thing, I’ll explain why in a minute), and because the staff we have also have to make adjustments to cope with a new style of management. Curtis and I both manage companies in different ways. For example, I like to always look for ways to improve what we do – I feel this is essential for both us as a company and for our customers who are always going to be in search of something better, and so they should be. The key is just to ensure you’re always improving and your own service is that  “something better”.

The reason I said above that it’s a benefit I haven’t run a hosting company before, is because while I’m not the most superb technical person in the world, I can surround myself with people who are excellent at what they do. It means they wont go cutting corners for the sake of a few dollars – I have to trust their judgments that what they’re doing is best for the customers and let them run the service at a top notch level while I handle the business side of things and a certain amount of customer support. I’ve always liked interacting with customers, and I guarantee if I’m ever a millionaire I’ll still be out there taking phone calls and replying to emails. Having the slight detachment from the actual technical side of things ensures our customers don’t have things “messed up” by someone trying to save a few bucks for their business.

Andrew Gould is our General Manager, and he’s the one making the decisions to do with technical solutions and improvements, plus he’ll take on board any suggestions I throw his way (turns out I occasionally have a good idea). More and more as time goes on and we get used to each other and our operating styles, Andrew will get more and more control over how the business operates and what the best result for our customers is. He’ll shape the way our customer service works (as he always has, it’s what’s made our customers so happy). If we keep the customers happy and don’t get greedy, the business will look after itself. I have no doubts about that at all. While we do agree with “if it aint broke, don’t fix it”, there’s no point falling behind and having things become outdated and stale. Our customers requirements will evolve and so must we to stay successful and useful. If Virgin Music was still selling LP’s, they’d be dead and gone 20 years ago. You have to move with the customer.

More than anything, the learning curve for all of us at Chombo isn’t “how will the new boss want us to do things?” it’s “how does the customer want us to do things?’.

Michael

Some of our customers are probably wondering what our “new position” is on things since we “passed the baton” so to speak. So let’s look over something that was written by Curtis when he was Managing Director and my views on it:

“The reliance on promotions to generate new leads is not congruent with the type of business that we are attempting to build. Don’t get me wrong – I am not suggesting that the use of giveaways and “90% off if you buy within the next two days!” promotions don’t get bums on seats – admittedly, if we were building a business to sell then we would have spent the predominance of our budget on advertising and ridiculous “prizes” rather than network infrastructure and ongoing operational-expenditure support.”

Curtis had a very strong view on this, and I do too. We need to have a purpose for running any promotion or giveaway, its primary function must be one that is beneficial to the business and not one made simply to “pump up the numbers”. Chombo is in a position that it does not require growth at extreme rates, but recently as most people are aware I made the decision to cut the price of our entry level $9.95 plan to just $3.95, as an ongoing price to the next 500 signups. Immediately, this made our competitors speculate on why we were doing it, and what purpose it served. Well, despite some of the more colourful ideas of why, we did indeed have a valid and reasonable business case for doing so. Chombo’s strength is undoubtedly its customer service and free value-adds in terms of the amount of inclusions. Lately, this has been expanded to include 8.30am to 5.30pm Monday-Friday Telephone Support, at no charge (some providers choose to charge an additional fee for this kind of thing).

The problem with this model is that it’s inevitably going to cause a higher entry price as it has higher running costs. That’s no secret. What I decided to do is show people who aren’t with Chombo what they’re missing out on by not being with us, and offer them a price level that they could sign up without feeling they were paying more than the most budget of providers sells for, despite the additional services we offer. So we did just that, we cut a whopping 60% off our plan price, and launched our free telephone support. This will get people on board so we can prove to them exactly what it is that we do better than the rest, and why it’s not always the best option to sign up just based on price, yet many people still do. Inevitably, our prices will go back up by some amount at some stage (except to the 500 people who take advantage of this promotion, it’s an ongoing price for their accounts), but people should start to realise exactly what they’re getting for that extra few bucks a month – a lot more than a few bucks worth!

“Maybe we’re doing it wrong. I’d like to know your thoughts.”

I’ll echo this, tell us what you think in the comments or email michael@chombo.com.au – I’m very interested in what people think of the way we do business.

Michael

Sorry for the delay in updates – it’s been a hectic few months and the following post will most likely provide some insight as to why.

How do you define a value added proposition in an industry whereby there is little, if anything to differentiate between you and your competitors? Sure, the ridiculous data allowances helped (but appear to have been now mimicked by a number of our competitors with little regard for their sustainability and with seeming disregard for the business model associated with offering such services) and advertising “exceptional service” is a misnomer as the phrase is usually a standard affair on any web template you can download from TemplateMonster (with all due respect to many of the talented designers which cater to this market).

That was a freight train of a sentence, but I digress;

Staying ahead in this industry is hard. With so much diversity (read: competition) there’s little to discourage your competitors from copying pretty much everything you do. So what’s tiny, two-man-one-woman operation to do? Leverage any and all advantages you have to drive your costs down so you can leverage unencumbered operational expenditure to repurpose as capital expenditure.

In other words, we built a datacentre.

But wait Curtis, aren’t datacentres ridiculously expensive to build, maintain and administer? They sure are, if you’re doing it wrong.

The traditional model of co-location implies significant inefficiency and overheads in the hope the telehouse vendor can service the most number of customers: in order to leverage your investment you need to cater for the lowest common denominator. The beauty of building a facility tailored specifically to the requirements of your business is that you are able to strip out un-required and unnecessary infrastructure used to service requirements that your business just doesn’t need – and tailor those you do specifically to your requirements.

There’s a smug sense of self-satisfaction knowing that you can service a full cabinet full of dual-core web head-ends for less than $500/month in electrical and mechanical costs (for the purists out there, yes that does include chillier load). Connectivity costs real money of course, but short of building our own national network, that’s something we just can’t avoid. I’ll leave the wasting that kind of money to the Australian taxpayer.

What does that mean for our customers? Let’s sum it up in a few bullet points:

• We now have PoPs in three highly available datacetres/carrier hotels across Brisbane, one of them our own.
• We have have the potential to reduce our overheads per server to approximately 40% of what it was when we initially began offering services.
• We have an independent location from which to deliver services, away from any reliance on a particular carrier or location.
• The clean room of our datacentre is nice and cool, so we won’t have to swelter this summer.

With these changes, our service offerings are now even more profitable than they were six months ago. Will we reduce prices? Probably not, we’ll use this money to bring new products to market and seek to continuously improve the service offerings that we provide currently.

Onwards and upwards – I have no doubts that next year will be a year of significant growth, and we have nobody else to thank for this but you.

Curtis