(Ping! Zine Issue 6) – Ever rented a car from Avis or Hertz? You know how their service works: the agent swipes your credit card and hands over keys and perhaps a map. There’s no tuxedoed chauffer to drive you around town, no instructor to show you how to operate the vehicle, and no guard to help secure your belongings. You wouldn’t complain, because you know and they know that such services don’t belong in car rental packages. What’s more, you wouldn’t dream of blaming your agent if you took the wrong highway exit, were ticketed for speeding, or fell victim to theft because you forgot to lock the doors.
In my mind, the dedicated hosting and car rental businesses are very much alike. The vendor provides the equipment, while the customer is free to use it as he wishes – within legal limits. The main difference is, the driver licensing system assures Avis of its users’ competency. In contrast, any man, woman or child with a credit card could instantly become a proud server administrator, and there begins the headaches for their hosting providers.
I have never visited Web Hosting Talk’s dedicated server forum without coming across headlines that scream “Terrible Service from Company X!” More often than not, such complaints are related to an unmanaged server provider’s refusal to troubleshoot unsupported software or hand-hold the customer through routine sysadmin tasks. The latest thread I followed involved a user complaining about his unplugged server; it had been repeatedly compromised and was sending out denial of service attacks.
“I can’t afford this downtime that they’re making me pay for. I keep getting hacked and they’re not doing a thing about it. I’ve been disconnected three times in two weeks which makes me very uncomfortable using this service any longer.”
Just think: if you left your rental car’s ignition on and doors wide open, would you or Avis be to blame if someone else drove it – along with your belongings – into the sunset? You would have absolutely no claim to reduced rental fees. What’s more, you – or your insurance provider, let’s hope – would be held responsible for repairing or replacing the vehicle.
At a time where ever faster servers are widely available at rock bottom prices and with increasingly generous bandwidth allowances, it’s important that you, the user, recognize these tempting offers for what they are. In the words of Peter Abraham, CEO of WeManageServers.com, “unmanaged hosting means you are 100% responsible.” If your in-house staff does not include personnel with appropriate sysadmin skills – described as follows by three industry veterans – the sweet hosting deals you see are unlikely to lead to the profitability or convenience you envisioned. So know your limits – and seek help if necessary – before you sign up. Good luck!
Needs versus Skills
By Haralds Jass and Landon Stewart, Superb Internet
The key to a positive hosting experience is ensuring that there’s a match between your level of competency and the tasks you hope to accomplish. Let’s use reseller hosting as an example.
You’re using the server to host a single website, which you’ll administer through a control panel without ever logging in as root, installing custom applications or creating other users.
You should be familiar with the use of FTP, your control panel of choice, and basic shell commands for setting file permissions. You should also have basic scripting knowledge for any CGIs you plan to use. Of course, very few users fall into this category. You most likely don’t need a dedicated server for this level of simple use.
You would like to set up a simple reseller operation
You should have at least a general understanding of the process behind each service you offer, including DNS, virtual hosting, permission-based file systems, system security, and how email is conveyed between networks and mail servers. You may need to contract some external help on an as-needed basis, to secure and update/patch the server, for instance. In the event any problems arise, you should have enough technical background to communicate intelligently with your hosting provider’s support staff. They have a lot of knowledge, which won’t do much good if you can’t clearly describe the tasks you’ve performed and the errors you’ve received.
You’re running a growing reseller hosting business
You should have in-depth knowledge of how your operating system of choice runs, and how to troubleshoot common OS problems. You should be familiar with various daemons and their configuration files. You should be able to examine system logs, monitor system load and update/patch the system. This of course requires knowledge of where and how to obtain updates.
Ideally, all dedicated server users should have Level 3 knowledge. Too often, unfortunately, we see Level 1 or Level 2 users try to accomplish Level 3 tasks. This becomes a particularly serious problem when a live machine is used as a testing ground for the server owner’s sysadmin skills.
One solution is to contract with your service provider, or an external party, to assist with server management. Another is to get two servers: a good production machine and a low-end testing box. The latter should be used for experimenting with new packages, configuration options for critical daemons, etc, before any changes are applied to the live server. It would be a good investment, in terms of both education and ensuring the stability of the production server.
The Sysadmin Checklist
By Scott Webb, EasyServerManagement.com
A good sysadmin has the willingness to learn, research and “read the manual”. He or she should also be comfortable with performing the following:
Turn off telnet and unused/insecure services
Login via SSH
Navigate the file system in shell (cd /etc/rc.d/ini.d/; cd ../../; etc)
Use basic shell commands (ls, tail, mkdir, rm, mv, cp,etc)
Install a firewall
Adjust the firewall and iptables rules
Compile and install basic programs
tar and untar archives/programs
Restart services from command line.
Monitoring of logs for unusual activity
Upgrade Control Panel as patches are released
Upgrade of OS level programs/rpms
Monitor load and services
Restart services as needed
Of course, the most important sysadmin task is to fight the never ending battle of making sure that the server is secure and up to date.
Questions and Answers about Server Security
By Peter Abraham, WeManageServers.com
Q: I’ve just signed up for a server; is it a brand new machine?
A: Hosting providers typically recycle servers when customers leave. It’s quite possible that you will be given a used machine which may or may not have been compromised in the past.
Q: What about IP addresses? Am I getting brand new IPs?
A: IP addresses are typically recycled as well. This means it is possible that the IPs assigned to your server or mail server may be in a spam database. As a result, your mail could be blocked until you work with the databases (there are hundreds) to get unlisted.
Q: Since the server has just been put online, immediate attacks are unlikely – right?
A: Dead wrong. Take a look at this report from http://www.honeynet.org/papers/stats/:
Between April and December 2000, seven default installations of Red Hat 6.2 servers were attacked within three days of connecting to the Internet. Based on this, we estimate the life expectancy of a default installation of Red Hat 6.2 server to be less then 72 hours. The last time we attempted to confirm this, the system was compromised in less than eight hours. The fastest time ever for a system to be compromised was 15 minutes. This means the system was scanned, probed, and exploited within 15 minutes of connecting to the Internet. Coincidentally, this was the first honey pot we ever setup, in March of 1999.
In case you’re wondering, the situation has not improved since. There have been reports as recent as February 2004 in Web Hosting Talk (http://www.webhostingtalk.com/), the forums for EV1Servers (http://forum.ev1servers.net/), and other hosting-related forums of servers being hacked the same day they were put online.
Q: I’m nobody in the global marketplace; why would anyone want to hack me?
A: Based on statistics over the past 8 years, up to two-thirds of all hacker activity has been random. Think of the Internet as a parking lot, with hackers walking around and checking on each car door to see whether they are locked or can be easily opened. The following is stated on http://project.honeynet.org/papers/enemy/ —
The script kiddie is someone looking for the easy kill. They are not out for specific information or targeting a specific company. Their goal is to gain root the easiest way possible. They do this by focusing on a small number of exploits, and then searching the entire Internet for that exploit. Sooner or later they find someone vulnerable. It is this random selection of targets that make the script kiddie such a dangerous threat. Sooner or later your systems and networks will be probed, you cannot hide from them. I know of admins who were amazed to have their systems scanned when they had been up for only two days, and no one knew about them. There is nothing amazing here. Most likely, their systems were scanned by a script kiddie who happened to be sweeping that network block.
Q: There was only one Einstein. It must take some one with a high IQ, great computer expertise beyond compare, etc. to be a hacker. There aren’t too many of those types of people running around. They have other targets to go after – right?
A: Wrong. There are software tools which provide anyone (from your great grandmother to your seven year old child) with the ability to use a mouse to break into systems in less than five minutes. They are browser-based or applications that run on Windows. And should the hacker have problems, there are on-line help sites which give step-by-step instructions. This means the potential for hackers beyond counting.
Q: Can I manage server security tasks through my control panel?
A: Don’t count on it. Your control panel will not harden the operating system by itself. It also won’t review your system log files, tell you how to read the logs, or provide instructions on what to do with the log file info. While some control panels do include the ability to update themselves, they often lack the ability to update applications outside of their control.
Q: I have a firewall. Does that mean my server is secure?
A: While a firewall increases your protection, security needs to be done in layers. A firewall is very similar to guards at the borders between countries. If you have the right paperwork, you get through the border gate. Does that mean that countries that check for the right paperwork never ever have a spy enter the country through the border? Firewalls allow valid traffic (AKA a person or party with the right paperwork) through it (the firewall) to the destination (a server in this case). Most hackers work by using already published, tested ways to attack by using valid traffic. If your system is not up-to-date, and you do not have a number of security layers, you are not secure from hackers. To be clear, any system connected to a network can be compromised. However, the more appropriate, well thought out layers of security, the harder to compromise, and the more likely the attacker will seek an easier target.
Q: Shouldn’t it be my hosting provider’s job to worry about hackers?
A: Remember, “unmanaged” means you are 100% responsible.
Q: Shouldn’t they provide security patches and updates?
A: Remember, “unmanaged” means you are 100% responsible.
Q: What if I’m working with a managed hosting provider?
A: Many companies state that they provide “managed hosting”, or even “fully managed hosting”. But we are unaware of any company that provides true, full, proactive, managed security – to the extent that you can focus 100% on running your business. For example, in almost all cases, managed hosting providers apply security patches only upon your request. This means you yourself must be familiar with the availability and appropriateness of any patches. Most service providers also would not review log files on an ongoing basis.
Q: What are the daily routines of managing a server?
A: While the time involved will vary based on skill, the number of servers, the operating systems involved, and the applications running, the following are the general day-to-day tasks:
1. One to several times a day, review server log files (quantity varies by operating system and applications, but there can be dozens) for error messages, hacker-like activity, etc.
2. Update security measures based on review of the log files.
3. Fix any errors noted in the log files.
4. Review various security announcements for operating system and applications which must be updated for security purposes.
5. Apply security updates as needed.
6. Monitor the operating system, applications, and servers to ensure they are up as close to 100% of the time as possible.
7. Take appropriate action based on monitoring results.
There are a number of ways to automate the daily routine; and the methods will vary by operating system and application. However, the investment of time is still daily to one degree or another. For instance, Logwatch from http://www2.logwatch.org:81/ can summarize many (but not all automatically) of your log files on one server into one email message; and, you can control the detail and frequency the program runs. However, you still have to be able to understand the messages reported, take the time to read them, and take the time to act upon the results.
Know Your Enemy: The Tools and Methodologies of the Script Kiddie
Know Your Enemy: II Tracking the blackhat’s moves
Know Your Enemy: III They Gain Root
Know Your Enemy: Statistics: Analyzing the past … predicting the future
Days of the Honeynet: Attacks, Tools, Incidents
U.S. funds study of tech monocultures (tidbit – “Massive digital epidemics–such as the Code Red, Slammer and MSBlast worms–have infected hundreds of thousands of computer systems, leaving scientists to wonder if worse is in store for the Internet.”
Feeling Vulnerable?: If you’re bedeviled by swarms of vulnerability alerts, you can take control by practicing good management.
The Attacker’s Arsenal