The reality, unfortunately, is much darker. Instead of using the RMF, and its accompanying standards as the framework they are intended to be, they are instead generally treated as a veritable gospel that can never be questioned, thought about, reasoned about, or otherwise adapted to the situation at hand. This creates a situation that inverts the incentives and often creates systems that have much lower actual security, and substantially increased risk, but have lots of paperwork to get the approval.

Now that I’ve taken some organizations through the FedRAMP process, I can say that it is an improvement. It is more risk-focused, and more interested in being a collaborative effort to actually identify risk and address them wherever possible. It still suffers from a serious paperwork overhead, however, and worse, there are some conceptual gaps within it that do not address the neads of large cloud providers. I’m going to try and address some of the ideas that I think need to be tackled within FedRAMP to succeed with the likes of Amazon, Windows Azure, Google, Rackspace, etc. Without these changes, or something more effective even, I believe that FedRAMP, for all its admirable goals, will wither and die.

In follow-on articles, I’m going to cover some ideas for how to scale FedRAMP to both larger and, where I can, smaller cloud service providers. Most of my experience is with the gorillas in the yard, so it will focus on that, but I’d like to see it made more flexible for the organizations just starting, especially when they’re in the SaaS/PaaS space.

The topics I intend to cover are:

• Issues of technical scale – How do you scale FedRAMP to deal with the issues faced by the likes of Amazon, Google, Microsoft, Rackspace, et. al.?
• DevOps v. The Paperwork Monster – Specifically how do you deal with the enormous velocity exhibited by most cloud providers? The RMF isn’t really used to coping with this rate of change. No calculus, I promise.
• Importance of automation – Spot checks of technical controls are fine, but the key is all inside the automation, which is rarely covered by many 3PAO.

If anyone has any other ideas they’d like to see addressed, I’d be happy to delve into them. Just as a note for my qualifications, I’m the quality manager for a major 3PAO, and technical lead on large-scale cloud projects.

The fact that conspiracy theories are percolating up to local party leaders and even the halls of Congress should be a warning sign for the GOP. As the faithful know, you reap what you sow, and the steady diet of hyperpartisan media has seeded these conspiracy theories in the minds of party activists to the extent that they are starting to shape policy debates. The embarrassing incidents are evidence of a larger problem that needs to be confronted: when you do not condemn the use of hate and fear to serve as a recruiting tool against your political opponents, the ability to reason together is undermined and self-government is compromised. There is a cost to condoning extremism when it seems to benefit “your team.”

Even The Economist has weighed in on the lunacy of the issue.

As much as it pains me to say, I think the Internet has amplified the crazy by allowing the lunatics among us to connect with each other and reinforce their belief system. At some level, it is the basic “walled garden” cognitive bias, where people associate only with people who reinforce their belief system, thereby removing all doubt and thought from the process. The Internet has simply allowed us to define our garden even more narrowly than ever before, and while this has benefited small groups that often felt isolated – for example, people with a rare disease – it has also increased the “echo chamber” effect. From outside the wall, the system seems obviously broken, but from within, all is in harmony because any element that might undermine that harmony has been removed.

Combine the dominant confirmation biases exhibited by most people – and part of the reason people misunderstand science – with things like the gambler’s fallacy and herd instinct, and you have an environment ripe for exploitation. 50 years ago, the lunatics were yelling on the street corner, and were largely ignored by society. The Internet gave them the ability to find their own “kind”, and reinforce their beliefs, thereby creating a more monolithic social structure that can be used to garner perceived rationality.

Or at least, that’s my view.

So, after six months with the car, I decided that it’s about time to put together my thoughts. The good and the bad, though the good far outweigh the bad.

The Bad

I have to start with the dreadful MyFord Touch system, “designed” by Microsoft. It’s not the worst user interface I’ve ever seen, but it is slow, sometimes oddly unresponsive, and periodically requires some kind of maintenance reboot, sometimes in the middle of a drive. Enough has been written about the system that I don’t need to go into it more here; suffice it to say that it’s not Microsoft’s finest hour. Fortunately, as a small redeemly grace, there has been a code release that was downloadable, and I was able to upgrade the software. The upgrade increased performance a little bit, and generally made the system more stable. Unfortunately, the system – even though it’s capable of connecting to a WiFi network in your house – is unable to download its own updates directly, and instead you have to download them, stick them on a USB drive and sit in the car for the entire time with the engine running while it does an update, which can take an hour.

The next three issues are all tied together. Rear-facing visibility is sub-optimal. It’s certainly worse than my Mazda 3, and comes with some serious blind spots. Unfortunately, unlike the Titanium model of the Focus, you can’t get the ST with either ultrasonic parking sensors, or a back-up camera. This is a gigantic, glaring oversight.

The last big issue is the dealer. No matter how much the domestic car makes improve their product – and they have made gigantic strides – the dealers are still sub-par at best. They fail on sales; they fail on service; and they fail in general communication.

The Good

As I said, the good far outweight the bad for me. For the good, I’m going to just hit them in bullet points:

• Handling is Teutonic. The handling is more like that of a BMW than anything you’ve ever run into with a domestic car before. The limits are very high, with the car pulling close to 1G on the skid pad. That’s serious sports-car territory.
• Oversteer. Yes, Virginia, you can hang the tail out of a front-wheel-drive car. The tuning of the stability system, and the chassis as a whole mean that you can create lift-off oversteer on demand if you put the suspension in sport mode.
• Brakes are excellent. They’re not fancy Brembo brakes, but even a day in the mountains doesn’t cause any fade, and the initial bite is excellent. Pedal feel is outstanding. They are comparable to my Infinti G35 w/Brembo brakes.
• Engine is a locomotive. 270ft-lbs of torque from just off idle, and it just pulls. The performance curve is not that of a turbo-charged 4 cylinder, but more like a straight 6, or even a nice old-school V8. There is minimal turbo lag.
• Sounds great. Unlike most “great sounding cars”, though, it’s all intake noise coming through a special Sound Symposer system. It’s a butterfly valve that lets some of the intake noise into the cabin, but only when you step on it. Otherwise, nothing.
• Sleeper and a car for hooning around. I don’t know how else to describe it other than Jekkyl and Hyde. Keep your foot out of the “go fast” pedal, and the car is a quiet, well mannered, cruiser. Step on it, and you can do all sorts of things that your parents would never approve of. For me, this is the greatest asset of the car.

For a couple years, I used it extensively and took thousands of photos, but soon it became a bit of a hassle to keep with me. The DSLR is a bulky camera structure, and when you go to take a picture, it can intimidate your subject if you’re trying to take a picture of people. It’s a “serious camera”, and comes with all the problems associated with it.

So, a couple years later, I bought a Canon Powershot S90. While it might look like a simple point-and-shoot, but it was the first of a new generation of serious small cameras. It had good glass (f/2 at its fastest), a lower resolution sensor that had very low noise levels and good low-light performance, and the ability to operate in full manual with the creation of RAW images. The camera was pocketable in pants, and I took this camera with me all over the world, taking some amazing pictures with it. There’s 16x20 enlargements in my house from this camera that people have commented on repeatedly. Just amazing.

But, it’s time for somethng “new”, and when a good friend bought a Fujifilm X-Pro1, and started showing me the amazing work he was getting out of it, I started looking at the Fujifilm line of cameras. One of the things I learned was that I actually mainly used a 35-50mm effective focal length on my lenses. There is something freeing about not worrying about the lens. My Nikon D50 has had a f/1.8 35mm lens (50mm effective) on it most of its life, and the zooms stay in the bag.

Then, miracle of miracles, Fujifilm announced the X100S, an update to one of the most well reguarded, and troubled camera released. After a few days of research and thought, I decided to buy one from my favorite camera shop in NYC (B&H), and now the wait comes. The hardest part is seeing reviews of the new X100S start to be released, and how amazing the camera is looking.

Now, where’s mine?

It is perhaps the single most over-engineered flashlight in existance. The body is CNC machined aircraft-grade aluminum that’s been hard annodized. But, that’s not what makes it the most over-engineered flashlight, this is:

Yes, that’s a complete circuit board with microcontroller on it. Not just any microcontroller, though, but an Arduino compatible one. This means that you can program the flashlight just like an Arduino, using the same simple coding environment you’re used to. Then, you just plug in your USB cable, and away you go.

Is it overkill? Is a rechargable flashlight with a three-axis accelerometer absurd? Yes, yes it is. And that’s why I love it. It also happens to be the best flashlight I’ve ever seen, with a CREE XM-L U2 LED that puts out 500 lumens, and a beautiful total internal refraction lens.

People often talk about big data as though it is a measurable quantity, something quantitative. And it is, but that’s inadequate to understand the different nature of it. For me, the more important aspect is qualitative. Big data isn’t just about the number of gigabytes that your system deals with, but instead about the underlying nature of that data. Take, for example, traditional account data, where we might store orders, and shipping information. This data has a high signal-to-noise ratio. If, instead, we look at things like website logs, telemetry from embedded devices, or even a stream of tweets, we’re talking about what is traditionally a very low signal-to-noise ratio.

Big data, then, isn’t just about the size of your haystack, but instead finding the needle hidden within.

Weighing in at approximately 200 pages, or approximately 1.52x10E-27 grams (thanks information theory) on my iPad, there is little fluff in the book, and that is a great thing. I’ve been using Django since v0.96 came out in early 2007, and use it for a majority of my web work, and yet the book contained a lot of interesting ideas and insight. The result of reading the book was a TODO list with many items on it for all my projects to address a lot of issues that I’d not really thought about.

What more do you want from a book than for it to make you rethink how you approach solving problems? Go buy it.

• Kafka: a Distributed Messaging System for Log Processing (CiteSeerX)
• Paxos Made Moderately Complex (Cornell PDF)
• Pregel: A System for Large-Scale Graph Processing (Github PDF)
• Nonlinear Time-Series Prediction with Missing and Noisy Data (CiteSeerX)
• Bayesian Time Series: Models and Computations for the Analysis of Time Series in the Physical Sciences (CiteSeerX)
• Probabilistic Similarity Search for Uncertain Time Series (CiteSeerX)
• Processing a Trillion Cells per Mouse Click (CiteSeerX)
• Only Aggressive Elephants are Fast Elephants (arXiv)
• Uncertain Time-Series Similarity: Return to the Basics (arXiv)
• Statistical Distortion: Consequences of Data Cleaning (arXiv)

The last few are all from VLDB 2012, and I have another dozen or so papers from the same conference that I want to work my way through. Looking at these, you can see that a lot of it is an attempt to deal with streams of data, specifically in real-time, as best as possible.

• Disco: Running Commodity Operating Systems on Scalable Multiprocessors (CiteSeerX)
• End-to-end Arguments in Systems Design (CiteSeerX)
• Weighted Voting for Replicated Data (CiteSeerX)
• A Glossary of Time Granularity Concepts (PDF)
• An Access Control Model Supporting Periodicity Constraints and Temporal Reasoning (CiteSeerX)
• SharedDB: Killing One Thousand Queries With One Stone (CiteSeerX)

Some of these are quite old. For example, David Gifford’s paper on weighted voting was published in 1979, but it set forth the beginning of weighted r+w quorums that are now quite common. I’m pretty sure I had read it before, but sometimes older papers just need to be re-read to make sure that no ideas were missed.

Anyway, for various reasons too tedious to go into, I’ve allowed my inbox (also known as Dropbox) to accumulate over a hundred papers that I intended to read, but haven’t found time to yet. That doesn’t begin to include all the amazing blog articles, etc., that accumulate in Instapaper at all times. The Internet may be an amazing thing, but it also is a source of unlimited future reading. So, starting this year, and today to be exact, I’ve decided to try and put time aside every day to read a few of the things I’ve accumulated and try and slowly work down my backlog. I was asked by a friend on Twitter to keep track of what I’m reading, and so, this is the start.

• Hancock: A language for analyzing transactional data streams (CiteSeerX). A DSL for performing some relatively basic stream processing on large amounts of “sensor data”, in this case primarily telephone calls. Interesting ideas: 1) persistence mechanism that mirrors UNIX sensabilities with directories as containers; 2) view representation for abstracting data requirements over time, namely exact versus approximate representations
• Crash-only Software (CiteSeerX). What if we just gave up and quit trying to recover from errors? Sometimes it’s faster to just crash the system and reboot. Came with a couple interesting papers I want to read, namely Decoupled storage: Free the replicas! and Session State: Beyond Soft State
• PASSing the provenance challenge (Harvard). Integrating data provenance into the Linux operating system.
• CryptDB: Protecting Confidentiality with Encrpyted Query Processing (CiteSeerX). A new approach to encryption in the database that seems very different than anything I’ve seen in the commercial sector yet. One of the things I like is that it is adaptive and blends different approaches to encryption – including homomorphic encryption where appropriate – to obtain maximum functionality with minimized risk. Definiately want to look at this to play with, and it seems to currently work with MySQL and PostgreSQL with varying degrees.

A final note, and something that continues to amaze me in 2012: if your paper is not available for free to read, then why are you publishing your paper? I continually run into annoying pay walls – ACM and IEEE, I’m looking at you – that do nothing but impede research and progress. Now, usually you can Google around and find the paper hosted somewhere, but if you are an academic and your profession is (supposedly) about the progress of human knowledge, how can you subscribe to this kind of walling off of knowledge?

First, it’s necessary to say that this isn’t some Alfred North Whitehead tome of inscrutable kōans. Instead, it is written in a breezy and approachable style that masks the powerful themes contained; themes that are terrifying if you sit atop the old order. As Mary Poppins says, a spoonful of sugar helps the medicine go down, and in this case the medicine is one that will empower entire new industries and bring others to their knees. The entire book revolves around the idea that developers – and not managers and executives – are the power brokers in the IT industry, and to a lesser extent that all things are IT. For those of us on the inside of this transformation, this seems a somewhat obvious observation in retrospect, but when Stephen first offered this idea in 2010 – akin to Tim O’Reilly’s early observation of the power of the alpha geeks in 2002 – this wasn’t as clear as it is now. The near-alchemic combination of open source, the Internet, and its bastard love-child cloud computing, have rewritten the rules for everyone, whether they know it or not.

Do yourself a favor, take an hour out of your life to read Stephen’s mini-book, and then spend many more hours thinking about what it all means to your world. Like all good writing, it should leave you with more questions than you had when you started. Use those questions to re-evaluate and re-shape your own future.

Oh, and did I mention that it’s free? Thanks to the support of the team at New Relic, Stephen was able to get the book edited and published for free by O’Reilly. It really is a different world.

To begin, let’s take a look at some of the assumptions we have to meet:

1. $0.01 per gigabyte per month
2. 99.999999999% durability, which is not to be confused with availability.
3. 3-5 hour typical retrieval times
4. “Unlimited” scalability

Let’s get this out of the way: Amazon is not using tape. I can’t provide any evidence for this, but various AWS people have publically made their distate for tape well known. So, with that assumption, I think we can safely say that they are based on traditional rotating hard drives.

The rest of this is going to be very speculative, but I think it likely fits into the general approach that AWS has taken. To analyze the entire system, we’re going to talk about a couple of areas:

1. Data at rest costs
2. Power
3. Data ingest and retrieval
4. Bandwidth costs

Data at rest costs

Since I’ve already said that Glacier is based on traditional rotating media, we need to decide what kind they’re using. My guess would be 3TB SATA drives, as they currently represent the best capacity for your money. So how much is Amazon paying for these drives? I would estimate that Amazon is paying less than $100 for a 3TB drive, perhaps even under $75. General retail is around $150, but Amazon has (conservatively) over 100PB of data in S3 currently, so they’re buying power is excellent.

Assuming that Amazon has an approximately 3x overhead for all storage – including erasure coding and management administrivia – that’s upwards of 100,000 hard drives spinning in S3, but this is a very conservative estimate. Based on a $75 per-unit pricing, we’re talking about $0.025 per gigabyte, and with a tripling for overhead that’s $0.075 per gigabyte.

Now, those drives have to live somewhere. While you could just pile them in the corner, they’re harder to get data in and out of at that point. Instead, let’s presume that Amazon has a custom storage server. It wouldn’t be the first company to do such a thing. Since you don’t need any redundant power supplies or anything in the system, let’s assume they’ve got a box that 45 drives and costs them about $1,500 delivered. That gives you an amortized cost of $44.44 per drive in capital expense.

Put all that data together and we have a capital cost of $4,875 for 135TB of raw capacity, or 45TB of “usable capacity”. That is $0.1083/gigabyte. Think about that. In less than 1yr of usage, AWS would recoupe all the costs associated with the storage itself, and it’s easy to imagine a 7-10 year usable life for the hardware given some of the things that are discussed in the next section.

Power

Forrester Research estimates that the cost of power will exceed the capital cost of servers over their useful life. Do you know how you fix that problem?

Turn them off.

The key is the 3-5 hour retrieval time that AWS quotes. I’ll go into it more, but it’s key to being able to turn the servers off as often as possible. The overall goal is to turn a server on, fill up all it’s hard drives and then turn it back off for as long as possible. In fact, you could spin hard drives up and down as needed, saving even more space. You’re basically treating them as “tape” by filling them up sequentially (approximately).

A powered off server uses almost no power. There’s a tiny bit for the baseband controller that you need to remotely turn it on, but that’s nothing compared to spinning up the hard drives.

One additional area AWS may be benefitting from is that as such a huge hard drive purchaser, they might be able to get custom motors or at least firmware that allow the hard drive to spin slower and use less power. Because of the totally different performance characteristics, a hard drive running at 5,400 RPM, or even 3,600 RPM, is likely to be plenty fast enough for the application.

Note that this also likely has something to do with the odd penalty for early deletion. That means they have to keep the server spun up to refill the deleted data.

If you can turn the server off, that means it’s not generating heat. That means you can use less A/C, which is a huge capital and expense item for a data center. Finally, if you were to build dedicated parts of a data center to just hold Glacier components, then you can run the whole thing at a higher temperature because you only have a small number of servers running concurrently and the remaining air becomes a bit of a heat sink allowing it to absorb the BTUs.

Finally, hard drives have a certain number of hours they’re rated for, but as Google discovered, there’s a lot of interesting failure issues involved. Hard drives that aren’t spinning last long, though of course there are limits even to this. Eventually, they’ll fail, but the wear-and-tear is much lower.

Data ingest and retrieval

So how does data get into Glacier? Dollars to donuts says that you don’t talk directly to Glacier. I would imagine there’s a bit of hierarchical storage going on, with S3 being used as a staging repository. Put simply, data is uploading by customers to Glacier through a front end that stores it in S3 until enough data has accumulated to justify spinning up a new set of servers and filling them with data. Once that’s done, the data can be aged out of the S3 “cache”.

When you ask for a retrieval, however, I suspect there’s a coalescing of requests to make sure that as many requests from the same set of machines as possible is satisfied at once. This, combined with the power discussed earlier, is why there’s a 3-5 hour lag, and maybe longer. It also is a psychological pressure not to treat Glacier as a cheap version of S3, but instead as a true “cold storage”. Once retrieved, the data is staged into S3 and available for download.

My guess is that the future capability to do aging in and out of S3 is linked in this. Most likely, AWS is just working out the bugs in their own use before making it generally available.

Bandwidth costs

When you buy large amounts of bandwidth, you’re generally buying symmetrical bandwidth. Buy 100Gbps and you actually are talking 200Gbps, with 100 in each direction. I have no actual data for AWS, but in my experience, most hosting providers have more egress data than ingress. I’m sure there’s exceptions, but that’s generally the case. This is part of why AWS can offer ingress bandwidth for free. For them, it likely is something approaching free, because it’s already provisioned and sitting idle. Also, every gigabyte you upload costs you money to store.

Conclusion

So there you have it. No miracles, just the intelligent application of a lot of holistic system tuning for a specific workload. Now some have supposed that this is just a layer on top of S3, but I think that’s likely not true just because of the totally different workload impacts. What I suspect is it’s derived from S3 code wise, but doesn’t share the same infrastructure in anyway.

Here’s the big problem I see: even at its most optimal, FedRAMP is too slow. It’s not specific to FedRAMP, as those involved have thought through the issues and done a great job with what they have to work with. The problem is that “what they have to work with” is the NIST SP800-53 framework, and it’s just not something that fits well with the cloud world.

I take that back, it’s not cloud that’s the problem, it’s modern architectures. I faced similar challenges when dealing with a huge SOA-based system developed specifically for a government agency. What is a system when you have hundreds (or thousands) of services that are interdependent in intricate ways? How do you even begin to think about that? Clouds add another layer of complexity and vagueness to the whole recipe.

“The cosmos is all that is, or ever was, or ever will be. Our contemplations of the cosmos stir us. There is a tingling in the spine; a catch in the voice; a faint sensation, as if a distant memory, of falling from a great height. We know we are approaching the grandest of mysteries.” – Carl Sagan

With those eloquent words, Carl Sagan launched his epic series Cosmos. I was 8 years old when this carefully spoken man, with a measured cadence, launched my mind on its own voyage into the curiosity and questioning that underpins science. Recently, I re-watched the original series, and while the special effects are dated, the message, and the man, are timeless.

For three months, Carl Sagan took the entire country on a journey through the universe, as both a place and more critically as an idea. I would sit rapt, unable to do anything else, and the words inspired me. I wonder where today’s youth will find their inspiration? It is difficult to contemplate a world without Carl Sagan. Science and technology have advanced so far in the last 20 years, and today we have so few people who can clearly articular the science while not forgetting the awe that the average person experiences when contemplating their place in the universe.

First, let’s take a look at some of the top books right now on Amazon that are available both in paper and eBook format:

Lots of people will look at the cost and think “wow, you don’t save any money” buying an eBook, but they misunderstand the economics. The cost of producing a paperback book is trivial to the overall cost. If you’re producing 50,000 copies, you’re talking $2-3 in total cost to print and distribute. That’s not the problem. The idiosyncrasy of Fifty Shades of Grey is an artifact of the agency model.

The problem is that the value of an eBook is lower. Not only does the publisher not have to produce the physical book, but they also take from you all your rights by implementing DRM. With a physical book I can loan it without restraint to anyone I wish. I can give it to anyone. I can do anything I want. With an eBook, though, I’m limited to what the publisher wants to allow, and many publishers do not allow anything at all.

If, instead, you offered me an eBook for the current 8-15% discount, but gave it to me in an unrestricted ePub format, then I’d say you have a good value. Until then, I’ll find it difficult to stomach the pricing model for DRM-protected eBooks.

“Four million” sounds really impressive, doesn’t it? That’s a lot of data, and I won’t deny that, but it’s not that much data when it comes down to it. For capacity planning, if all I had was a daily rate, I took that rate over 12 hours to compensate a bit for what is termed the peak busy hour. It’s not an elegant solution, but it solves the back-of-the-napkin estimate world just fine. So, 4M/day is about 93 events per second.

On the surface, that seems like a lot, but it’s not. Even when I was immersed in the SEIM world on a day-to-day basis, we were dealing with many customers who were attempting to digest 10,000 EPS. That translates to approximately 432,000,000 events per day, or several orders of magnitude greater. Mind you, that was in the early 2000s, and the world has gotten a lot more intense than it was back then. I know of organizations dealing with 100,000 EPS, or more, today.

Now, if we assume the average security event is (uncompressed) approximately 64 bytes, once some normalization happens, then you’re talking about 6.25 megabytes per second of data to deal with. It’s a lot, and 99.999% of it is useless when it arrives. It only becomes interesting later, but you won’t know how far in the future that will be.

To put it all in perspective, if you could print a single event on a single line on a sheet of paper, and assuming you can print 66 lines on a page (something we used to do in the days of line printers), then that’s 151 pages per second in my old experiences, and over 1,515 pages today. Every second. Stick that in your shredder and think about it.

So, that brings me to my point: before you go around bragging about performance, it’s best to understand what state-of-the-art really is. Big data arrived a very long time ago in the security world, but the tools and technologies still haven’t caught up.

The first layer of defense you have control over is what packets end up at your systems, and what you do with them. For a Linux machine, this is controlled by the iptables component of the operating systems. The goal of this moat around your system is to try and keep a vast majority of the stupid at bay. There’s lots of things that you should never, ever, see, and that there’s simply no reason to even bother with.

What I’m going to do is walk you through the foundation rule set (my starter moat) that I base everything else on, which you can find as a gist on GitHub. Please feel free to use however you wish, though if you find a mistake I would ask you just let me know by putting a comment on the gist itself. Feel free to add your own alligators and flaming spikes.

Categorizing Flows

The first thing we need to do is group our traffic into different chains of rules that will be applied. This makes it a bit easier to deal with. Note that the gist has a lot of this as comments.

-N ICMP_IN
-N ICMP_OUT
-N SPOOF_LOG_DROP
-N SPOOF_IN
-N SPOOF_OUT
-N BAD_TCP_FLAGS

The first two, ICMP_IN and ICMP_OUT are somewhat self explanetory. We want to treat all ICMP carefully. The next three, SPOOF_LOG_DROP, SPOOF_IN and SPOOF_OUT are all about address spoofing protection, something everyone should be doing, but usually isn’t. The last one, BAD_TCP_FLAGS is looking for all sorts of nasty behavior that people use for either OS detection, or often to try and find exploits in a system.

We’ll be going through them in roughly that order.

ICMP Management

-A ICMP_IN -p icmp --icmp-type 8 -j DROP
-A ICMP_IN -p icmp -i eth0 --icmp-type 0 -m state --state ESTABLISHED,RELATED -j ACCEPT
-A ICMP_IN -p icmp -i eth0 --icmp-type 3 -m state --state ESTABLISHED,RELATED -j ACCEPT
-A ICMP_IN -p icmp -i eth0 --icmp-type 11 -m state --state ESTABLISHED,RELATED -j ACCEPT
-A ICMP_IN -p icmp -i eth0 -j DROP
-A ICMP_OUT -p icmp -o eth0 --icmp-type 8 -m state --state NEW -j ACCEPT
-A ICMP_OUT -p icmp -o eth0 -j DROP
-A INPUT -p icmp -j ICMP_IN
-A OUTPUT -p icmp -j ICMP_OUT

In line 1, we just ignore all the ICMP echo requests (type 8). Ping is a good example of a use of an echo request. There’s just no reason to respond to them normally. If, however, you have a tool that needs to ping your system to get a response, then you’ll need to modify the filter slightly to be address-specific. Line 2 drops anything that’s an echo response (type 0) which we didn’t initiate. Next, lines 3 and 4 drop the destination unreachable (type 3) and TTL (type 11) responses if they’re not related to something we sent. These are another sneaky way to peek into a system.

Then, we drop everything else, because they fail the sanity check. Generally, a system would only send an echo request in response to a ping command, and there’s only three responses that make any sense to those in the modern world: response, TTL-exceeded and destination unreachable.

Spoofed Packets

Now that we’ve dealt with incoming packets, we’re going to allow echo requests (ping) to leave the system. Everything else ICMP-related, such as redirects, timestamp requests, etc., shouldn’t be coming or going, and so we drop them without note. Finally, we attach our rule chains to the core rule chains, INPUT and OUTPUT.

-A SPOOF_LOG_DROP -j LOG --log-prefix "IPT: spoofed "
-A SPOOF_LOG_DROP -j DROP
-A SPOOF_IN -i eth0 -s <MYIP> -j SPOOF_LOG_DROP

With ICMP traffic out of the way, we need to deal with traffic coming and going to addresses that don’t pass the sanity check. You can find a lot of these addresses in RFC3330, “Special-Use IPv4 Addresses”. People often forget there’s more out there than just the addresses in RFC1918. So, since we want to keep an eye on this, the first thing we do (lines 1-2) is set up some log configuration. Log messages matching this rule chain will be prefixed with “IPT: spoofed”. IPT just stands for IP tables.

So, before we go any further, we need to make sure nobody is spoofing our own addresses. In line 3, you’ll see something <MYIP>, which needs to be replaced by whatever IP address is used by the host. The rule says “if I see something with a source address that is mine on my ethernet connection, drop it”. You shouldn’t see it showing up there. Ever. If you do, you likely either have a serious security problem, or need to talk to someone about how networking is set up in detail.

-A SPOOF_IN -i eth0 -s 10.0.0.0/8 -j SPOOF_LOG_DROP
-A SPOOF_IN -i eth0 -s 172.16.0.0/12 -j SPOOF_LOG_DROP
-A SPOOF_IN -i eth0 -s 192.168.0.0/16 -j SPOOF_LOG_DROP

Next, we block all the standard RFC1918 addresses. Now, if you’re actually using them internally, you can’t do this, but this is from situations where my server only has a publicly routable address.

-A SPOOF_IN -i eth0 -s 198.18.0.0/15 -j SPOOF_LOG_DROP
-A SPOOF_IN -i eth0 -s 169.254.0.0/16 -j SPOOF_LOG_DROP
-A SPOOF_IN -i eth0 -s 192.0.2.0/24 -j SPOOF_LOG_DROP

Here, we block (line 1) the official “benchmarking” networks, defined in RFC2544. While I’ve yet to see them in the wild, they shouldn’t show up, and part of the goal of this rule set is to make sure we set a sanity benchmark. Next, line 2 drops link local traffic (RFC3927). Link local addresses are those that are “randomly” assigned when an interface doesn’t have a static address, and can’t use something like DHCP to dynamically assign one. Again, it should never show up in a “normal” situation. Line 3 drops TEST-NET traffic. TEST-NET, as defined in RFC5737 is intended only for use in documentation. Once again, it should never show up in production use.

-A SPOOF_IN -i eth0 -s 224.0.0.0/4 -j SPOOF_LOG_DROP
-A SPOOF_IN -i eth0 -s 240.0.0.0/4 -j SPOOF_LOG_DROP

Since I almost never have any use for multicast, I drop everything associated with the standard multicast blocks, defined in RFC5771.

-A SPOOF_IN -i eth0 -s 127.0.0.0/8 -j SPOOF_LOG_DROP
-A SPOOF_IN -i eth0 -s 0.0.0.0/8 -j SPOOF_LOG_DROP
-A SPOOF_IN -i eth0 -s 255.255.255.255/32 -j SPOOF_LOG_DROP

Now, we also shouldn’t see loopback addresses, or other bonkers addresses showing up on our Ethernet interface. See below for information on the loopback protections.

-A SPOOF_OUT -i eth0 -s ! <MYIP> -j SPOOF_LOG_DROP

One thing many people forget to do is block their systems from becoming a source of problems. So, we block any outgoing traffic on our Ethernet interface that isn’t originating from my IP address.

-A INPUT -j SPOOF_IN
-A OUTPUT -j SPOOF_OUT

And now, finally, we attach these new rule chains to the primary ones, just a we did before.

TCP Flags

That brings us to the last “protection” set of rules: those associated with all sorts of crazy flags in the TCP packet. If you’ve forgotten, the TCP packet has 9 potential flags. Read LSB to MSB:

• NS: ECN-nonce concealment protection (RFC3540)
• CWR: Congestion Window Reduced flag is set by the sender to indicate that it received a TCP segment with the ECE flag and had responded in congestion control mechanism (RFC3168)
• ECE: ECN-Echo indicates:
  • If SYN flag is set, that the TCP peer is ECN capable.
  • If SYN flag is clear, that a packet with Congestion Experienced flag in IP header set is received during normal transmission (RFC3168).
• URG: the Urgent pointer field is significant
• ACK: the Acknowledgment field is significant. All packets after the initial SYN packet sent by the client should have this flag set.
• PSH: Push function. Asks to push the buffered data to the receiving application.
• RST: Reset the connection
• SYN: Synchronize sequence numbers. Only the first packet sent from each end should have this flag set.
• FIN: Finished. No more data from sender

Only some of these can be set “together”, and often you find people probing systems to see how they respond to various flag combinations. For example, one of the techniques for OS detection used by nmap is to play with the various flags to see how a host responds. We don’t use the SPOOF_LOG_DROP-style reaction because we want to change the log message so we know what’s going on.

-A BAD_TCP_FLAGS -p tcp --tcp-flags SYN,FIN SYN,FIN -j LOG --log-prefix "IPT: Bad SF flag "
-A BAD_TCP_FLAGS -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
-A BAD_TCP_FLAGS -p tcp --tcp-flags SYN,RST SYN,RST -j LOG --log-prefix "IPT: Bad SR flag "
-A BAD_TCP_FLAGS -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
-A BAD_TCP_FLAGS -p tcp --tcp-flags SYN,FIN,PSH SYN,FIN,PSH -j LOG --log-prefix "IPT: Bad SFP flag "
-A BAD_TCP_FLAGS -p tcp --tcp-flags SYN,FIN,PSH SYN,FIN,PSH -j DROP
-A BAD_TCP_FLAGS -p tcp --tcp-flags SYN,FIN,RST SYN,FIN,RST -j LOG --log-prefix "IPT: Bad SFR flag "
-A BAD_TCP_FLAGS -p tcp --tcp-flags SYN,FIN,RST SYN,FIN,RST -j DROP
-A BAD_TCP_FLAGS -p tcp --tcp-flags SYN,FIN,RST,PSH SYN,FIN,RST,PSH -j LOG --log-prefix "IPT: Bad SFRP flag "
-A BAD_TCP_FLAGS -p tcp --tcp-flags SYN,FIN,RST,PSH SYN,FIN,RST,PSH -j DROP

But sometimes, we need things set together, and if they aren’t, then it doesn’t make sense from a network stack perspective. Then, we have some things that can not exist in the first SYN packet, so they must be accompanied by the ACK flag. If they’re not, we don’t want them.

-A BAD_TCP_FLAGS -p tcp --tcp-flags ACK,FIN FIN -j LOG --log-prefix "IPT: Bad F-A flag "
-A BAD_TCP_FLAGS -p tcp --tcp-flags ACK,FIN FIN -j DROP
-A BAD_TCP_FLAGS -p tcp --tcp-flags ACK,PSH PSH -j LOG --log-prefix "IPT: Bad P-A flag "
-A BAD_TCP_FLAGS -p tcp --tcp-flags ACK,PSH PSH -j DROP
-A BAD_TCP_FLAGS -p tcp --tcp-flags ACK,URG URG -j LOG --log-prefix "IPT: Bad U-A flag "
-A BAD_TCP_FLAGS -p tcp --tcp-flags ACK,URG URG -j DROP

Then, we have people who think it’s OK to have no flags or all the flags set:

-A BAD_TCP_FLAGS -p tcp --tcp-flags ALL NONE -j LOG --log-prefix "IPT: Null flag "
-A BAD_TCP_FLAGS -p tcp --tcp-flags ALL NONE -j DROP
-A BAD_TCP_FLAGS -p tcp --tcp-flags ALL ALL -j LOG --log-prefix "IPT: All flags "
-A BAD_TCP_FLAGS -p tcp --tcp-flags ALL ALL -j DROP

Oh, and merry Christmas. Normally, I’m all for Christmas, but, these are just insane:

-A BAD_TCP_FLAGS -p tcp --tcp-flags ALL FIN,URG,PSH -j LOG --log-prefix "IPT: Xmas flags "
-A BAD_TCP_FLAGS -p tcp --tcp-flags ALL FIN,URG,PSH -j DROP
-A BAD_TCP_FLAGS -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j LOG --log-prefix "IPT: Merry Xmas flags "
-A BAD_TCP_FLAGS -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j DROP

And then, just attach it to the main INPUT rule chain.

-A INPUT -p tcp -j BAD_TCP_FLAGS

Normal Traffic Controls

Now we get into more “normal” traffic controls. First, we want to allow everything on the loopback (lo) interface. This is used for both local servers (such as databases, proxies, etc.) and for SSH tunneling:

-A INPUT -i lo -j ACCEPT

And drop it if it is on the loopback network, but not coming through that interface:

-A INPUT -i ! lo -d 127.0.0.0/8 -j REJECT

We also want to allow all traffic associated with previously permitted connections. These are generally called “established” connections:

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

Now, it might be helpful if we allow traffic to originate from the system to other places. On some systems, I also tighten this down to be only a very small subset of traffic, perhaps only HTTP, but that’s the next step, and this is the 81% rule.

-A OUTPUT -j ACCEPT

And that brings us to inbound application traffic. This is traffic we expect to be coming in, such as web browser traffic to a web server, or SSH:

-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT

Now we need to tweak some of the logging information. We don’t want to get overwhelmed with logs and have that turn into a denial-of-service attack itself. So, to prevent that, we restrict it to bursts and 60/minute:

-A INPUT -m limit --limit-burst 100 --limit 60/min -j LOG --log-prefix "IPT: denied " --log-level 7

Repeat after me: that which is not explicitly permitted is denied:

-A INPUT -j REJECT

Also, forwarding is evil. Do not forward on this host. Ever.

-A FORWARD -j REJECT

And that’s the basic set of rules. You can customize these till your heart’s content, but this is a start. Sadly, it won’t be all the security you need, but it’s better than what many people have sitting out there.

Not too long ago, I decided to get rid of my blog. While I had enjoyed writing in it, it had stagnated for a while, and I didn’t really have any inspiration to write more at the time. Cue the ticking of the clock, and some time has passed. Part of the inspiration comes from Nathan Marz in his post “You should blog, even if you have no readers”. Now, I feel like I’m ready to return to a more regular writing habit.

This time around, though, I’m not interested in dealing with the disaster that is WordPress. Instead, I’ve decided to go with a “static site generator”: Octopress. In time, I will bring over a lot of the writing from my old site.

Now, let the games begin!

Take airports. The security at airports continues to be abysmal at best, insulting in the worst, and misleading in all actuality. I know people who have boarded plans without photo ID through Dulles International Aiport (IAD) in Washington, DC, and people who are prototypical passengers that would not be suspicious (this is not race based, but behavioral) who are harassed to no end, repeatedly. Why? Not because they are suspicious, but because they were convenient. People are somehow “comforted” by seeing people searched, even if those people are not the ones who could potentially present a security risk.

This is typified by the either intentional or negligent avoidance of dealing with private jets (otherwise known as “political supports”), cargo transports, and other vectors for attacks that are substantially more effective. Why weren’t they dealt with? Because they’re not visible to the “average” person, yet they represent the actual targets that an intelligent terrorist would go after. We’ve been fortunate to not have been targeted by an overly intelligent and concerted effort, just a rather clumsy one so far.

This is also typified in the methodology of Federal agencies in Washington, DC that I work with on a daily basis. Many have erected all sorts of physical barriers, but ignore thousands of other vectors that are exposed. Again, lots of smoke, no fire. The effective reduction of risk is a difficult job, and requires enormous intellectual resources — not necessarily monetary resources. Much of the current government approaches are focused on spending money — often just to line pockets of supporters or constituents — rather than the examination of risks strategies.

As I said, risks can not be eliminated, but can only be mitigated, reduced and managed. I have addressed too many people involved in the “war on terrorism” who feel that there is a 100% solution out there, when that is the fatal flaw. The assumption of perfection creates it’s own enormous risk. Only when we understands the risks can be manage them.

Today, we are no more safe than we were September 10th due to anything the government has done, but we have lost countless liberties that will be difficult to regain. Any increase in security is largely due to the attention of the American people, not the government.