Posts tagged decision making
I just finishing giving a third version of a presentation that I put together on lessons Infosec/Risk/Platform owners can learn from classic Operations Research/Management Science type work. The talk (“Operating * By the Numbers”) was shared in Reykjavik (Nordic Security Conference), Seattle (SIRACon 2013), and in Silicon Valley (BayThreat). Thanks everyone who attended, especially those of you who asked questions and provided feedback.
A few folks have asked for reading lists. Some asked for the quick run-through sample from my bookshelf, others want some further reading. Here’s the quick run through:
- Introduction to Mathematical Statistics and Its Applications (5th Edition), Richard J. Larsen and Morris L. Marx
- Out of Control: The New Biology of Machines, Social Systems, & the Economic World, Kevin Kelly
- The Illuminatus! Trilogy Robert Shea & Robert Anton Wilson
- How to Protect Yourself from Crime, Ira Lipman (Guardsmark)
- Hackers: Heroes of the Computer Revolution – 25th Anniversary Edition, Steven Levy
- Computer Crime: A Crimefighter’s Handbook, David Icove, Karl Seger, William VonStorch
- Maximum Security: A Hacker’s Guide to Protecting Your Internet Site and Network, Anonymous
- Information Security Risk Analysis, Thomas R Peltier
- A First Course in Probability, Sheldon Ross
- Strategy, Basil H. Liddell Hart
- Mostly Harmless Econometrics: An Empiricist’s Companion, Joshua D. Angrist and Jörn-Steffen Pischke
- The Dilbert Principle, Scott Adams
- Introduction to Topology: Third Edition, Bert Mendelson
- Exploratory Data Analysis (Quantitative Applications in the Social Sciences), Frederick Hartwig with Brian E Dearing
- Game Theory Evolving: A Problem-Centered Introduction to Modeling Strategic Interaction (Second Edition), Herbert Gintis
- Practical Statistics Simply Explained (Dover Books on Mathematics), Russell Langley
- Excel Data Analysis For Dummies, Stephen Nelson
- Operations Management: Contemporary Concepts, Roger Schroeder
And I also want to give another shout-out to Combat Modeling, by Alan Washburn and Moshe Kress, of the Naval Postgraduate School. It’s a pricey text, but take a look at the table of contents & the topics they cover. Really interesting work to consider for control system designers.
Also, I haven’t read these personally but they are on my “to read” list as they came recommended by fellow quant/risk nerds:
- The Principles and Applications of Decision Analysis : 2 Volume Set, Ronald A. Howard and James E. Matheson
- Decision Analysis for the Professional (pdf link), Peter McNamee & John Celona
And here’s a link to one of my blog posts (Quant Ops), which includes a few references and some thinking on the topic from a different angle.
Risk modeling, while it sounds specific, is actually super-contextual. I think my own perspective on the topic (the different types of modeling, what they are good for) was best summed-up in a paper/presentation combo I worked on with Alex Hutton for Black Hat & SOURCE Barcelona in 2010. Probably video from Barcelona is the best reference if you want to look that up (yes, lazy blogger is lazy), but let me summarize by the (from my perspective) three general purposes of risk models:
- Design: Aligned most with system theory. The models try to summarize the inputs (threats, vulns, motives, protections) and the outputs (generally, loss and in some cases “gains”) of a system, based on some understanding of mechanisms in the system that will allow or impede inputs as catalysts/diffusers of outputs. Generally I would lump attack tree modeling and threat modeling into this family, just a different perspective on a system as a network architecture or design of a protocol, software, or network stack. Outside of risk/security, a general “business model” is equivalent, which attempts to clarify the scope, size, cost,and expected performance of the project.
- Management: Aligned most with the security/risk metrics movement, and (to some extent) aligned with “GRC”-type work, management-focused risk models are set-up to measure and estimate performance, i.e. to answer a questions about “how well are controls mitigating risk” or “to how much risk are we exposed”. One could think of the output of the design phase being a view as to what types of outcomes to expect, and then the management phase will provide a view on what outcomes are actually being generated by a system/organization. Outside of risk/security, a good example of a management model is the adoption of annual/quarterly/ongoing quality goals, and regular review of performance against targets.
- Operations: Operational models are a different beast. And my favorite. Operational models aren’t trying to describe a system, they are embedded into the system, they influence the activities taking place in the system, often in real-time. I suppose any set of heuristics could be included in this definition, including ACL’s. I prefer to focus on models that take multiple variables into consideration – not necessarily complex variables – and generate scores or vectors of scores. Why? Because generally the quality of decision (model fit, accuracy, performance, cost/benefit trade-off) will be more optimized, i.e. better. Outside of risk/security, a good example is dynamic traffic routing used in intelligent transport systems.
“Framework” is another term that I’ve heard used in a number of different ways but it seems to really be an explanation of a selected approach to modeling, and then some bits on process – how models and processes will be applied in an ongoing approach to administer the system. Even Wikipedia shies away from an over-arching definition, the closest we get is “conceptual framework“: described as an outline of possible courses of action or to present a preferred approach to an idea or thought. They suggest we also look at the definition for scaffolding: “a structure used as a guide to build something” – (yes, thank you, I want us to start discussing risk scaffolding when we review architecture, pls)
Recently, I was interviewed for the ActiveState blog on DevOps & Platform as a Service (PaaS); that interview made it to Wired.com (here). A discussion on the topic was timely, as I’ve been thinking about DevOps and other agile delivery chain mechanisms quite a bit lately, mainly as I am applying them in my current gig which my colleagues are I describe as “Business Ops”. Next month at Nordic Security 2013 I’ll be presenting “Operating * By the Numbers” (If you’re wondering why there’s no abstract, it’s because I’m still perfecting “Just In Time” deck development…just kidding. Sort of.*)
Anyway, I thought it might be a good idea to explain What I’m Talking About When I Talk About DevOps (apologies to the incomparable Haruki Murakami). This will be my first time trying to explain where I’m going with this whole DevOps thing, so it might get fuzzy. Bear with me. I reserve the right to change my mind later, of course (I’m cognitively agile that way, haha), so if you have comments or criticisms I’m very open to hearing your thoughts.
Connection between DevOps & Risk
DevOps, if you’ve not heard of it before, is a concept/approach to managing large-scale software deployments. It seems to be most popular/effective at software-based or online services, and it is “big” at highly scaled out companies like Google, Etsy, and Netflix. Whether consumer-facing or B2B, these services need to be fast and highly-reliable/available. The DevOps movement is one where deployments and maintenance are simplified (simplicity is easier to maintain than complexity) through standardization and automation, lots of instrumentation & monitoring, and an integration of process across teams (most specifically, Dev, QA & Ops). More on “QA” later.
But…the thing about DevOps is, that while it is a new concept in the world of online services, it draws heavily from Operations Management, which is not new. The field of Operations Research was forged in manufacturing but the core concepts are easily applied across other product development cycles. In fact this extension is largely overdue, since a scan through semi-recent texts on operations management shows IT largely described as an enabling function (e.g. ERP) but not a product class in and of itself. (BTW, in some curriculums, Operations Management is cross-listed or referred to as Decision Science, which is a core component of risk/security analytics.)
Last week I stopped into SOURCE Dublin to give a follow-up to my recent talk in Boston, another foray into game theory (Games We Play: Payoffs & Chaos Monkeys) — this time w/some more advanced mathiness and references back into behavioral economics. Anyway, I still owe some explanatory blog posts to support some of the materials I had to rush through (to get everything into 45 minutes), but first thing I wanted to share is my working reading list. I’m finishing up reading some other books which I’ll post later but this is a good overview and will get folks interested in the topics headed in the right direction.
- Ariely, Dan. Predictably Irrational: The Hidden Forces That Shape Our Decisions. New York, NY: HarperCollins, 2008. Print.
- Axelrod, Robert M. The Evolution of Cooperation. New York: Basic, 1984. Print.
- Fisher, Len. Rock Paper Scissors: Game Theory in Everyday Life. New York: Basic, 2008. Print.
- Gibbons, Robert. Game Theory for Applied Economists. Princeton, NJ: Princeton UP, 1992. Print.
- Gintis, Herbert. Game Theory Evolving: A Problem-Centered Introduction to Modeling Strategic Interaction. Princeton, NJ: Princeton UP, 2000. Print.
- Ignacio Palacios-Heurta (2003) “Professionals Play Minimax” Review of Economic Studies, Volume 70, pp 395-415. (http://www.palacios-huerta.com/docs/professionals.pdf)
- Jackson, Leyton-Brown & Shoham. Game Theory. (Stanford University and University of British Columbia: Coursera), http://www.coursera.org, Accessed 2013.
- Kahneman, Daniel. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011. Print.
- Leyton-Brown, Kevin, and Yoav Shoham. Essentials of Game Theory: A Concise, Multidisciplinary Introduction. [San Rafael, Calif.]: Morgan & Claypool, 2008. Print.
- Meadows, Donella. Thinking in Systems: A Primer. Chelsea Green Publishing, 2008. Kindle edition.
- Polak, Ben. Game Theory (Yale University: Open Yale Courses), http://oyc.yale.edu, Accessed 2012. License: Creative Commons BY-NC-SA
- Thomas, L. C. Games, Theory, and Applications. Chichester: E. Horwood, 1984. Print.
- Wikipedia sections on Game Theory section, Economics, and Probability.