Appendices¶
STIX Development Libraries¶
Subject: Re: [cti-users] Re: Announcing python-stix2
Sender: <cti-users@lists.oasis-open.org>
Delivered-To: mailing list cti-users@lists.oasis-open.org
Received: from lists.oasis-open.org (oasis-open.org [66.179.20.138])
by lists.oasis-open.org (Postfix) with ESMTP id B1A965818580;
Wed, 15 Mar 2017 09:48:17 -0700 (PDT)
Message-ID: <58C96F55.4030104@apl.washington.edu>
Subject: Re: [cti-users] Re: Announcing python-stix2
Date: Wed, 15 Mar 2017 09:44:05 -0700
From: Stuart Maclean <stuart@apl.washington.edu>
To: "Back, Greg" <gback@mitre.org>, Eyal Paz <eyalp@checkpoint.com>,
"cti@lists.oasis-open.org" <cti@lists.oasis-open.org>,
"cti-users@lists.oasis-open.org" <cti-users@lists.oasis-open.org>
References: <AAD76FB4-D5FE-41FA-9BDD-1EE6BBE4062D@mitre.org>
<2956157C13955D458A922DDE028FEE7E0209A8F61D@IL-EX10.ad.checkpoint.com>
<D3938F78-9B27-48B2-87BA-E57803DEBA81@mitre.org>
In-Reply-To: <D3938F78-9B27-48B2-87BA-E57803DEBA81@mitre.org>
On 03/15/2017 09:22 AM, Back, Greg wrote:
>
> Hi Eyal,
>
> MITRE isn't working on anything, and I'm not personally aware of
> anyone else who is either. But if someone is, hopefully they're on one
> of these lists and will respond.
>
> Greg
Greg, all,
I have followed the STIX development over the past couple of years, and
even developed a Java library for STIX manipulation, see
https://github.com/uw-dims/stix-java. That was when STIX 1.1 was current.
I watched with some dismay as the XML schema way of doing things in STIX
1.1 was dismantled in favor of JSON for STIX 2. While XML schemas are
complicated, they are very precise, and a document can be checked
against a schema to assert its validity. Further, the richness of tools
for XML schemas, notably the JAXB/xjc tools for Java, gives developers a
huge leg-up in implementing an API for STIX manipulation. I just
cranked the JAXB handle over the .xsd file set, and hey presto I had a
set of Java classes with which to start my API.
Going out on a limb here, I also think that Java developers LIKE more
discipline that Python developers. Static typing vs duck typing? The
XML schema way of representing information is disciplined, and leads to
fewer 'You meant X? I thought you meant Y' gotchas at runtime.
Extrapolating, I think the JSON way of data representation is less
disciplined than the XML way, hence the natural inclination of Python
developers to prefer JSON.
Looking at the STIX 2 specs, I admire effort the authors must have put
in. But from a library builder's point of view, there being no
machine-ingestable docs (ie the xsd files in STIX 1.1), I am back to
square one (if I have missed any machine-readable docs, I apologize).
So, in a nutshell, my own feeling is that there will be no Java-language
STIX 2 manipulation tools, a sad fact, and I sincerely hope I am wrong
in this respect. I do know that I won't add to my STIX 1.1 Java effort.
Stuart
----
Reference Links¶
Note
Over the life of the project, the PI lead the effort to research new technologies and guide the project team in learning these technologies in order to leverage them to acheive the integration goals of the project. The best references were organized and maintained as part of the PI’s home page and all team members where urged to look there first and to let the PI know when they find new information that is not already there. These references are included here to help guide anyone else following this same path.
Containerization, Virtualization, “Microservice Architectures”¶
- Microservices, Wikipedia
- Microservices: a definition of this new architecture, my Martin Fowler
- Operating-system-level virtualization (a.k.a., “containers”), Wikipedia
- Linux Containers
- Container Overview, from CoreOS
- When to use containers (and when not to), by Steven Vaughan-Nichols, May 15, 2017
- The Twelve-Factor App (“The twelve-factor methodology can be applied to apps written in any programming language, and which use any combination of backing services (database, queue, memory cache, etc).”)
Docker¶
Caution
Docker and all the related entities in the Docker ecosystem are changing constantly (and at a fast pace) so many of the resources below can become outdated or obsolete in a matter of months. Docker Releases occur regularly, so pay attention to dates and version numbers in the resources below when considering implementing something, but also look for useful tips and tricks that may still be relevant over time. Also, you may want to Subscribe to Docker Weekly Newsletter to maintain your own situational awareness of updates and high-quality articles, blog posts, and videos that can help you keep up.
Docker intro¶
- Docker (“Build, ship, and run any application, anywhere.”)
- Docker: Build, Ship and Run Any App, Anywhere, by Martijn Dwars, Wiebe van Geest, Rik Nijessen and Rick Wieman
- Docker Reference Architecture: Designing Scalable, Portable Docker Container Networks, by Mark Church, Docker web site, October 18, 2016
- Docker Command Line
- Jeff Lindsay on Best Practices for Working with Containers and Docker
- GitHub docker/docs/sources/articles/dockerfile_best-practices.md (Best practices for writing Dockerfiles)
- Cooking with Docker: useful tips for beginners, by Blair Hudson
- Docker traps (and how to avoid them), by Mr. Blue Coat
- What is Docker?
- Introduction to Docker, Solomon Hykes (dotCloud founder and CTO of Docker), Twitter University
- 5 Reasons to Start Using Docker, by David Bolton, June 1, 2015
- GitHub prakhar1989/docker-curriculum (“A comprehensive tutorial on getting started with Docker! http://prakhar.me/docker-curriculum/”)
- Play with Docker classroom, Docker web site
Tips and Tricks for More Advanced Docker Use¶
- What’s eating my disk? Docker System Commands explained, by Nils De Moor, April 12, 2017
- Docker Container Anti Patterns, by Arun Gupta, October 30, 2016
- What’s New In Docker 1.12, by Emmet O’Grady, August 2, 2016
- Docker Tips And Tricks: Some tips about running and building with docker, by Luís Armando Bianchin, February 15, 2016
- Flexible Docker entrypoint scripts, by camptocamp SA, March 22, 2016
- Deploying public keys in Docker containers, by camptocamp SA, March 22, 2016
- When and why should I use apt-get update?, stackexchange, May 19, 2014
- Best practices for writing Dockerfiles, Docker.com web site
- Create a base image, Docker.com web site
- Building good docker images, by Jonathan Berghnoff, October 3, 2014
- Where are my containers? Dockerized service discovery with Consul, by Jose Luis Ordiales, January 23, 2015
- Automatic container registration with Consul and Registrator, by Jose Luis Ordiales, February 3, 2015
- Trapping signals in Docker containers, by Grigoriy Chudnov, April 7, 2015
- Docker Inspect Template Magic, by Adrian Mouat
- Wrangling Grafana and InfluxDB into a Docker image, by Si Beaumont, February 17, 2016
- An Example of Docker Multi Host Clustering Part 1, by Joshua Davis, January 20, 2016
- An Example of Docker Multi-Host Networking with Hadoop - Part 2, by Joshua Davis, February 10, 2016
- An Example of Docker Multi-Host Networking With Hadoop Part 3 - Validating your Swarm Cluster, by Joshua Davis, February 18, 2016
- See also: dockercleanup
Books and guides (and a mindmap!)¶
- The Docker Ecosystem (mindmap by Krishnan Subramanian)
- The Docker Book, by James Turnbull
- O’Reilly Docker cookbook, by Sebastien Goasguen
- GitHub how2dock/docbook (Sample code and Vagrant files for O’Reilly Docker cookbook)
- How to Use Docker on OS X: The Missing Guide, by Chris Jones
Why Docker?¶
- Sailing Past Dependency Hell With Docker, by Alex Johnson, July 2, 2015
Architecture¶
- Docker vs. VMs? Combining Both for Cloud Portability Nirvana, by Thorsten von Eicken, September 2, 2014
- Understand the architecture (v1.10), Docker web site
- Higher Order Infrastructure - Microservices on the Docker Swarm, by Nicola Paolucci, GOTO 2016
Docker in Production¶
- Why Docker is Not Yet Succeeding Widely in Production, by Simon Hørup Eskildsen, July 2015
- The Realities of Docker in Production, by Ben Ball, March 31, 2015
- Docker in Production, by Jérôme Petazzoni, September 22, 2014
- Docker deployments, by Paul Showalter & Karl Matthias, New Relic, September 10, 2014
- GitHub 6si/shipwright (The right way to build, tag and ship Docker containers.)
- GitHub newrelic/centurion (A mass deployment tool for Docker fleets)
- GitHub newrelic/check_docker (A Go Nagios check for Docker)
- GitHub newrelic/go_nagios (Go lang package for writing Nagios checks)
Docker and Development¶
- How To Cook Microservices (with Ruby spices) is a continuously updated collection of
- notes, insights and ideas about building software platforms empowered by microservices architecture with Ruby language
- Docker for Developers, by Jérôme Petazzoni, November 25, 2014
- Tips for running Docker in development, by Philip Kallberg, May 27, 2015
- GitHub wsargent/docker-cheat-sheet (Docker Cheat Sheet)
- Test, Develop, Build, Stage with Docker, by Simone Di Maulo, May 2, 2015
- A Docker Dev Environment in 24 Hours! (Part 1 of 2), by John Fiedler, October 31, 2013
- A Docker Dev Environment in 24 Hours! (Part 2 of 2), by John Fiedler, November 5, 2013
- GitHub relateiq/docker_public (Instant RelateIQ Development Environment - code from parts 1 and 2)
- Top 10 Open-Source Docker Developer Tools, by Lucas Carlson, March 5, 2014
- Executable Images - How to Dockerize Your Development Machine, by Quinten Krijger, August 29, 2015
- Development Environments with Vagrant, Docker, and Supervisord, by Tyler H.T. Cipriani, May 25, 2014
Docker and Security¶
- Docker Security, Docker web site
- Docker Ramps Up Container Security, by Jack M. Germain, May 13, 2016
- Understanding and Hardening Linux Containers (PDF), NCC Group whitepaper by Aaron Grattafiori, April 20, 2016
- Docker and High Security Microservices: A Summary of Aaron Grattafiori’s DockerCon 2016 Talk, by Daniel Bryant on August 14, 2016
- Containing and Attack with Linux Containers (PDF), Shmoocon 2016 presentation by Jay Beale, InGuardians, January 17, 2016
- Using Linux Containers to Jail Programs, by Jay Beale, Raleigh B-Sides, October 9, 2015
- GitHub docker/docker-bench-security (The Docker Bench for Security is a script that checks for all the automatable tests included in the CIS Docker 1.6 Benchmark. https://dockerbench.com)
- Introducing a *Super* Privileged Container Concept, by rhatdan, November 6, 2014
- Are Docker containers really secure?, by Daniel J Walsh, July 22, 2014
- Bringing new security features to Docker, by Daniel J Walsh, September 3, 2014
- Someone said that 30% of the images on the Docker Registry contain vulnerabilities, by jpetazzo, May 27, 2015
- Exploring Docker Volumes for Phases of Development, by Alan Kent, May 31, 2015
- Understanding Docker Security and Best Practices, Docker blog, May 5, 2015
- Container Security: Just The Good Parts, by Trevor Jay, April 29, 2015
- Docker SELinux Experimentation with Reduced Pain, by zwischenzugs, April 29, 2015
- The sad state of sysadmin in the age of containers, by Erich Shubert [pointing out some extremely poor security practices by many in DevOps]
- Docker security gets thumbs-up despite containers’ rapid rise, by Toby Wolpe, ZDNet News, January 12, 2015
- Docker security in the future, by Daniel J Walsh, March 19, 2015
- Docker Security: Best Practices for your Vessel and Containers, by Michael Boelen, January 22, 2015
- GitHub GDSSecurity/Docker-Secure-Deployment-Guidelines (Deployment checklist for securely deploying Docker)
- CoreOS vs. Project Atomic: A Review, Major Hayden, May 13, 2014
- Analysis of Docker Security, by Than Bui
- Storage Concepts in Docker: Persistent Storage, by Mark Lamourine, October 10, 2014 [Has a good discussion of SELinux labeling of directories for use by Docker containers]
- Docker’s just a bit dodgy, but ready for rollout says Gartner, by Simon Sharwood
- Snappy Security
- Launch secure LXC containers on Fedora 20 using SELinux and sVirt, by Major Hayden
- See also the SElinux, Apparmor, GRsecurity section
Orchestration¶
- Docker Orchestration workshop - Jérôme Petazzoni, Feb 21, 2016
- Part 1 (1:21:09)
- Part 2 (0:59:37)
- Parts 3 and 4 (3:20:54)
- GitHub jpetazzo/orchestration-workshop (Slides and examples for the workshop)
- Docker Orchestration & Metrics, Tiffany Jernigan, Seattle Docker Meetup, December 2016
- Docker 1.12 Swarm Mode Deep Dive Part 2: Orchestration, YouTube by Andrea Luzzardi, July 28, 2016
- Create a swarm cluster with Docker 1.12 swarm mode, by Luc, July 5, 2016
- ContainerPilot and the Autopilot Pattern, by Tim Gross, Container Summit Austin, July 19, 2016
- 3 Node Swarm Cluster in 30 seconds (Docker 1.12), by John Zaccone, July 29, 2016
- Apollo
- Capgemini Apollo: An Open Source Microservice and Big Data Platform, by Daniel Bryant, June 6, 2015
- GitHub Capgemini/Apollo, An open-source platform for cloud native applications based on Apache Mesos and Docker. http://capgemini.github.io/devops/apollo/”)
- How Apollo Uses Weave and Weave Scope, by Graham Taylor, June 30, 2015
- Atlassian Orchestration with Docker: multi-host support for the win!, by Nicola Paolucci, December 16, 2015
- DockerCon EU: Keynote on Orchestration (Docker Machine, Swarm, Compose)
- Docker 101: Dockerizing Your Infrastructure, by Stanley Lewis
- Docker and Maestro for fun, development and profit, by Maxime Petazzoni
- Docker Containers and Kubernetes with Brian Dorsey, YouTube, December 23, 2014
- CoreOS Meetup: etcd and Kubernetes, YouTube, March 19, 2015
- Docker orchestration with maestro-ng, by heisel
- GitHub signalfuse/maestro-ng (Orchestration of Docker-based, multi-host environments http://www.signalfuse.com)
- GitHub signalfx/maestro-base (Base Docker image for Maestro-enabled components http://www.signalfuse.com)
- GitHub WIZARD-CXY/docker-cassandra (Docker image for Cassandra (Maestro orchestration))
- GitHub iantruslove/docker-elasticsearch (Docker image for ElasticSearch (Maestro orchestration))
- GitHub signalfx/docker-zookeeper (Docker image for ZooKeeper (Maestro orchestration) http://www.signalfuse.com)
- GitHub signalfx/docker-kafka (Docker image for Kafka (Maestro orchestration) http://www.signalfuse.com)
- (See also A production ready Docker workflow. Part 3: Orchestration tools)
- Getting Started with CoreOS and Docker using Vagrant, by Luke Bond
- Deploying Docker Containers on a Vagrant CoreOS Cluster with fleet, by Luke Bond
- Vessel automates the setup & use of dockerized development environments
Using a private registry¶
- Docker Registry, Docker web site
- Deploying a registry server, Docker web site
- Running Secured Docker Registry 2.0, by Jaroslav Holub, April 28, 2015
- How To Set Up a Private Docker Registry on Ubuntu 14.04, by Nik van der Ploeg, October 15, 2014
- How to Secure Your Private Docker Registry, by Alex Welch, January 2, 2015 (goes with previous post)
- Setup Your Own Docker Registry on CoreOS, Vultr, May 7, 2015
- Setting Up Docker Private Registry, by beingasysadmin, January 14, 2015
- Docker: How to Use Your Own Private Registry, Twitter University, November 12, 2013
Configuration management and service discovery¶
- Demystifying Service Discovery under Docker Engine 1.12.0, by ajeetraina, July 27, 2016
- Consul (“Service discovery and configuration made easy. Distributed, highly available, and datacenter aware.”)
- Consul Part 1: Service discovery, the easy way, YouTube video by OpsForce, July 17, 2016
- Consul Part 2: Service health and templates, YouTube video by OpsForce, August 3, 2016
- Configuration management with Consul, by Michael de Jong, October 26, 2015
- Configuration management from Git to Consul, by Ryan Breen, May 8, 2015
- GitHub Cimpress-MCP/git2consul (“Mirrors the contents of a git repository into Consul KVs.”)
- GitHub Cimpress-MCP/fsconsul (“Write Consul K/Vs to the filesystem.”)
- GitHUb Cimpress-MCP/gosecret (“A Go library for encrypting and decrypting slices of byte arrays.”)
- Distributed Configuration Management and Dark Launching Using Consul, by Bill Monkman, November 26, 2014 (Has examples of creating configuration management kv store)
- GitHub kelseyhightower/confd (“Manage local application configuration files using templates and data from etcd or consul”)
- An Introduction to Using Consul, a Service Discovery System, on Ubuntu 14.04 (Part 1 of 3), by Justin Ellingwood, August 15, 2014
- How to Configure Consul in a Production Environment on Ubuntu 14.04 (Part 2 of 3), by Justin Ellingwood, August 15, 2014
- How To Secure Consul with TLS Encryption on Ubuntu 14.04 (Part 3 of 3), by Justin Ellingwood, August 15, 2014
- Understanding Modern Service Discovery with Docker, by Jeff Lindsay
- Consul Service Discovery with Docker, by Jeff Lindsay
- Automatic Docker Service Announcement with Registrator, by Jeff Lindsay
- A High Available Docker Container Platform Using CoreOS And Consul, by Mark van Holsteijn
- GitHub democracyworks/consul-coreos (“Bootstraps a Consul cluster on CoreOS using fleet and etcd”)
- Simulating service discovery with Docker and etcd, by Aaditya Talwai
- GitHub Cimpress-MCP/git2consul (“Mirrors the contents of a git repository into Consul KVs.”)
- GitHub ianbytchek/docker-coreos-ansible-toolbox (“Using Ansible with CoreOS? Use CoreOS Ansible toolbox!”)
Networking and Docker containers¶
- Docker native networking
- Docker Stacks and Attachable networks, by Alex Ellis, February 17, 2017
- Docker 1.12 Swarm Mode Deep Dive Part 1: Topology, YouTube by Andrea Luzzardi, July 28, 2016
- Splendors and Miseries of Docker Network, by Aleksandr Tarasov, November 16, 2015
- Add Docker Machine to Swarm cluster after creation, stackoverflow thread, January 4, 2016
- Docker Networking takes a step in the right direction, Docker blog, April 30, 2015
- Weave
- GitHub weaveworks/weave (“Simple, resilient multi-host Docker networking http://weave.works”)
- Multi-host Docker deployment with Swarm and Compose using Weave 0.11, by errordeveloper, May 27, 2015
- Networking Docker Containers with Weave on CoreOS, Weave web site
- Using Weave Scope Standalone to Visualize and Monitor Docker Containers, Weave web site
- Using Docker Machine and Swarm with Weave 0.10, by errordeveloper, May 6, 2015
- How to set up networking between Docker containers, by Dan Nanni, March 20, 2015
- Using Docker Machine with Weave 0.10, by Ilya Dmitrichenko, April 22, 2015
- GitHub infrabricks/powerstrip-demo (This is a TLS powerstrip weave demo, installed with docker-machine!)
- Elasticsearch, Weave and Docker, by errordeveloper, January 20, 2015
- GitHub errordeveloper/weave-demos/hello-apps/elasticsearch-js/scripts/run_elasticsearch_2_clusters.sh
- Curator: Tending your time-series indices, by Aaron Mildenstein, January 20, 2014
- GitHub elastic/curator (Curator: Tending your Elasticsearch indices”)
- Logstash how to remove old logs, by Andriy Podanenko
- Deploying and migrating an Elasticsearch-Logstash-Kibana stack using Docker Part 1, by Ryan Wallner, January 12, 2016
- Deploying and migrating an Elasticsearch-Logstash-Kibana stack using Docker Part 2, by Ryan Wallner, January 12, 2016
- I just created a Cassandra cluster that spans 3 different network domains, by using 2 simple shell commands. How cool is that?, by Yaron Rosenbaum, October 8, 2014
- Running a Weave Network on CoreOS, by errordeveloper, October 28, 2014
- How to Network Docker Containers with Weave, by Benjamin Ball, September 14, 2014
- Getting Started with Weave and Docker on CoreOS (GitHub fintanr/weavegs)
- Docker, Weave, Raspberry Pi and a bit of Networked Cloud Computing!, by Alexander Grendel, December 9, 2014
- Pipework
- GitHub jpetazzo/pipework (Software-Defined Networking tools for LXC (LinuX Containers))
- Advanced Docker Networking with Pipework, by Sam Leathers
- Flannel
- CoreOS + Layer Meetup 10/6/14: Brian “Redbeard” Harrington - Warming Up With Flannel, YouTube, October 7, 2014
- Docker Networking - CoreOS Flannel, Sreenivas Makam, January 18, 2015
- OpenVPN
- How To Run OpenVPN in a Docker Container on Ubuntu 14.04, by Kyle Manna, February 2, 2015
- Start an OpenVPN Container as a systemd managed service under CoreOS, by c-garcia, March 15, 2015
- OpenVPN in a container, by Ed Vielmetti, November 20, 2015
- Securing wifi traffic with OpenVPN and Docker, by Christopher Bunn, April 03, 2015
- Docker + Joyent + OpenVPN = Bliss, by Jérôme Petazzoni, September 10, 2013
- GitHub jpetazzo/dockvpn (“Recipe to build an OpenVPN image for Docker”)
- Integrating Proxy With Docker Swarm (Tour Around Docker 1.12 Series), by Viktor Farcic, August 1, 2016 [Good networking diagrams]
- Docker networking overview, by Filip Verloy, February 17, 2016
- Docker Swarm and experimental multihost networking with docker-machine and boot2docker, by IIkka Anttonen, July 15, 2015
- Three Solutions to Bi-directional Linking Problem in Docker Compose, by Abdelrahman Hosny, July 1, 2015
- Docker Networking Meetup - Intro to Weave/Flannel, by Dhananjay ‘DJ’ Sampath, January 23, 2015
- 5 ways Docker is fixing its networking woes, by Serdar Yegulalp, October 20, 2014
- Docker networking - IP per container, one /24 per pod(worker), by Vicente De Luca, April 30, 2015
- See also: Network Configuration
Persistent storage in Docker containers¶
- Docker: Storage Patterns for Persistence, by Karim Vaes, February 11, 2016
- Comprehensive Overview of Storage Scalability in Docker, by Jeremy Eder, September 30, 2014
- Storage Concepts in Docker: Shared Storage and the VOLUME directive, by Mark Lamourine, October 7, 2014
- Storage Concepts in Docker: Persistent Storage, by Mark Lamourine, October 10, 2014 [Has a good discussion of SELinux labeling of directories for use by Docker containers]
- Share disks through NFS on a CoreOS cluster?, stackexchange post, November 25, 2014
- Enabling and Mounting NFS on CoreOS, by Scott Lowe, February 20, 2015
- Exploring Docker Volumes for Phases of Development, by Alan Kent, May 31, 2015
- Quickly build arbitrary size Hadoop Cluster based on Docker, by KiwenLau, May 29, 2015
- See also: Hadoop section of Databases and database tools
Persistent services within Docker containers¶
- Automatically start containers, Docker web site
- Introducing dumb-init, an init system for Docker containers, by Chris K., Jan 6, 2016
- Docker All The Things: Nginx And Supervisor, by Matthew McKeen, December 14, 2013
- Running node and nginx in docker with supervisor, stackoverflow post, December 2, 2014
- Creating a Docker Container to run PHP, NGINX and Hip Hop VM (HHVM), July 15, 2014
- Roll your own Docker registry with Docker, Compose, Supervisor, and Nginx, by Philipp Wintermantel, March 18, 2014
Clustering Docker containers¶
- Using Docker Stack And Compose YAML Files To Deploy Swarm Services, by Viktor Farcic, January 23, 2017
- Running a Small Docker Swarm Cluster, Scott Lowe, March 6, 2015
- Docker Machine, Compose & Swarm, by MediaGlasses
- Clustering Using Docker Swarm 0.2.0, by Arun Gupta
- A quick overview of Docker Swarm, by Olivier Robert, February 2, 2016
Logging/monitoring activity of containers¶
- Collecting All Docker Logs with Fluentd, by Kiyoto Tamura, July 7, 2015
- Automating Docker Logging: ElasticSearch, Logstash, Kibana, and Logspout, by Nathan LeClaire, April 27, 2015
- Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker, sematext group, November 24, 2015
- Scalable Docker Monitoring with Fluentd, Elasticsearch and Kibana 4, by manu, November 21, 2014
- Performance tuning ELK stack, by Josh Reichardt, March 30, 2015
- syslog logging driver for Docker, by Mark Wolfe, May 3, 2015
- Real-time monitoring of Hadoop clusters, by Attila Kanto, October 7, 2014
- Apache Hadoop 2.6.0 on Docker, by Janos Matyas, September 15, 2014
Cleaning up (or keeping Docker images as small as possible)¶
- Reducing the size of Docker Images, by Richard Woudenberg, March 18, 2015
- Making Debian Docker Images Smaller, by Dave Beckett, April 18, 2015
- How I shrunk a Docker image by 98.8% – featuring fanotify, by Jean-Tiare LE BIGOT, April 25, 2015
- Low on disk space, cleaning up old Docker containers, by johnarce, May 20, 2014
- Squashing Docker Images, by Jason Wilder, August 19, 2014
- Optimizing Docker Images, by Brian DeHamer, July 28, 2014
- Create The Smallest Possible Docker Container, by Adriaan de Jonge, July 4, 2014
- How To Create The Smallest Possible Docker Container Of Any Image, by Mark van Holsteijn, June 30, 2015
Docker and Continuous Integration¶
- Docker Jenkins, by Yuri Kushch, February 17, 2017
- Easy CD-CI with Jenkins, Docker Swarm and Docker secrets, by KrazY CoCoon Lab, March 6, 2017
- Fully automated development environment with docker-compose, by Andrew Orsich, February 22, 2017
- Continuous Integration, Delivery or Deployment with Jenkins, Docker and Ansible by Viktor Farcic, February 11, 2015
- Deployment Pipeline using Docker, Jenkins, Java and Couchbase, by Arun Gupta, September 9, 2016
- Scaling to Infinity with Docker Swarm, Docker Compose and Consul, by Viktor Farcic, July 2, 2015
- Putting Jenkins in a Docker Container, by Maxfield F. Stewart, Riot Games Engineering blog
- GitHub maxfields2000/dockerjenkins_tutorial (“A repository for items learned in my Getting Started with Jenkins and Docker tutorial series”)
- Delivering eBay’s CI Solution with Apache Mesos – Part I, by The eBay PaaS Team, April 4, 2014
- Delivering eBay’s CI Solution with Apache Mesos – Part II, by The eBay PaaS Team, May 12, 2014
- DockerCon Video: Delivering eBay’s CI Solution with Apache Mesos & Docker, by Victor Coisne, June 23, 2014
- Disaster-proofing slaves with Docker Swarm and the CloudBees Jenkins Platform, by Tracy Kennedy, June 19, 2015
- Continuous Integration and Delivery with Docker, by Jaroslav Holub, May 28, 2015
- Jenkins, Docker Hub (Official Jenkins Docker Image)
- Running Jenkins in Docker Containers, by Peter Sellers, February 11, 2015
- Import Jenkins Configuration to a dockerized jenkins, by Allan Espinoza, January 25, 2015
- Docker in Docker with Jenkins and Supervisord, by Johan Haleby, March 14, 2015
- Jenkins in a Docker container, by by Thomas Einwaller, September 1, 2014
Operating systems for running Docker¶
CoreOS¶
- CoreOS
- Container Overview, CoreOS web site
- CoreOS Clustering, CoreOS web site
- CoreOS Cluster Architectures, CoreOS web site
- CoreOS Overview and Current Status, Slideshare by Sreenivas Makam, April 17, 2016
- etcd, CoreOS web site
- Running CoreOS on Vagrant, CoreOS web site
- Booting CoreOS via PXE, CoreOS web site
- GitHub kelseyhightower/coreos-ipxe-server (“CoreOS iPXE server”)
- Customizing Docker, CoreOS web site
- Deploying a Service Using fleet CoreOS web site
- Installing CoreOS to Disk, CoreOS web site
- Ignition: Better Machine Configuration (CoreOS Fest 2015), May 27, 2015 (has some details of boot sequence that are hard to learn)
- Lessons Learned From Building Platforms on Top of CoreOS (CoreOS Fest 2015), May 27, 2015
- 10 Lessons Learned Using CoreOS, by Gabriel Monroy (slides for CoreOS Fest 2015 talk)
- Pay attention to these bug reports (and what is said in them)
- Network settings should be set in oem cloud-config.yml (coreos/bugs #11, April 28, 2014)
- custom iptables - boot order, (coreos/bugs #58, June 26, 2014)
- docker-tcp.socket fails: Socket service docker.service already active, refusing. (coreos/coreos-vagrant bug #172, September 30, 2014)
- How does docker-tcp.socket actually enable Docker’s remote API on CoreOS?, superuser post, January 5, 2015
- How can I customize bashrc, bash_profile or profile on a CoreOS installation?, Stackoverflow post by Richard, Jun 2 2015
- GitHub https://github.com/coreos/coreos-overlay/tree/master/app-shells/bash/files/
- CoreOS: Orchestrating the Fleet, YouTube video by Brian Waldon, July 8, 2014
- Digital Ocean tutorial series: Getting Started with CoreOS
- How To Troubleshoot Common Issues with your CoreOS Servers, (part 8 of 9), September 18, 2015
- How To Secure Your CoreOS Cluster with TLS/SSL and Firewall Rules, (part 9 of 9), December 7, 2015
OpenNode OS¶
- OpenNode OS (“Lightweight bare-metal cloud OS combining Linux Containers and KVM full virtualization options into payload optimized solution.”)
- NodeFabric Host Image (“NodeFabric delivers hyperconverged database and storage solution for highly available, self-healing and load-balanced cloud services”)
RancherOS¶
- RancherOS (A minimalist distribution of Linux designed from the ground up to run Docker containers.)
- Announcing RancherOS
Project Atomic¶
- Project Atomic (Trusted Distributions, Atomic Updates)
Ubuntu “Snappy”¶
Configuration management and automated provisioning¶
- Collins (“Infrastructure management for engineers”)
- GitHub coreos/etcd (A highly-available key value store for shared configuration and service discovery)
- Brandon Philips Explains etcd, by Phil Whelan
- CoreOS: etcd 2.0, by Brandon Philips
- The Marriage of Ansible and Docker, by Zuletzt geändert von Unbekannter Benutzer, June 5, 2015
- docker, ansible and vagrant, by Wojtek Oledzki, January 1, 2015
- Ubuntu Cloud-Init Technology, YouTube video by ubuntucloud, December 8, 2010
- The Assimilation Project
- Ansible and Docker, by Ash Wilson
- OpenStack
- Building HA Clusters with Ansible and Openstack, by Remy van Elst, July 25, 2014
- GitHub RaymiiOrg/ansible (ansible/openstack-example)
- Automating Openstack with cloud init run a script on VM’s first boot, by Remy van Elst, March 11, 2015
- Provisioning IaaS Clouds with Dynamic Ansible Inventories and OpenStack Metadata, by Lukas Pustina
- GitHub ewindisch/dockenstack (OpenStack Devstack on Docker)
- OpenStack: Docker wiki section from OpenStack
- OpenStack is overkill for Docker: New tooling is necessary for effectively managing Docker at scale, by Matt Asay, August 10, 2015
- Up and Running with Docker Machine and OpenStack, by Spencer Smith, Solinea blog, July 8, 2015
- OpenStackClient and OpenStack Python SDK, youtube video by Dean Troyer, October 27, 2015
- Life Without DevStack: OpenStack Development With OSA, by Miguel Grinberg
- Agile Configuration Management – Intermezzo by Marcus Philip, Diabol
- Top 5 Open Source Linux Server Provisioning Software, by NIXCRAFT
Virtualbox¶
- Virtual Box Headless Cheatsheet: Headless Virtual Machine Install/Import and setup, by Tim Arneaud, October 9, 2012
Packer¶
- GitHub boxcutter/ubuntu (“Virtual machine templates for Ubuntu”)
- GitHub tylert/packer-build (“Packer Automated VM Image and Vagrant Box Builds”)
- Packer: In 10 minutes, from zero to bootable VirtualBox Ubuntu 12.04, by @kappataumu, September 8, 2013
Vagrant¶
- Vagrant
- How To Use Vagrant To Create Small Virtual Test Lab on a Linux / OS X / MS-Windows
- How to Create and Share a Vagrant Base Box, by George Fekete, July 17, 2014
- Vagrantfile Explained: Setting Up and Provisioning with Shell, by George Fekete, July 19, 2014
- Multi-Machine environment documentation from Hashicorp
- Using Vagrant and Ansible, Ansible documentation
- Change Insecure Key To My Own Key On Vagrant, by ermaker, November 18, 2015
- GitHub AAFC-MBB/vagrant-specify7 (“Package to launch a Specify7 instance in a Vagrant VM”)
- IP lookup for Vagrant private networking, by wamonite, July 12, 2014
- Elegant virtualization with Vagrant, by zenonharley, July 21, 2015
Ansible¶
- Ansible (web site)
- Playbooks Best Practices
- GitHub ansible/ansible-examples (“A few starter examples of ansible playbooks, to show features and how they work together. See http://galaxy.ansible.com for example roles from the Ansible community for deploying many popular applications.”)
- docker - manage docker containers
- Alternate “Best Practices” (possibly conflicting, but helpful to consider none the less)
- Laying out roles, inventories and playbooks, by Michel Blanc, July 2, 2015
- Best practices to build great Ansible playbooks, by Maxime Thoonsen, October 12, 2015
- Ansible (Real Life) Good Practices, by Raphael Campardou, March 19, 2014 (has pre-commit Git hook for
ansbile-vault
) - Lessons from using Ansible exclusively for 2 years, by Corban Raun, March 24, 2015
- 6 practices for super smooth Ansible experience, by Maxim Chernyak, June 18, 2014
- GitHub enginyoyen/ansible-best-practises (“A project structure that outlines some best practices of how to use ansible”)
- More Tips and Tricks, slideshare by bcoca, October 11, 2016 https://www.slideshare.net/bcoca/more-tips-n-tricks
- Episode #43 - 19 Minutes With Ansible (Part 1/4), Justin Weissig, sysadmincasts.com, January 13, 2015
- Episode #45 - Learning Ansible with Vagrant (Part 2/4), Justin Weissig, sysadmincasts.com, March 19, 2015
- GitHub jweissig/episode-45 (“Episode #45 - Learning Ansible with Vagrant”)
- Episode #46 - Configuration Management with Ansible (Part 3/4), Justin Weissig, sysadmincasts.com, March 26, 2015
- Episode #47 - Zero-downtime Deployment with Ansible (Part 4/4), Justin Weissig, sysadmincasts.com, April 2, 2015
- GitHub jweissig/episode-47 (“Episode #47 - Zero-downtime Deployments with Ansible (Part 4/4)”)
- Graduating Past Playbooks: How to Use Ansible When Your Infrastructure Grows Up, by Rob McQueen
- GitHub nylas/ansible-flask-example (“Example using ansible-test and wrapper roles to implement a simple flask webapp”)
- The Fedora Project ansible playbook/files/etc repository for fedora infrastructure
- How Twitter Uses Ansible, YouTube video by Ansible, May 21, 2014
- GitHub ePages-de/mac-dev-setup (“Automated provisioning of your Apple Mac (Java) development machine using Ansible”)
- Advanced Ansible concepts, gotchas, things to keep in mind…
- Security hardening for openstack-ansible, Openstack web site
- Automated Security Hardening with OpenStack-Ansible, by Major Hayden, Openstack Austin Summit, May 1, 2016
- GitHub openstack/openstack-ansible-security (“Security Role for OpenStack-Ansible http://openstack.org”)
- Templating
- Jinja2 for better Ansible playbooks and templates, by Daniel Schneller, August 25, 2014
- Ansible: “default” and “bool” filters, by dddpaul-github, November 30, 2015
- Ansible loop through group vars in template, Stackoverflow post, November 18, 2014
- Ansible loop over variables, Stackoverflow post, October 28, 2014
- Jinja2 Template Inheritence
- [jinja2] Help with blocks, Reddit /r/Ansible post by by FlowLabel
- Dynamic Inventory
- Dynamic Inventory, Ansible documentation
- Adapting inventory for Ansible, by Jan-Piet Mens
- Creating custom dynamic inventories for Ansible, by Jeff Geerling, June 11, 2015
- Writing a Custom Ansible Dynamic Inventory Script, by Adam Johnson, December 4, 2016
- Using DNS as an Ansible dynamic inventory, by Remie Bolte, January 1, 2016
- Facts vs. Variables
- Fact Caching and gathering, Ansible documentation
- Fastest way to gather facts to fact cache, Stackoverflow post, September 1, 2015
- Ansible Custom Facts, serverascode.com
- Ansible Plugins
- Ansible module development in Python - 101, by Yves Fauser, Ansible Munich Meetup - going into 2016, February 23, 2016
- Ansible: Modules and Action Plugins, by Nicholas Grisey Demengel, January 20, 2015
- An action plugin for Ansible to handle SSH host keys and DNS SSHFP records, by Jan-Piet Mens, November 3, 2012
- v2 callback plugin migration (thread), Google Groups
- Front-ends for Ansible
- Ansible Tower
- DevOps Automation – Ansible+Semaphore is Indispensable!, by Thaddeus, code-complete.com
- GitHub ansible-semaphore/semaphore (“Open Source Alternative to Ansible Tower https://ansible-semaphore.github.io/semaphore”)
- Building an Automated Config Management Server using Ansible+Flask+Redis, by deepakmdas (beingsysadmin), April 21, 2015
- rundeck (“Go fast. Be secure.”)
- stackstorm (“Event-Driven Automation”)
- GitHub StackStorm/st2 (“StackStorm (aka “IFTTT for Ops”) is event-driven automation commonly used for auto-remediation, security responses, facilitated troubleshooting, complex deployments, and more. Includes rules engine, workflow,1800+ integrations (see /st2contrib), native ChatOps and so forth.”
- New In StackStorm: Ansible Integration, by Eugen C., June 5, 2015
- Handling multi-stage or multi-deployment environments
- Multistage environments with Ansible, by Ross Tuck, May 15, 2014
- Multi-stage provisioning, by Victor Volle, Ansible Munich Meetup - going into 2016, February 23, 2016
- Ansible Tips and Tricks on ReadTheDocs
- How to Use Ansible Roles to Abstract your Infrastructure Environment, by Justin Ellingwood, February 11, 2014
- Jinja2 for better Ansible playbooks and templates, by Daniel Schneller, August 25, 2014
- Ansible - some random useful things, by David Goodwin, August 4, 2014
- Tagging, ThinkAnsible, June 4, 2014
- Scalable and Understandable Provisioning with Ansible and Vagrant, by Julien Ponge, October 15, 2013
- Alejandro Guirao Rodríguez - Extending and embedding Ansible with Python, YouTube video from EuroPython 2015
- etcd + ansible = crazy delicious, by UnicornClouds
- How I Fully Automated OS X Provisioning With Ansible, by Daniel Jaouen
- Ansible tips, by Deni Bertović, October 13, 2014
- Debugging Ansible Tasks, by Greg Hurrell, August 7, 2015
- GitHub dellis23/ansible-toolkit (“Ansible toolkit hopes to solve [some Ansible playbook] problems by providing some simple visibility tools.”)
- GitHub ks888/ansible-playbook-debugger (“A Debugger for Ansible Playbook”)
- Hacking ansible, slideshare, October 15, 2014 (“a quick presentation on ansible internals and a focus on the ease of expansion through the plugin”)
- ansible-exec: ansible-playbook wrapper for executing playbooks, by Hagai Kariti, August 26, 2014
- Using virtualenv Python in local Ansible, by Matt Behrens, April 5, 2014
- Ansible: A Simple Rollback Strategy for Roles and Playbooks, by Valentino Gagliardi, June 25, 2014
- Proposal for fixing playbooks with dynamic include problems, Ansible Development Google Group post
Storing Secrets for Development and Configuration¶
Managing Secrets In Docker Swarm Clusters, by Viktor Farcic, February 23, 2017
Docker Compose v3.1 file format now supports Docker 1.13.1 Secret Management, by ajeetraina, February 15 2017
Secrets at Scale: Automated Bootstrapping of Secrets and Identity in the Cloud, by Ian Haken, USENIX, January 30, 2017
Ask HN: In a microservice architecture, how do you handle managing secrets?, HackerNews post, January 9, 2016
- Variable separation using Ansible
- Variable File Separation, Ansible documentation
- How to open source provisioning script yet keep secrets secret? Is a two repository approach recommended?, self.ansible post by sovietmudkipz
Ansible Vault
- Safely storing Ansible playbook secrets, On Web Security blog, June 23, 2015
- Best Practices: Variables and Vaults, Ansible web site
- Ansible: Using Vault, video tutorial by ServersForHackers, Feb 16, 2015
- Managing Secrets with Ansible Vault – The Missing Guide (Part 1 of 2), by Dan Tehranian, July 24, 2015
- Managing Secrets with Ansible Vault – The Missing Guide (Part 2 of 2), by Dan Tehranian, July 24, 2015
- Ansible: How to encrypt some variables in an inventory file in a separate vault file?, Stackoverflow post by Adam Matan, May 13 2015
- GitHub Gist tristanfisher/Ansible-Vault how-to.md (“A short tutorial on how to use Vault in your Ansible workflow. Ansible-vault allows you to more safely store sensitive information in a source code repository or on disk.”)
- Encrypting Login Credentials in Ansible Vault, by Dan Tehranian, August 17, 2015
Hashicorp Vault
- Introduction to Vault, YouTube video by Seth Vargo, December 24, 2015
- Vault: A tool for managing secrets, by Mitchell Hashimoto, April 28, 2015
- 2 Key-Value Stores compared - Keywhiz (Square) & Vault (Hashicorp), YouTube video, June 12, 2015
- GitHub hashicorp/vault (“A tool for managing secrets. http://vaultproject.io”)
- Storing Secrets at Scale with HashiCorp’s Vault: Q&A with Armon Dadgar, by Daniel Bryant, September 9, 2015
- GitHub jhaals/ansible-vault (“ansible lookup plugin for secrets stored in Vault by HashiCorp”)
- Managing all your secrets with Vault - Review and Walkthrough, by Martin Rusev, January 29, 2016
Terraform¶
- GitHub hashicorp/terraform (“Terraform is a tool for building, changing, and combining infrastructure safely and efficiently. https://www.terraform.io/”)
- Seth Vargo on Hashicorp Terraform, YouTube video by Seth Vargo, February 17, 2015
- Terraform Deploying Consul Cluster in One Command, Vimeo video from HashiCorp
- How To Use Terraform with DigitalOcean, by Mitchell Anicas, September 25, 2017
- Getting Started with Terraform for Digitalocean, by Eric Wright, January 9, 2017
- Terraform Infrastructure Design Patterns, by Bart Spaans, September 14, 2015
- Using Vagrant, Docker, & Terraform to streamline your development & demo environments, YouTube video by Nikhil Vaze, October 18, 2015
- Orchestrating Docker with Terraform and Consul by Mitchell Hashimoto, YouTube video, January 7, 2015
- Guide to automating a multi-tiered application securely on AWS with Docker and Terraform, by Greg Osuri
- ansible stuffs - how to securely get SSH fingerprints/public keys from newly created AWS instances, by Bill Cawthra, September 8, 2016
Nomad¶
- Nomad, by Armon Dadgar, September 28, 2015
- Introduction to Nomad, nomadproject.io web site
- GitHub hashicorp/nomad (“A Distributed, Highly Available, Datacenter-Aware Scheduler https://www.nomadproject.io/”)
Otto¶
- Otto: Development and Deployment Made Easy ottoproject.io web site
- First look at HashiCorp’s Otto, YouTube video by Mat Schaffer, September 29, 2015
- Mitchell Hashimoto - Introducing Nomad and Otto, YouTube video, October 27, 2015
- Otto: Getting Started, ottoproject.io web site
- GitHub hashicorp/otto (“Development and deployment made easy. https://ottoproject.io”)
Automated distributed system deployment options using Ansible¶
- ansible-dims-playbooks (ansible-dims-playbooks Documenations)
- Mantl (Mantl Documentation)_
- DC/OS
- Intel Trusted Analytics Platform
- Debops
- The Fedora Project ansible playbook/files/etc repository for fedora infrastructure
Using systemd
and upstart
for services¶
- Demystifying systemd: A Practical Guide, by Ben Breard and Lennart Poettering, Red Hat Summit, April 14-17, 2014 (PDF file)
- systemd for Administrators, eBook by psankar
- Systemd Essentials: Working with Services, Units, and the Journal, Justin Ellingwood, DigitalOcean, April 20, 2015
- How To Use Systemctl to Manage Systemd Services and Units, Justin Ellingwood, DigitalOcean, February 1, 2015
- Understanding Systemd Units and Unit Files, Justin Ellingwood, DigitalOcean, February 17, 2015
- How to debug Systemd problems, Fedora Project web site
- Auditing systemd: solving failed units with systemctl, by Michael Boelen, November 24, 2014
- systemd.service — Service unit configuration, Freedesktop.org man page
- Investigating systemd errors, ArchLinux web page
- SystemdForUpstartUsers, Ubuntu web site
- Gist damncabbage/unicorn.conf (“I have the most terrible Upstart config for Unicorn. Support start, stop, restart and reload (rolling restart).”)
- How can I tell upstart to restart a service when a different service is restarted?, by Marius Gedminas, AskUbuntu post, February 13, 2013
- Logging
- Reading the System Log, CoreOS web site
- How To Use Journalctl to View and Manipulate Systemd Logs, Justin Ellingwood, DigitalOcean, February 5, 2015
- GitHub systemd/journal2gelf (“Ships new systemd journal entries to a remote destination in Graylog Extended Log Format (GELF)”)
- Sending CoreOS Logs to Loggly, by Garland Kan, October 27, 2015
- Platform logging, Deis
- Responsive infrastructure - How to setup rsyslog, elasticsearch and kibana on CoreOS and AWS (2), by Simon Dittlman, April 3, 2015
- A UDP syslog client in Python — for Windows and UNIX, by Christian Stigen Larsen, December 3, 2013
Small form-factor hardware systems¶
- Raspberry Pi
- Raspberry Pi Wiki Hub
- RPi SD cards
- How to Flash an SD Card for Raspberry Pi, by Johnny Winters
- GitHub hypriot/flash (“Command line script to flash SD card images for the Raspberry Pi”)
- Docker Pirates ARMed with explosive stuff (hypriot blog)
- Getting started with Docker on your Raspberry Pi, hypriot blog, October 13, 2015
- Let Docker Swarm all over your Raspberry Pi Cluster, by Stefan, July 3, 2015
- Heavily ARMed after major upgrade: Raspberry Pi with Docker 1.5.0, March 3, 2015
- How to use Docker Compose to run complex multi container apps on your Raspberry Pi, April 6, 2015
- Using a Raspberry Pi as a PXE boot server
- Raspberry Pi as a PXE, TFTP, NFS, proxy DHCP server, by Adam Niedzwiedzki, February 7, 2014
- Network booting machines with a PXE server running in a Docker container, by Jérôme Petazzoni, December 7, 2013
- Raspberry Pi as PXE Server, by Cody Bunch, August 19, 2014
- How To: Setup a PXE Boot Server on Debian Part 2, May 9, 2011
- Raspbian Image with Docker 1.5.0 Released for Raspberry Pi Boards, by cnxsoft
- Docker on Raspberry Pi in 4 Simple Steps, by resin.io
- Visualize your Raspberry Pi containers with Portainer or UI for Docker, by Stefan, October 31, 2016
- Cloudy with a chance of Raspberries, by Matt Williams
- Swarming Raspberry Pi - Part 1, by Matt Williams
- Let’s build a PicoCluster for Docker Swarm, Hypriot blog, Mar 23, 2016
- 5 Node PicoCluster (Raspberry PI), PicoCluster web site
- DIY 5 Node Cluster of Raspberry Pi 3s, by Nick Smith, April 2016
- OpenVPN with 2 factor authentication on the Raspberry Pi, by Mark, Coder36 blog, 15 December 2014
- UP
- Beaglebone Black
- CuBox-i
- CuBox-i Mini Computer for XBMC/Kodi player, Android TV Box and Linux
- ArchLinux is one of the available operating systems from Solid Run
- Clustering small form-factor devices
- #Building ARM cluster Part 1: Collecting , wiring and powering devices, by Mateusz Kaczanowski
- #Building ARM cluster Part 2: Create and write system image with goback!, by Mateusz Kaczanowski
- #Building ARM cluster Part 3: Docker, fleet, etcd. Distribute containers!, by Mateusz Kaczanowski
Miscellaneous Distributed System Construction¶
- GitHub uw-dims/ansible-dims-playbooks (“DIMS Ansible playbooks”)
- GitHub trustedanalytics (“Components for Trusted Analytics Platform”)
- DebOps
- The Open Home Lab Stack, by Mighty Womble, October 1, 2017
- Boot my (secure)->(gov) cloud, by Nicki Watt, August 10, 2015
Scripting in bash
¶
- Basic grammar rules of Bash, BashHackersWiki
- Commandlinefu.com
- Find Duplicate Files (based on size first, then MD5 hash)
- Recursively remove all empty directories
- GitHub google/styleguide (“Style guides for Google-originated open-source projects”)
- Command line option parsing
- GitHub kward/shflags (Automatically exported from https://code.google.com/p/shflags)
- Easy Bash Scripting With Shflags, by Steve Francia, July 8, 2011
- Using getopts in bash shell script to get long and short command line options, Stackoverflow post
- Advanced Bash scripting
- How “Exit Traps” Can Make Your Bash Scripts Way More Robust And Reliable, by Aaron Maxwell
- I set variables in a loop that’s in a pipeline. Why do they disappear after the loop terminates? Or, why can’t I pipe data to read?, BasFAQ/024
- The Ultimate Bash Array Tutorial with 15 Examples, by Sasikala on June 3, 2010
- BashGuide/Arrays
- Debugging Bash scripts
- Debug your shell scripts with bashdb, by Ben Martin, November 24, 2008
- Debugging a script, Bash Hackers Wiki
- Why does my shell script choke on whitespace or other special characters?, StackExchange post by Gilles, May 24 2014
Postfix¶
- Postfix Configuration Parameters, postfix web site
- Postfix Standard Configuration Examples, postfix web site
- Basic settings in the Postfix main.cf file, Rackspace web site, December 29, 2015
- Setting up a mail server using Postfix in 5 minutes, by Rudd-O
- Question about DNSmasq and MX records, FreeBSD forums post by danaeckel, March 12, 2013
Dell Edge Server Stuff¶
Software Engineering Best Practices¶
- GitHub linuxfoundation/cii-best-practices-badge (“Core Infrastructure Initiative Best Practices Badge https://bestpractices.coreinfrastructure.org”)
- High Cohesion, Loose Coupling, the Bojan’s Blog, April 8, 2015
- Security for developers, Mediawiki
- Security for developers/Architecture, Mediawiki
Programming in Python¶
Learning Python¶
- Learn Python the Hard Way
- Python.org (official online documentation and tutorials)
- Video tutorials
- GitHub s16h/py-must-watch (“Must-watch videos about Python: Inspired by js-must-watch. Create pull requests to add more awesome links :-)” )
- Transforming Code into Beautiful, Idiomatic Python, YouTube video by Raymond Hettinger, March 20, 2013
- Python’s Class Development Toolkit, YouTube video by Raymond Hettinger, March 20, 2013
- Raymond Hettinger - Beyond PEP 8 – Best practices for beautiful intelligible code, PyCon 2015
- Slides can be found at: https://speakerdeck.com/pyconslides/transforming-code-into-beautiful-idiomatic-python-by-raymond-hettinger-1
- Some of the code can be found here: http://msoucy.me/2015/05/pycon-2015-beyond-pep8/
- Raymond Hettinger - Super considered super!, PyCon 2015
- Dive Into Object-oriented Python, by Leonardo Giordani, April 17, 2016
- 30 Python Language Features and Tricks You May Not Know About, by Sahand Saba, May 19, 2014
General Python references¶
- Python syntax and semantics, Wikipedia
- Open Sourcing a Python Project the Right Way, by Jeff Knupp, August 16, 2013
- Cookiecutter: Project Templates Made Easy, by Daniel Roy Greenfeld, August 17, 2013
- Hacking Secret Cyphers with Python
- Logilab Python Tools
- pip
- Useful Python Functions and Features
- Code Academy - Learn to code for free
- Check IO - Learn Python by playing a game
- Anaconda (Completely free enterprise-ready Python distribution for large-scale data processing, predictive analytics, and scientific computing)
Advanced Python Programming¶
- How to Create A New Python Module (and deploy it using pip), by liranbh, January 2017
- A guide to Python’s function decorators, by Ayman Farhat, January 2014
- Intro to some Python dependencies of OpenStack, by Run for the Hills
- Easy, clean, reliable Python 2/3 compatibility, by Python Charmers Pty Ltd
- Easy Signal Handling for Python Daemons, Jeff Hull, August 30, 2012
- The Top Mistakes Developers Make When Using Python for Big Data Analytics, by Karolina Alexiou
- Don’t Slurp: How to Read Files in Python, by mssaxm, September 27, 2013 by mssaxm
- Python: Multiprocessing large files, by Nick, March 29, 2012
- Dynaconf - Let your settings to be Dynamic, by Bruno Rocha, February 14, 2016
- Microservices with Python, RabbitMQ and Nameko, by Bruno Rocha, March 4, 2016
- How to use findall, finditer, split, sub, and subn in Regular Expressions, by mamikon, 2013
- Pandas
- Multi-Processing With Pandas, by Gouthaman Balaraman, November 19, 2014
- Pandas: Data Analysis with Python, by Carsten Schnober
- Exploring U.S. Traffic Fatality Data (using Python and Pandas), by Ben Van Dyke, December 16, 2015
- Adventures in Optimizing Text Processing (Lessons learned while post-processing 1.75 billion lines of Hadoop output), by Michael Richman, January 21, 2014
- Git revision numbers for setuptools packages, by Christopher Berner, February 26, 2012
- A template for Python packages, including a sound setup.py and Makefile for tagging releases by hcarvalhoalves
- How To Quickly Compute The Mandelbrot Set In Python, by Jean Francois Puget, December 28, 2015
- A Globals Class pattern for Python, by Steve Ferg, October 20, 2010
- GitHub Yelp/undebt (“A fast, straightforward, reliable tool for performing massive, automated code refactoring”)
- Create, use, and remove temporary files securely, Openstack web site
Command line programs in Python¶
- Writing Awesome Command-Line Programs in Python, YouTube video from EuroPython 2014, July 24, 2014
- GitHub openstack/cliff (“Command Line Interface Formulation Framework http://openstack.org”)
- [openstack-dev] [oslo][openstackclient] transferring maintenance of cliff to the OpenStack client team, by Doug Hellmann, February 26, 2015
- Oslo (OpenStack Common Libraries)
- GitHub OpenStack Oslo repos
- cmd2
- Python Apps the Right Way: entry points and scripts, by Chris Warrick, September 15, 2014
- worked examples of argparse and python logging, Gist by olooney
- Accessing value from either os.environ or argparse, Stackexchange post
Python IDEs, shells, and debuggers¶
- PyCharm
- PyCharm Overview, YouTube video by JetBrains, May 5, 2014
- 9 Reasons You Should be Using PyCharm, by Michael Kennedy, November 19, 2015
- How to Get Started with PyCharm and Have a Productive Python IDE, by Pedro Kroger
- Productive Python with PyCharm, by Paul Everitt, 2015
- Install PyCharm on Ubuntu Linux, by Akbar S. Ahmed, February 10, 2015
- Debug Support in Interactive Python Console, YouTube video by JetBrains, May 16, 2014
- Configure PyCharm to use virtualenv, by Akbar S. Ahmed, February 10, 2015
- An Epic Review of PyCharm 3 from a Vim User’s Perspective, by Andrew Brookins
- How do not waste time in PyCharm, YouTube vide by Paqmind, October 18, 2014
- IPython
- full: April 2014 Meetup - Matt Boehm and Python debuggers, DCPython tutorial session
Python virtual environments¶
- virtualenv source code
- Virtual Environments, Python Guide docs
- Python Power Tools: virtualenv video by Tuts+ Code
- Python Power Tools: virtualenvwrapper video by Tuts+ Code
- Python: pyenv, pyvenv, virtualenv – What’s the Difference?, by Abu Ashraf Masnun, April 10, 2016
- Reverse-engineering Ian Bicking’s brain: inside pip and virtualenv, presented by Carl Meyer at PyCon 2011, Atlanta
- Configure PyCharm to use virtualenv, by Akbar S. Ahmed, February 10, 2015
- Development Environment in Python - Part 1: Virtualenv and Pip, by Michael Herman
- Development Environment in Python - Part 2: Git and Github, by Michael Herman
- Running IPython cleanly inside a virtualenv
- Automatically load a virtualenv when running a script, stackoverflow, May 15, 2014
- Virtualenv’s bin/activate is Doing It Wrong, datagrok / gist:2199506, July 3, 2014
- Virtualenv the Docker way, by Justyna Ilczuk, Nov 16, 2014
Testing and Test Automation¶
- Ned Batchelder: Getting Started Testing - PyCon 2014, PyCon 2014
- Robot Framework
- How To Configure PyCharm for Robot Framework, YouTube video by KWoodScreencast, July 2014
- Robot Framework Tutorial 2- First Script
- GitHub sstephenson/bats (“Bats: Bash Automated Testing System”)
- Testing Bash scripts with Bats, by Spike Grobstein, September 7, 2013
- How to use Bats to test your command line tools, engineyard, April 29, 2014
- GitHub duggan/pontoon/test/bats
- GitHub jenkinsci/docker/tests
- More Projects using Bats
Sphinx + reST¶
- Sphinx
- Sphinx Style Guide
- Sphinx Tutorial v0.1 talk by Brandon Rhodes from PyCon 2013
- Documenting Your Project Using Sphinx
- Easy and beautiful documentation with Sphinx
- reStructuredText tool support
- Quick reStructuredText
- ReadTheDocs
- Don’t be afraid to commit
- Sphinx Autodoc Tutorial for Dummies
- How to generate sphinx documentation for python code running in an embedded system
- Problems with StructuredText
- Automatically Generating Documentation for All Python Package Contents, stackoverflow.com
- sphinx.ext.intersphinx – Link to other projects’ documentation
- GitHug Gist shimizukawa/conf.py (Sphinx: Link to outer non-sphinx document by using intersphinx. https://groups.google.com/d/topic/sphinx-users/P_FolrZVoNg/discussion)
- Python Continuous Documentation
- sphinxcontrib-bibtex
- sphinx-contrib at Bitbucket
- StackOverflow thread: Using Sphinx to write personal websites and blogs
- ABlog for Sphinx
Git¶
Git tips from the trenches, February 1, 2014
Learn Git Branching (really cool animated tutorial, visualizing Git concepts)
Git Interactive Rebase, Squash, Amend and Other Ways of Rewriting History, by Tute Costa, November 3, 2014
Git: Using topic branches and interactive rebasing effectively, by kumar303
Git - When to Merge vs. When to Rebase, by Derek Gourlay, February 15, 2016
- Viewing Git history
- 2.3 Git Basics - Viewing the Commit History, the Git Book
- gitk
- GitHub esc/git-big-picture
- Visualizing branch topology in git, Stackoverflow post by Benjol, December 3, 2009
- Git merging and conflict resolution
- kdiff3
- meld
- Git merging using Meld, Stackoverflow thread, June 21, 2012
- Three way git merging with meld, by Lukáš Zapletal, September 17, 2012
- Painless Merge Conflict Resolution in Git, by Kevin Wu Won, September 14, 2010
- Oops. I didn’t mean to do that… (Ugh!)
- How to undo (almost) anything with Git, GitHub blog post by jaw6, June 8, 2015
- Undo Almost Anything with Git webinar, YouTube video by Peter Bell and Michael Smith, February 11, 2014
- Undoing Changes, Git tutorial, attlasian.com
- GitHub for hosting your Git repositories
- Fork A Repo (GitHub Help)
- Forking Projects, (GitHub Guides)
- Syncing a fork (GitHub Help)
- Pushing to a remote (GitHub Help)
- git-flow utilities to follow the Vincent Dreisen branching workflow
- Getting Started – Git-Flow, yakiloo
hub (Git front end for GitHub)
- HubFlow (GitFlow for GitHub)
- Using GitFlow with GitHub (Note: At this time, this workflow is designed for developers who belong to the same GitHub organisation. Although you are welcome to use it for opensource projects, it is designed for companies like DataSift who are using private repos on GitHub.)
- Versioning
- GitHub Flow Like a Pro with these 13 Git Aliases
- Development flow with git flow, github, by Vysakh Sreenivasan
- GitFlow: safely merge develop changes to a feature branch, stackoverflow thread
Comparing Workflows, Atlassian Tutorials
- bumpversion (can flexibly increment version numbers and commit/tag in Git)
- bumpversion screencast showing bumpversion in action
- A Python Versioning Workflow With Bumpversion
- PEP 440 – Version Identification and Dependency Specification (Python versioning specification that addresses several limitations of the previous attempt at a standardized approach to versioning, as described in PEP 345 and PEP 386.)
- Git aliases and advanced tricks
- Includes in your git config, by Aurélien’s room, December 5, 2015
- Using git’s include for private configuration information (like github tokens)
- Protect Your Private Data Using Gitconfig Include Directive, Basetta.Bz, January 28, 2013
- One weird trick for powerful Git aliases, by Nicola Paolucci, October 3, 2014
- GitHubGist mwhite/git-aliases.md (The Ultimate Git Alias Setup)
- Must Have Git Aliases: Advanced Examples, by @durdn
- GitHubGist mcxiaoke/.gitconfig
- What are your favorite Git aliases?, Reddit
r/git
post, - SCM Breeze (GitHub Gist ndbroadbent/scm_breeze (“Adds numbered shortcuts to the output git status, and much more http://madebynathan.com/2011/10/18/git-shortcuts-like-youve-never-seen-before/”))
- Encrypting some/all of a Git repository
- encrypted repositories?, Gmane thread
- GitHub AGWA/git-crypt (“Transparent file encryption in git https://www.agwa.name/projects/git-crypt/”)
- GitHub Gist shadowhand/encrypted-git-repo.md (“Transparent Git Encryption”)
- 7.14 Git Tools - Credential Storage, Git book
- Mac OS X and Windows issues with Git
- Git on Mac OS X: Don’t ignore case!, by Howard Lewis Ship, July 28, 2010
- How do I commit case-sensitive only filename changes in Git?, Stackoverflow, July 16, 2013
- Case sensitivity in Git, Stackoverflow, January 18, 2012 (has solution for Mac OS X users)
version.py Python script for getting version number from Git to use in
setup.py
- Advanced tasks (DANGER)
- How to delete files permanently from your local and remote git repositories, by Anoopjohn, February 20, 2014
- GitHub aaronzirbes/shrink-git-repo.sh (“This script will help you remove large files from your git repo history and shrink the size of your repository.”)
- How to Shrink a Git Repository, by Steve Lorek, May 11, 2012
Agile/Scrum¶
- Agile methodology
- Scrum methodology
- Learn Scrum in 7 Minutes!, by techexcel, (YouTube video)
- Scrum and Kanban, Kanbantool,
- Introduction to Scrum - CollabNet Scrum Training Part 1, YouTube
- Jira Agile
- Humor about Agile/Scrum
- I Need Agile Methodology, YouTube [Play on the hilarious iPhone vs. Evo video. “NSFW because of language (though that’s probably relative to your work environment).”]
- Another perspective on SCRUM, YouTube
Continuous Integration and DevOps¶
- Continuous Integration
- Jenkins CI
- Pipeline as Code with Jenkins, Jenkins web sit
- GitHub Organization Folder Plugin, Jenkins Plugin web site
- Pipeline Stage View Plugin, Jenkins Plugin web site
- JENKINS_HOME directory
- Distributed builds
- Automatically Generating Jenkins Jobs, by Jim Graf, March 11, 2016
- Simple Continuous Integration / Deployment With Jenkins, by Randall Degges
- 7 Habits of Highly Effective Jenkins Users, by Andrew Beyer, July 28, 2011
- A Back Up And Restore Recipe for Jenkins Metadata, by chris, March 16, 2015
- How To Deploy Jenkins Completely Pre-Configured - Automating Jenkins, by Emmet O’Grady, October 11, 2016
- Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile, by Wilson Mar, August 3, 2016
- Creating Jenkins pipelines with Ansible, part 1, by Joel Wilsson, Augusst 6, 2016
- Creating Jenkins pipelines with Ansible, part 2, by Joel Wilsson, August 10, 2016
- GoCD (“Simplify Continuous Delivery”)
- What is DevOps?
- Implementing Continuous Delivery: Jenkins or Bamboo?, by Francis Adanza, April 12, 2016
Kanban method¶
- Kanban (Development), Wikipedia
- Kanban Applied to Software Development: from Agile to Lean, by Kenji Hiranabe, January 14, 2008
- Kanban vs. Scrum: Kanban is NOT for Software Development, but Scrum is!, by charlesbradley, February 4, 2013
- How To Get Started With Kanban In Software Development, by Derick Bailey, August 5, 2009
- Adventures in Lean, by Derick Bailey, November 19, 2008
Software Engineering Project Management¶
- 44 engineering management lessons, by Slava Akhmechet, October 3, 2014
Messaging using AMQP¶
- AMQP and Beyond: Messaging by Extending RabbitMQ, by Tony Garnock-Jones
- kombu - AMQP Messaging Framework for Python, Version: 1.0.5
- Postage - a RabbitMQ-based Component Python Library
- Federated Queues, RabbitMQ web site
- Distributed RabbitMQ brokers, RabbitMQ web site
- Alvaro Videla - Building a Distributed Data Ingestion System with RabbitMQ, YouTube, Jul 16, 2014
- Distributed log aggregation with RabbitMQ Federation, by Alvaro Videla, December 17, 2013
- Routing Topologies for Performance and Scalability with RabbitMQ, by Helena Edelson, April 1, 2011
- Configuring SSL for RabbitMQ, by Richard, January 27, 2013
- Securing the RabbitMQ Management Console with SSL, by Richard, January 27, 2013
Software licensing and Open Source¶
tl;dr Legal quick references
How to Make Money from Open Source, by pieterh, September 18, 2012
How To Capture an Open Source Project, by pieterh, October 21, 2013
OSI Hopes To Decrease Number of Licenses, Slashdot (Bruce Parens, creator of the Open Source definition, hopes to get back to just three Open Source licenses from which to chose, rather than over 30)
Understanding Open Source Software, by Red Hat’s Mark Webbink, Esq., GROKLAW
BSD and GPL Licensing, wikipedia
The Open Source Initiative has a list of open source licenses
Open Source Licensing: Links and Resources, wasabisystems.com
GPL v3 drafting process (Current Release: January 16, 2006: Next Release: Anticipated May or June 2006)
Trademark and OSS, Software Pluralism, University of Washington Law School
Derivative Works, Software Pluralism, University of Washington Law School
Patent Risks, Software Pluralism, University of Washington Law School
Dual-licensing model
- Dual Licensing in Open Source Software Industry, by Mikko Välimä, January, 2003
- Dual Licensing: Having Your Cake and Eating It Too, by Philip H. Albert, LinuxInsider, November 16, 2004
- Dual Licensing, OSS Watch
- MySL AB FLOSS Exception
Open Source licenses in the courts
- Litigation in Open Source, by Jason B. Wacha, Vice President of Corporate Affairs and General Counsel, MontaVista Software, Inc., August 8, 2004
- Linksys routers caught up in open source dispute
- Is Linksys shirking the GPL? (Maybe not.), O’Reilly OnLAMP.com
- Linux’s Hit Men, by Daniel Lyons, Forbes.com, October 14, 2003 [Discusses the problems of Cisco Systems’ purchase of Linksys (manufacturer of home routers), which had GPL’d code on the Broadcom chips it used.]
- GPL: IT’S THE LAW, by Nancy Cohen