Docker at Spotify

Nov 8, 201334 likes11,718 views

Spotify uses Docker and Helios to deploy over 100 backend services across 5000 servers. Docker provides repeatable deployments by running the same tested image in production. Helios ensures Docker containers are deployed and running as intended across servers. Spotify is moving more services to Docker and has its first Docker-based service in production using Helios to deploy containers across its infrastructure.

Docker at Spotify
!
Rohan Singh
@rohansingh

Spotify
“Music for every moment”
!
40+ million active users
!
20+ million songs available
(we’re in 55+ countries, differs by market)
!
Live in the US for 2 years
2

What are we working with?
!
100+ distinct backend services
!
5000+ production servers in 4 sites worldwide
!
Almost 300 servers per ops engineer
3

Build Debian packages
!
Requisition new hardware
!
Use combination of Puppet
and internal tooling (deployify)
5

old way:
SSH to a bunch of machines,
apt-get install || upgrade
!
deployify:
Uses fabric, which just SSH’s,
runs apt-get install || upgrade
6

Requisitioning new hardware is slow
!
Hardware utilization is low
!
We’ve automated and improved,
but still too slow
8

Deployments often fail…
!
…though usually only partially
9

Debian packaging is its own special hell
So many damn files!
!
debhelper
!
Distro & version specific
!
apt-get update
11

Deployments that are:
!
Repeatable
!
Straightforward
!
Fault-tolerant
13

Why Docker?
!
It gives us “repeatable”,
right out of the box
!
!
And it’s fairly straightforward, too!
14

Repeatability?
!
!
The code you test isn’t necessarily
the code that runs in prod
15

Repeatability — Docker
!
Build an image
!
Run tests against the image
!
Run the tested image in prod
16

Straightforward
!
Just one Dockerfile
!
Pretty simple syntax
!
Immutable images — easy to reason about
17

Fault-tolerance?
Even if your code is perfect,
deployments might fail
!
Puppet or Debian repos are down
!
Network issues
!
The machine dies
18

Fault-tolerance — Docker
Just one thing has to work —
pulling & running an image
!
If it fails, just try again
!
No machines left in weird states*
!
!
*(in theory)
19

Deploying Docker containers across
our entire fleet of servers
!
Some way to deal with Docker or container failures
!
That’s pretty much it
21

Looking around for a Docker platform
Several really feature-full offerings
!
Pretty opinionated and all-encompassing
!
All we need is reliably deploying containers
to a fleet of Docker hosts
22

Makes sure your containers are deployed & running
exactly where you want them…
!
…and that’s pretty much it
!
!
github.com/spotify/helios
24

What’s in the box?
Helios master
frontend that you talk to
!
Helios agent
runs alongside each Docker instance
!
Zookeeper
!
Straightforward CLI
25

How does it work?
Create a job —
tell Helios which Docker image to deploy
!
Deploy the job —
tell Helios where to run the Docker containers
26

Helios is just the beginning
!
Brought up plenty of Helios+Docker agents
in our private cloud for our devs
!
Integrated Helios with JUnit —
write JUnit tests run against Docker containers
29

This week —
first Helios+Docker-based service in production
!
!
In the future —
Containerizing more of our infrastructure
31

Give it a go
Open source:
github.com/spotify/helios
!
Try it out:
git clone https://github.com/spotify/helios.git
&& cd helios && vagrant up
!
!
!
Come play with us (spotify.com/jobs)
32

There is growing interest in running Apache Spark natively on Kubernetes (see https://github.com/apache-spark-on-k8s/spark). Spark applications often access data in HDFS, and Spark supports HDFS locality by scheduling tasks on nodes that have the task input data on their local disks. When running Spark on Kubernetes, if the HDFS daemons run outside Kubernetes, applications will slow down while accessing the data remotely. This session will demonstrate how to run HDFS inside Kubernetes to speed up Spark. In particular, it will show how Spark scheduler can still provide HDFS data locality on Kubernetes by discovering the mapping of Kubernetes containers to physical nodes to HDFS datanode daemons. You’ll also learn how you can provide Spark with the high availability of the critical HDFS namenode service when running HDFS in Kubernetes.

DevOps - Interview Question.pdfMinhTrnNht7

This document contains questions and answers related to DevOps concepts. It begins with definitions of DevOps and explains that DevOps aims to automate infrastructure and integrate development and operations teams. Key DevOps principles like infrastructure as code, continuous integration, deployment and monitoring are outlined. Popular DevOps tools like Git, Jenkins, Ansible, Docker and Nagios are listed. The document also includes questions on version control systems, Git, Ansible, Docker, Scrum methodology and more DevOps related topics.

How to test infrastructure code: automated testing for Terraform, Kubernetes,...Yevgeniy Brikman

DevOps IntroductionRobert Sell

Linux programming lecture_notesIMRAN KHAN

Linux is a free and open-source operating system assembled under a collaborative development model. The Linux kernel was first released in 1991 and has since been ported to run on various hardware platforms. It is widely used today for servers, supercomputers, embedded systems like Android, and desktop systems. Common Linux distributions include desktop environments like GNOME or KDE and include applications like Firefox, LibreOffice, and GIMP. Programming languages widely supported on Linux include C, C++, Java, Python, and Perl. The document then discusses advantages of Linux like low cost, stability, flexibility, security, and its open source nature.

Enhance your multi-cloud application performance using Redis Enterprise P2Ashnikbiz

This document provides an overview of Redis Enterprise. It discusses how Redis Enterprise is an in-memory multi-model database built on open source Redis that supports high-performance operational, analytics, and hybrid use cases. It offers deployment options including cloud, on-premises, and Kubernetes and supports a wide range of modern use cases including caching, transactions, streaming, messaging, and analytics workloads. Redis Enterprise provides features like high availability, security, and support for Redis modules to extend its capabilities.

Rundeck OverviewRundeck

Rundeck is an open source automation platform that allows users to define, schedule, and execute jobs across multiple servers to automate system administration tasks. It was started in 2010 and has over 100 contributors. Rundeck provides a web interface, API, and CLI to define workflows and orchestrate tasks. It supports plugins for popular DevOps tools and can integrate with other systems like ITSM tools. Rundeck Enterprise provides additional features like high availability, security controls, and support.

CI/CD Best Practices for Your DevOps JourneyDevOps.com

The journey to realizing DevOps in any organization is fraught with a number of obstacles for developers and other stakeholders. These challenges are often caused by key CI/CD practices being misunderstood, partially implemented or even completely skipped. Now, as the industry positions itself to build on DevOps practices with a Software Delivery Management strategy, it’s more important than ever that we implement CI/CD best practices, and prepare for the future. Join host Mitchell Ashely, and CloudBees’ Brian Dawson, DevOps evangelist, and Doug Tidwell, technical marketing director, as they explore and review the CI/CD best practices which serve as your stepping stones to DevOps and a successful Software Delivery Management strategy. The webinar will cover CI/CD best practices including: Containers and environment management Continuous delivery or deployment Movement from Dev to Ops By the end of the webinar, you’ll understand the key steps for implementing CI/CD and powering your journey to DevOps and beyond.

Application Performance Monitoring (APM)Site24x7

초보자를 위한 Git & GitHubYurim Jin

Introduction To DevOps | Devops Tutorial For Beginners | DevOps Training For ...Simplilearn

This presentation on "Introduction to DevOps" will help you understand what is waterfall model, what is an agile model, what is DevOps, DevOps phases, DevOps tools and DevOps advantages. In traditional software development lifecycle, there is a lot of gap between development and operations team. DevOps addresses the gap between developers and operations. The development team will submit the application to the operations team for implementation. Operations team will monitor the application and provide relevant feedback to developers. According to DevOps practices, the workflow in software development and delivery is divided into 8 phases, Now, let us get started and understand these 8 phases in DevOps. Below topics are explained in this "Introduction to DevOps" presentation: 1. Waterfall model 2. Agile model 3. What is DevOps? 4. DevOps phases 5. DevOps tools 6. DevOps advantages Simplilearn's DevOps Certification Training Course will prepare you for a career in DevOps, the fast-growing field that bridges the gap between software developers and operations. You’ll become an expert in the principles of continuous development and deployment, automation of configuration management, inter-team collaboration and IT service agility, using modern DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios. DevOps jobs are highly paid and in great demand, so start on your path today. Why learn DevOps? Simplilearn’s DevOps training course is designed to help you become a DevOps practitioner and apply the latest in DevOps methodology to automate your software development lifecycle right out of the class. You will master configuration management; continuous integration deployment, delivery and monitoring using DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios in a practical, hands-on and interactive approach. The Devops training course focuses heavily on the use of Docker containers, a technology that is revolutionizing the way apps are deployed in the cloud today and is a critical skillset to master in the cloud age. Who should take this course? DevOps career opportunities are thriving worldwide. DevOps was featured as one of the 11 best jobs in America for 2017, according to CBS News, and data from Payscale.com shows that DevOps Managers earn as much as $122,234 per year, with DevOps engineers making as much as $151,461. DevOps jobs are the third-highest tech role ranked by employer demand on Indeed.com but have the second-highest talent deficit. 1. This DevOps training course will be of benefit the following professional roles: 2. Software Developers 3. Technical Project Managers 4. Architects 5. Operations Support 6. Deployment engineers 7. IT managers 8. Development managers Learn more at: https://www.simplilearn.com/

Linux admin interview questionsKavya Sri

This document contains interview questions for a Linux administrator role. It includes questions about shell scripting, system administration tasks, networking, and more. Some example questions are how to take input in a shell script, write a script to convert file path slashes, and explain the differences between UDP and TCP. The document provides technical questions to assess a candidate's Linux knowledge and experience.

Git Tutorial For Beginners | What is Git and GitHub? | DevOps Tools | DevOps ...Simplilearn

The document provides information on version control systems and Git concepts like distributed version control, forking and cloning repositories, adding collaborators, branching and merging in Git. It includes step-by-step instructions for setting up a Git demo to showcase creating a repository, adding and committing files, connecting to a remote repository on GitHub, forking a project and cloning it locally, creating and switching branches, adding a collaborator, pulling changes, and merging branches.

User and groups administratorAisha Talat

Linux allows multiple users to access the system simultaneously. Users are uniquely identified by their UID, and can be regular users or superusers. Superusers have full access while regular users have limited access. The system administrator manages users and groups. Users can be created with the useradd command and assigned to primary and supplementary groups. User properties like login, UID, home directory and shell are set during creation. Users can be modified, locked, unlocked and deleted using related commands. Groups organize users and are managed using groupadd, groupmod, and groupdel. Permissions allow controlling access for users and groups.

Linux file systemMd. Tanvir Hossain

The document discusses Linux file systems. It describes that Linux uses a hierarchical tree structure with everything treated as a file. It explains the basic components of a file system including the boot block, super block, inode list, and block list. It then covers different types of file systems for Linux like ext2, ext3, ext4, FAT32, NTFS, and network file systems like NFS and SMB. It also discusses absolute vs relative paths and mounting and unmounting filesystems using the mount and umount commands.

Intro to open source observability with grafana, prometheus, loki, and tempo(...LibbySchulze

This document provides an introduction to open source observability tools including Grafana, Prometheus, Loki, and Tempo. It summarizes each tool and how they work together. Prometheus is introduced as a time series database that collects metrics. Loki is described as a log aggregation system that handles logs at scale without high costs. Tempo is explained as a tracing system that allows tracing from logs, metrics, and between services. The document emphasizes that these tools can be run together to gain observability across an entire system from logs to metrics to traces.

Ansible TowerKnoldus Inc.

How to Handle DEV&TEST&PROD for Oracle Data IntegratorGurcan Orhan

Most of us have development teams apart from test and operation teams using the different repository environments. And there are generally 3 different ODI installations and repositories which each of the teams use separately. Chaos is usually expected and happened who will test which development and what to deploy into production. In this session hear how ODI can handle your development hierarchy with ease of usage and in simplified/synchronized way for successful deployments. A simple project will be built up and will be enlarged to enterprise level step by step.

DevOpsGehad Elsayed

Controllo di versione e GitMarcello Missiroli

SRE vs DevOpsLevon Avakyan

Version Control & GitCraig Smith

This document provides an overview of version control and the distributed version control system Git. It discusses the history and benefits of version control, including backup and recovery, synchronization, undo capabilities, and tracking changes. Key aspects of Git are explained, such as branching and merging, the fast and efficient nature of Git, and how it allows for cheap local experimentation through branches. The document demonstrates Git workflows and commands and provides resources for further information.

ObservabilityMartin Gross

Observability refers to the ability to infer the internal state of a system from its external outputs. It is a property of the system, not an action like monitoring. For a system to be observable, it must externalize its state through logs, metrics, and events. Improving observability involves monitoring all components of an application from the front-end to backend services to infrastructure. Common metrics include requests processed, errors encountered, and response times for applications as well as CPU usage, disk I/O, and network traffic for infrastructure. Observability extends monitoring by helping understand why a system is not working in addition to whether it is working.

RedHat Certification Trackssuser113f26

The document outlines various Red Hat certification tracks covering topics like Ansible automation, OpenStack, OpenShift, Linux system administration, Java application development, virtualization, storage, and more. It lists the courses, exams, and certifications available in each track, along with recommended prerequisites. The certifications include Red Hat Certified System Administrator, Red Hat Certified Engineer, Red Hat Certified Specialist, and others.

OpenStack Administration by Mobarak Hossain Group Organizer BangladeshMobarak Hossain

This document provides an overview of an OpenStack training organized by Mobarak Hossain. It includes details about Mobarak Hossain's background and experience working for various companies in software and telecom. It also provides a brief history of cloud computing and OpenStack, describing some of the major OpenStack services and concepts. The document concludes with information about hands-on sessions for installing and using OpenStack.

What is Git | What is GitHub | Git Tutorial | GitHub Tutorial | Devops Tutori...Edureka!

This DevOps Tutorial on what is Git & what is GitHub ( Git Blog series: https://goo.gl/XS1Vux ) will let you know all about Version Control System & Version Control Tools like Git. You will learn all the Git commands to create repositories on your local machine & GitHub, commit changes, push & pull files. Also you will get your hands on with some advanced operations in Git like branching, merging, rebasing etc. Below are the topics covered in this tutorial: 1. Version Control Introduction 2. Why version Control? 3. Version Control Tools 4. Git & GitHub 5. Case Study: Dominion enterprises 6. What is Git? 7. Features of Git 8. What is a Repository? 9. Git Operations and Commands

Intro to open source - 101 presentationJavier Perez

This document provides an overview of open-source software and how to get started with it. It discusses the history of open-source software dating back to 1955. It defines key open-source concepts like licenses, roles, and best practices for contributing. It also highlights the large open-source ecosystems existing today and the top companies contributing to open-source. The document aims to address common questions or concerns about open-source software.

Immutable infrastructure with Docker and containers (GlueCon 2015)Jérôme Petazzoni

"Never upgrade a server again. Never update your code. Instead, create new servers, and throw away the old ones!" That's the idea of immutable servers, or immutable infrastructure. This makes many things easier: rollbacks (you can always bring back the old servers), A/B testing (put old and new servers side by side), security (use the latest and safest base system at each deploy), and more. However, throwing in a bunch of new servers at each one-line CSS change is going to be complicated, not to mention costly. Containers to the rescue! Creating container "golden images" is easy, fast, dare I say painless. Replacing your old containers with new ones is also easy to do; much easier than virtual machines, let alone physical ones. In this talk, we'll quickly recap the pros (and cons) of immutable servers; then explain how to implement that pattern with containers. We will use Docker as an example, but the technique can easily be adapted to Rocket or even plain LXC containers.

Delivering Personalized Music DiscoveryRohan Singh

This document summarizes Spotify's approach to delivering personalized music recommendations to millions of users. Spotify collects data from over 24 million active users across 55 countries to generate recommendations for each user daily based on their listening history and behaviors. This data is stored in Apache Hadoop and recommendations are generated using this data and stored in Apache Cassandra databases worldwide. The recommendations are then served to users on Spotify's discovery page to suggest new music.

More Related Content

What's hot (20)

CI/CD Best Practices for Your DevOps JourneyDevOps.com

Application Performance Monitoring (APM)Site24x7

초보자를 위한 Git & GitHubYurim Jin

Introduction To DevOps | Devops Tutorial For Beginners | DevOps Training For ...Simplilearn

Linux admin interview questionsKavya Sri

Git Tutorial For Beginners | What is Git and GitHub? | DevOps Tools | DevOps ...Simplilearn

User and groups administratorAisha Talat

Linux file systemMd. Tanvir Hossain

Intro to open source observability with grafana, prometheus, loki, and tempo(...LibbySchulze

Ansible TowerKnoldus Inc.

How to Handle DEV&TEST&PROD for Oracle Data IntegratorGurcan Orhan

DevOpsGehad Elsayed

Controllo di versione e GitMarcello Missiroli

SRE vs DevOpsLevon Avakyan

Version Control & GitCraig Smith

ObservabilityMartin Gross

RedHat Certification Trackssuser113f26

OpenStack Administration by Mobarak Hossain Group Organizer BangladeshMobarak Hossain

What is Git | What is GitHub | Git Tutorial | GitHub Tutorial | Devops Tutori...Edureka!

Intro to open source - 101 presentationJavier Perez

CI/CD Best Practices for Your DevOps JourneyDevOps.com

Application Performance Monitoring (APM)Site24x7

초보자를 위한 Git & GitHubYurim Jin

Introduction To DevOps | Devops Tutorial For Beginners | DevOps Training For ...Simplilearn

Linux admin interview questionsKavya Sri

Git Tutorial For Beginners | What is Git and GitHub? | DevOps Tools | DevOps ...Simplilearn

User and groups administratorAisha Talat

Linux file systemMd. Tanvir Hossain

Intro to open source observability with grafana, prometheus, loki, and tempo(...LibbySchulze

Ansible TowerKnoldus Inc.

How to Handle DEV&TEST&PROD for Oracle Data IntegratorGurcan Orhan

DevOpsGehad Elsayed

Controllo di versione e GitMarcello Missiroli

SRE vs DevOpsLevon Avakyan

Version Control & GitCraig Smith

ObservabilityMartin Gross

RedHat Certification Trackssuser113f26

OpenStack Administration by Mobarak Hossain Group Organizer BangladeshMobarak Hossain

What is Git | What is GitHub | Git Tutorial | GitHub Tutorial | Devops Tutori...Edureka!

Intro to open source - 101 presentationJavier Perez

Viewers also liked (20)

Immutable infrastructure with Docker and containers (GlueCon 2015)Jérôme Petazzoni

Delivering Personalized Music DiscoveryRohan Singh

Docker {at,with} SignalFxMaxime Petazzoni

Docker and Maestro for fun, development and profitMaxime Petazzoni

Orchestration for the rest of usJérôme Petazzoni

Orchestration, resource scheduling…What does that mean? Is this only relevant for data centers with thousands of nodes? Should I care about Mesos, Kubernetes, Swarm, when all I have is a handful of virtual machines? The motto of public cloud IAAS is "pay for what you use," so in theory, if I deploy my apps there, I'm already getting the best "resource utilization" aka "bang for my buck," right? In this talk, we will answer those questions, and a few more. We will define orchestration, scheduling, and others, and show what it's like to use a scheduler to run containerized applications there.

Speechfinalslidesquincigstudent

The document discusses church oversight and political activity by churches. It notes that the IRS can only begin a church tax inquiry if a high-level Treasury official reasonably believes, based on written evidence, that the church may not qualify for tax exemption or may not be paying taxes on unrelated business activities. However, churches currently have a lack of oversight and could endorse political campaigns, have pastors use donations for personal use, or make private deals. The document encourages citizens to voice any concerns to politicians or the IRS to ensure fairness and equal treatment.

Smoszewo dalsza historiapzgomaz

Regulamin konkursupzgomaz

Speechfinalslides2quincigstudent

This document discusses oversight of churches and restrictions on their political activities. It notes that without oversight, churches could endorse political candidates or use donations for purposes other than intended. It explains that the IRS can only begin an inquiry if high-level officials reasonably believe based on written evidence that a church may not qualify for tax exemption or may not be paying applicable taxes. It encourages readers to voice any concerns to politicians and the IRS. The document aims to help readers trust their institutions and ensure churches follow rules rather than push political agendas.

Speechslides for classquincigstudent

SSL (Secure Socket Layer) is a tool that provides security on the Internet by allowing secure interactions with websites and keeping data safe. It works by encrypting data during transmission between a server and client. There are security risks to consider, such as unverified connections, and it is important because many sensitive online activities like banking and email require secure connections to protect users' personal information and financial data.

SXSW Interactive 2014CowanDeBaets

This document provides information about a panel discussion on the changing entertainment industry landscape due to new distribution platforms like Netflix. The panel will discuss how Netflix's original series "House of Cards" was both a TV show and movie by being released all at once online, how this success has attracted other companies to create new online content, and how these new productions can get attention and make money given changing consumption habits. The moderator, Steven Masur, will lead a discussion with experts from Netflix, Participant Media, Specific Media, and others on these topics and how audiences can get involved in the evolving entertainment industry.

LN Case StudyBlain Banick

Haynes and Boone wanted to improve its business development efforts by building a sophisticated technology platform. It implemented various LexisNexis solutions including Client Analysis to gain insights into clients. This has helped Haynes and Boone improve client analyses, direct business development more effectively, and provide timely intelligence to lawyers and leaders. The new system delivers accurate client data in real-time that has increased productivity and informed strategic decisions.

speech class ssl documentquincigstudent

SSL (Secure Socket Layer) is a tool that provides security on the Internet and allows users to interact securely with websites like banks by encrypting data and passwords. It works by having servers set up to use SSL certificates which establish a secure connection and allow sensitive data to be shared privately. SSL is important because it keeps users' data private and secure when conducting activities online that require transmitting private information.

4.τα εικαστικα μασvkarab2

X Światowe Igrzyska Sportowe Głuchychpzgomaz

Honorowy Prezes PZGpzgomaz

Creative IT MindsRAJESH KRAJ, Digital Marketing and Data Analytics Expert

Digital marketing refers to a set of practices for promoting your brand and services over the web through websites. Pushing aside the conventional norms of shouting about an enterprise in every possible manner, marketing has become smarter, cheaper and more result oriented. The same takes into account a lot of permutations in fetching the most appropriate keywords, based on which the entire advertising campaign is structured. Transforming start-ups into brands following a strategic approach at syncing with the search engine behaviors, we at IT Minds Lab are a dedicated digital marketing company in India with a bag full of insightful ideas. While your competitor has incarnated a digital presence to reach out to a wider customer base, it becomes all the more imperative for you to approach the best digital marketing agency.

Świat Głuchoniemych Gazeta z 1927r.pzgomaz

Kongrespzgomaz

Powstanie Warszawskie oczami 12 letniego cywilapzgomaz

Immutable infrastructure with Docker and containers (GlueCon 2015)Jérôme Petazzoni

Delivering Personalized Music DiscoveryRohan Singh

Docker {at,with} SignalFxMaxime Petazzoni

Docker and Maestro for fun, development and profitMaxime Petazzoni

Orchestration for the rest of usJérôme Petazzoni

Speechfinalslidesquincigstudent

Smoszewo dalsza historiapzgomaz

Regulamin konkursupzgomaz

Speechfinalslides2quincigstudent

Speechslides for classquincigstudent

SXSW Interactive 2014CowanDeBaets

LN Case StudyBlain Banick

speech class ssl documentquincigstudent

4.τα εικαστικα μασvkarab2

X Światowe Igrzyska Sportowe Głuchychpzgomaz

Honorowy Prezes PZGpzgomaz

Creative IT MindsRAJESH KRAJ, Digital Marketing and Data Analytics Expert

Świat Głuchoniemych Gazeta z 1927r.pzgomaz

Kongrespzgomaz

Powstanie Warszawskie oczami 12 letniego cywilapzgomaz

Similar to Docker at Spotify (20)

Container DaysPatrick Mizer

This document summarizes a company's journey to implementing Docker containers in production. It begins with using Docker in hackathons and development environments. Early production uses involved deploying individual services as containers but lacked orchestration. They then tried microservices and deploying to a self-hosted registry, which had stability issues. They eventually selected Rancher for orchestration and Quay.io for the registry. Secret configuration is managed using SaltStack pillars. Templating allows a single docker-compose file across environments. Rancher-compose is used to deploy versions to environments and roll back if needed. This overcomes earlier challenges and provides a smooth path to containerized microservices in production.

Dockerize or dieCreate Digital

Docker is getting much love & attention these days. Despite that the version 1.0 was released just this spring, pretty much everyone is working on their own version of a system that would allow developers to automate container creation. A combination of CoreOS and Deis seem to be promising, providing a familiar Heroku-style deployment mechanism and eliminating the unnecessary complexity of manual container management. The slides give an overview of the tools mentioned and the benefits one gets by adopting them.

Docker in Production at the Aurora TeamHaufe-Lexware GmbH & Co KG

Shipping NodeJS with Docker and CoreOSRoss Kukulinski

Docker Meetup Paris: enterprise DockerArnaud MAZIN

This document discusses software management in the enterprise using Docker containers. It begins with an overview of a traditional software factory model and then examines how Docker could be integrated at various points, including the developer workstation, continuous integration servers, and production servers. Several example Docker-based platforms are described, along with considerations around configuration management and deployment orchestration. The key takeaways are that there is no single integration pattern and hybrid approaches may be needed, integration and topology tools are still maturing, and image-based deployments could initially be easier than rebuilding applications from source on each environment.

The Tale of a Docker-based Continuous Delivery Pipeline by Rafe Colton (ModCl...Docker, Inc.

The ModCloth Platform team has been building a Docker-based continuous delivery pipeline. This presentation discusses that project and how we build containers at ModCloth. The topics include what goes into our containers; how to optimize builds to use the Docker build cache effectively; useful development workflows (including using fig); and the key decision to treat containers as processes instead of mini-vms. This presentation will also discuss (and demo!) the workflow we’ve adopted for building containers and how we’ve integrated container builds with our CI.

Hermetic environment for your functional testsAndrii Dzynia

What are the most common problems with testing environments? - You are not the only one who is using it. - Test failures are not repeatable. - Test data can be easily messed up due to tests overlap. Those problems are introducing flakiness in your tests, increase frustration level and decrease confidence in quality of a product you are building. Forcing your development team to have a testing queue increases delivery time dramatically. Creating zillions of environments does not sound as cheapest solution either. At Spotify we experimented with different approaches on how testing environments can be configured: from shared environment to mocks, stubs and hermetic servers. During my presentation I will share the lessons we learned, what worked, what not and what is the direction we are pursuing in order to stabilise our testing suites.

Dockercon EU 2014Rafe Colton

My talk from Dockercon EU in Amsterdam, Dec 2014. Original abstract: The ModCloth Platform team has been building a Docker-based continuous delivery pipeline. This presentation discusses that project and how we build containers at ModCloth. The topics include what goes into our containers; how to optimize builds to use the Docker build cache effectively; useful development workflows (including using fig); and the key decision to treat containers as processes instead of mini-vms. This presentation will also discuss (and demo!) the workflow we’ve adopted for building containers and how we’ve integrated container builds with our CI.

Docker at Djangocon 2013 | Talk by Ken CochranedotCloud

Ken Cochrane gave a presentation on Docker and Docker's suitability for Django projects. He began with an introduction to Docker, explaining how it uses Linux containers to package applications into lightweight portable containers. He then discussed several common use cases for Docker like local development, continuous integration/deployment, and testing. The presentation concluded with a demo of Docker commands and a discussion of upcoming Docker 1.0 features.

Django and DockerDocker, Inc.

Spotify: Data center & Backend buildoutDavid Poblador i Garcia

Container (Docker) Orchestration ToolsDhilipsiva DS

This document discusses various Docker orchestration tools including Kubernetes, Marathon, Rancher, Helios, Ansible Container, Docker Swarm, and others. It provides brief descriptions and usage examples for several of the tools. Kubernetes and Marathon are covered in slightly more detail, with notes on prerequisites and basic usage commands provided. The document aims to introduce developers to common options for orchestrating Docker containers at scale.

Orchestrating Linux ContainersBergamo Linux Users Group

Dockerizing RailsiGbanam

This document discusses dockerizing Rails applications for repeatable deployments with zero downtime. It introduces Docker, explaining that containers provide lightweight virtualization at the OS level compared to traditional virtual machines. Rails principles of programmer happiness and convention over configuration are discussed. The document demonstrates dockerizing a sample ToDo app using Dockerfiles, images, containers, Docker Compose for orchestration, and Docker Swarm for scaling across multiple nodes. It compares monolith vs microservices architectures and Swarm vs Kubernetes orchestration tools.

The journey to container adoption in enterpriseIgor Moochnick

This document discusses the journey to container adoption in enterprises. It begins by describing traditional monolithic architectures and then discusses how container technologies like Docker and Mesos enable new paradigms like microservices that emphasize speed, agility and loose coupling. It covers challenges around deployment, testing, monitoring and failure handling with containers and discusses emerging tools and approaches to address these challenges. Finally, it considers future directions like Kubernetes and stream processing architectures.

Java Developer Intro to Environment Management with Vagrant, Puppet, and Dock...Lucas Jellema

Creating and managing environments for development and R&D activities can be cumbersome. Quickly spinning up databases and web servers, using physical resources in a smart way, installing application components, and having all the elements talk to each other can take a lot of time. This session takes you by the hand and introduces you to Vagrant and Oracle VM VirtualBox for quickly provisioning VMs in which Docker containers run platform components, applications, and microservices—all set up by use of Puppet and interacting with Git(Hub). You’ll start from zero on your laptop and end with both local and public cloud environments in which to develop, test, and run various types of applications. Lean governance and evolution of the environments are discussed too.

Docker Swarm and Traefik 2.0Jakub Hajek

The Docker ecosystem and the future of application deploymentJérôme Petazzoni

Ten years ago, virtualization ignited a revolution which gave birth to the Cloud and the DevOps initiative. Today, with containers, we are at the dawn of a similar breakthrough. How can we capture the value of containers? How can we use their features to implement microservices and immutable infrastructures, while retaining as much as possible of our existing practices? The answer is in the rich ecosystem that developed around Docker, an open-source platform to build, ship, and run applications in containers. In this keynote we’ll explore what the applications of tomorrow will look like, how they’ll be deployed and distributed – and how to leverage those tools today.

Docker in Production: How RightScale Delivers Cloud ApplicationsRightScale

This document discusses how RightScale delivers cloud applications using Docker. It begins with an introduction to Docker and outlines three approaches to using Docker: containerizing code, composing applications, and deploying a sea of containers. It then details RightScale's plan of attack for using Docker which takes a gradual, phased approach. The document also covers RightScale's development to production workflow when using Docker and compares operations before and after adopting Docker. It concludes with discussing next steps such as improving monitoring, composition, and orchestration when using Docker.

DockerCon 15 Keynote - Day 2Docker, Inc.

The document summarizes Day 2 of DockerCon. It discusses Docker being ready for production use with solutions for building, shipping, and running containers. It highlights Docker Hub growth and improvements to quality. Business Insider's journey with Docker is presented, covering lessons learned around local development and using Puppet and Docker Hub. Future directions discussed include orchestration tools and image security.

Container DaysPatrick Mizer

Dockerize or dieCreate Digital

Docker in Production at the Aurora TeamHaufe-Lexware GmbH & Co KG

Shipping NodeJS with Docker and CoreOSRoss Kukulinski

Docker Meetup Paris: enterprise DockerArnaud MAZIN

The Tale of a Docker-based Continuous Delivery Pipeline by Rafe Colton (ModCl...Docker, Inc.

Hermetic environment for your functional testsAndrii Dzynia

Dockercon EU 2014Rafe Colton

Docker at Djangocon 2013 | Talk by Ken CochranedotCloud

Django and DockerDocker, Inc.

Spotify: Data center & Backend buildoutDavid Poblador i Garcia

Container (Docker) Orchestration ToolsDhilipsiva DS

Orchestrating Linux ContainersBergamo Linux Users Group

Dockerizing RailsiGbanam

The journey to container adoption in enterpriseIgor Moochnick

Java Developer Intro to Environment Management with Vagrant, Puppet, and Dock...Lucas Jellema

Docker Swarm and Traefik 2.0Jakub Hajek

The Docker ecosystem and the future of application deploymentJérôme Petazzoni

Docker in Production: How RightScale Delivers Cloud ApplicationsRightScale

DockerCon 15 Keynote - Day 2Docker, Inc.

Recently uploaded (20)

Introduction to QM, QA, QC, Bug's priority and severityArshad QA

Contractor Hot Work Permit Software for Work SafetySHEQ Network Limited

Portland Marketo User Group: MOPs & AI - Jeff Canada - May 2025BradBedford3

Jeff Canada is the first MOPs hire at OpenAI, the creator of ChatGPT. He is a team of 1 in a super fast growing company, which is familiar story for many of us. His presentation, originally presented at Mopsapolooza 2024, he gives you an outline of First Steps to Smarter MOPs with the warning label: This is all brand new to everyone; don’t have to jump in head first! Jeff's story is how he was able to accomplish more via his “AI employees”. Jeff will talk about how he has used OpenAI to help him staff his team with: AI Researcher AI Analyst AI Content Generator AI Developer These additional teammates assist with Vendor and Event Selection, Content Generation, Coding Cleanup, and Thinking! His wrap up includes, Guardrails, words of caution, and steps to get you started.

Skilling up your dev team - 8 things to consider when skilling-up your dev teamDerk-Jan de Grood

Slides of my DevOps Pro Europe 2025 presentation Vilnius: Most IT organizations face challenges of being underskilled or understaffed, making it difficult to find skilled developers and manage workload efficiently. This leads to risks, dependencies, and delays, particularly when critical tasks depend on a few key developers. Investing in employee development is crucial for improving performance and attracting talent, but it requires strategic planning and collaboration. Companies are aware of this, so why do they keep failing? Successful upskilling involves team autonomy, leadership buy-in, and dedicated focus. This presentation outlines eight key considerations for effective upskilling: defining clear roles, identifying needed skills, conducting gap analyses, planning for future needs, exploring diverse training methods, securing leadership support, actively monitoring progress, and embedding upskilling into HR processes. By addressing these aspects, organizations can foster technical excellence and continuous improvement.

Validationapproach for sap data -.pptxaichannellsh

Progecad 2025 Professional Cracked [Latest]Luisa Weist

progeCAD 2025 Professional Crack is a compatible 2D / 3D CAD application that works with AutoCAD DWG files and imports Autodesk Revit, IFC, and SolidWorks files! This Site is providing ✅ 100% Safe Crack Link: Copy This Link and paste it in a new tab & get the Crack File ↓ ➡ 🌍📱👉COPY & PASTE LINK👉👉👉 👉 https://crackproduct.com/progecad-professional/

CFCamp2025 - Keynote Day 1 led by Luis Majano.pdfOrtus Solutions, Corp

We’re honored to share the official keynote presentation that opened CFCamp 2025, led by Luis Majano, creator of ColdBox, BoxLang, and CEO of Ortus Solutions. This PDF features the full slide deck from Day 1’s keynote, where Luis presented a powerful vision for the future of modern CFML development, highlighted the evolution of BoxLang, and shared how Ortus is helping shape a dynamic future for developers around the world. A heartfelt thank you to the CFCamp team for the opportunity to lead the keynote and showcase the innovation, community, and open source spirit driving the next chapter of CFML. 🚀

Best Practices Salesforce Training & Documentation.pptxMichael Orias

LightBurn 1.7.0.6 + Crack Download🔓Berkeley

Introduction to Programming presentation.pptxHorusCarlosVilln

Shortcomings of EHS Software – And How to Overcome ThemTECH EHS Solution

Shortcomings of EHS Software—and What Overcomes Them What you'll learn in just 8 slides: - 🔍 Why most EHS software implementations struggle initially - 🚧 3 common pitfalls: adoption, workflow disruption, and delayed ROI - 🛠️ Practical solutions that deliver long-term value - 🔐 Key features: centralization, security, affordability - 📈 Why the pros outweigh the cons Perfect for HSE heads, plant managers, and compliance leads! #EHS #TECHEHS #WorkplaceSafety #EHSCompliance #EHSManagement #ehssoftware #safetysoftware

Salesforce Experience Cloud Consulting.pdfVALiNTRY360

VALiNTRY360’s Salesforce Experience Cloud Consulting services empower organizations to build personalized, engaging digital experiences for customers, partners, and employees. Our certified Salesforce experts help you design, implement, and optimize Experience Cloud portals tailored to your business goals. From self-service communities to partner collaboration hubs, we ensure seamless integration, enhanced user engagement, and scalable solutions. Whether you're improving customer support or streamlining partner communication, VALiNTRY360 delivers strategic consulting to maximize the value of Salesforce Experience Cloud. Trust us to transform your digital experiences into powerful tools that drive loyalty, efficiency, and growth. Partner with VALiNTRY360 to elevate every user interaction. For more info visit us https://valintry360.com/salesforce-experience-cloud

Optimising Claims Management with Claims Processing SystemsInsurance Tech Services

(Ethical) Alternatives to Piracy: A Quick Guide to Free and Open Source Softw...s-m-quadri

This presentation, titled "Ethical Alternatives to Piracy", originates from a scholarly effort at Deogiri Institute of Engineering and Management Studies. It addresses the urgent ethical, legal, and cybersecurity challenges posed by the widespread use of pirated software. Piracy is defined as the unauthorized use of paid digital content, and it is a practice fraught with risks—ranging from legal consequences and data breaches to malware infections, software crashes, and absence of support or updates. Alarming statistics indicate that 91% of users in India engage in piracy, a number that has surged due to increased digital dependence in the post-pandemic era. The presentation transitions from critique to constructive solution by introducing reliable, free, and ethical alternatives via Free and Open Source Software (FOSS) and free operating systems. Emphasizing India's moral and cultural ethos, the speaker argues for a shift from piracy to integrity-driven digital usage, especially among engineering professionals. Two robust Linux-based alternatives to Windows are discussed: Ubuntu Studio, aimed at creatives with its XFCE desktop and Long Term Support (LTS) stability; and Pop!_OS, designed for STEM users, offering a GNOME desktop that balances performance with usability. A broad selection of FOSS tools is explored as viable substitutes for expensive proprietary software. These include Blender (3D animation; alternative to Maya), Natron (compositing; alternative to After Effects), DaVinci Resolve (professional-grade editing), Kdenlive (video editing), VLC (media playback and conversion), OBS Studio (screen recording and streaming), and Audacity or Ardour (audio editing and mixing). Visual design needs are met by Inkscape and GIMP, serving as counterparts to Adobe Illustrator and Photoshop, respectively. Productivity is addressed through LibreOffice as an alternative to Microsoft Office, and tools like VS Code, Android Studio, VirtualBox, Synaptic, and Stacer round out the system management and development environment options. The central argument is not to vilify proprietary software, but to discourage unethical use through piracy. FOSS alternatives offer a legitimate, secure, and often equally powerful path for students, professionals, and institutions. The presentation concludes by encouraging a transition to open-source ecosystems—not just as a technical decision, but as a step toward ethical, sustainable, and secure digital practice. It is a call to adopt responsible engineering values and become part of the global open-source movement.

Multiple Platforms of Unity Game Development.pdfNova Carter

Unity Game Development stands out for its unparalleled flexibility across multiple platforms, making it a top choice for developers aiming to reach a broad audience. With Unity, creators can build a game once and deploy it seamlessly across mobile devices, desktops, gaming consoles, web browsers, and even AR/VR systems. This multi-platform capability reduces development costs and effort while ensuring consistent performance and user experience across devices. Whether targeting casual mobile gamers or console enthusiasts, Unity empowers developers to scale their games effectively and maintain a competitive edge in today’s diverse gaming landscape.

Linux Improvements in Memory Corruption Based ProtectionsVlatko Kosturjak

Advance Sales Commission Payments in OdooAxisTechnolabs

♦️Sales Commission in Odoo is really easy to use and manage commissions payment, Sales Commission Dashboard for Quick counts of Total Sales Commission, Total Invoice Commission, Total payment Commission. Save your time for individual sales commission calculation and create the invoice for any commission and many more useful features. 🔗Visit and Buy Now : https://bit.ly/4fBJ5Ok ♦️Axistehnolabs Present Manage your sales commission Payment in odoo with Sales commission dashboard, calculation of commission based on invoice, product, product categories and product margin ✅Sales Commission Odoo module includes : 👉Sales Commission Based On Sale Order 👉Sales Commission Based On Invoice 👉Sales Commission Based On Payment 👉Calculate Commission Based on Products 👉And More..... ✅Just Click on below sales commission payment odoo App link🔗 And explore more interesting features of sales commission payment odoo module ✅App download now :🔗 Odoo 18 : https://bit.ly/4fBJ5Ok Odoo 17 : https://bit.ly/3KCDt9i Odoo 16 : https://bit.ly/3w0iKEZ Odoo 15 : https://bit.ly/3vAszKL Odoo 14 : https://bit.ly/35QTVBI Odoo 13 : https://bit.ly/3J8exDY Odoo 12 : https://bit.ly/3shwCIO 👉Explore more odoo Apps : https://bit.ly/3oFIOCF 👉Want A Free DEMO ? :📩 [email protected] 👉Want to discuss ?: 🌐https://www.axistechnolabs.com/ 👉Looking Odoo services : 👇 https://www.axistechnolabs.com/services/odoo-development 👉Contact Us : 091066 49361 #odoo #Odoo18 #odoo17 #odoo16 #odoo15 #odooerp #odooapp #Odooservices #odoodev #salescommission #odootag #odoonewfeatures #PaymentCommission #salescommissionrates #odooapps #odoomodule #odooerp #growth #businesssolution #salescommissionpayment #saleordercommission #InvoiceCommission #MonthlyCommission #odoonewfeatures #commissionspaymentModule #sales #business #commissionpayment #GenerateCommission #levelupwithodoo #socialmedia #digitalmarketing #OdooSalesCommissionmodule #odooimplementation #axistechnolabs #OdooSolutions

Daily Agile Snippets That Boost Team Focus and FlexibilityOrangescrum

Professional Consulting Resume of AL Davisald303873

Key Characteristics of High-Performing Insurance Broker SoftwareInsurance Tech Services

Introduction to QM, QA, QC, Bug's priority and severityArshad QA

Contractor Hot Work Permit Software for Work SafetySHEQ Network Limited

Portland Marketo User Group: MOPs & AI - Jeff Canada - May 2025BradBedford3

Skilling up your dev team - 8 things to consider when skilling-up your dev teamDerk-Jan de Grood

Validationapproach for sap data -.pptxaichannellsh

Progecad 2025 Professional Cracked [Latest]Luisa Weist

CFCamp2025 - Keynote Day 1 led by Luis Majano.pdfOrtus Solutions, Corp

Best Practices Salesforce Training & Documentation.pptxMichael Orias

LightBurn 1.7.0.6 + Crack Download🔓Berkeley

Introduction to Programming presentation.pptxHorusCarlosVilln

Shortcomings of EHS Software – And How to Overcome ThemTECH EHS Solution

Salesforce Experience Cloud Consulting.pdfVALiNTRY360

Optimising Claims Management with Claims Processing SystemsInsurance Tech Services

(Ethical) Alternatives to Piracy: A Quick Guide to Free and Open Source Softw...s-m-quadri

Multiple Platforms of Unity Game Development.pdfNova Carter

Linux Improvements in Memory Corruption Based ProtectionsVlatko Kosturjak

Advance Sales Commission Payments in OdooAxisTechnolabs

Daily Agile Snippets That Boost Team Focus and FlexibilityOrangescrum

Professional Consulting Resume of AL Davisald303873

Key Characteristics of High-Performing Insurance Broker SoftwareInsurance Tech Services

Docker at Spotify

1. Docker at Spotify ! Rohan Singh @rohansingh

2. Spotify “Music for every moment” ! 40+ million active users ! 20+ million songs available (we’re in 55+ countries, differs by market) ! Live in the US for 2 years 2

3. What are we working with? ! 100+ distinct backend services ! 5000+ production servers in 4 sites worldwide ! Almost 300 servers per ops engineer 3

4. Deployment at Spotify 4

5. Build Debian packages ! Requisition new hardware ! Use combination of Puppet and internal tooling (deployify) 5

6. old way: SSH to a bunch of machines, apt-get install || upgrade ! deployify: Uses fabric, which just SSH’s, runs apt-get install || upgrade 6

7. What’s wrong with that? 7

8. Requisitioning new hardware is slow ! Hardware utilization is low ! We’ve automated and improved, but still too slow 8

9. Deployments often fail… ! …though usually only partially 9

10. ! Machines get denormalized easily 10

11. Debian packaging is its own special hell So many damn files! ! debhelper ! Distro & version specific ! apt-get update 11

12. What do we want? 12

13. Deployments that are: ! Repeatable ! Straightforward ! Fault-tolerant 13

14. Why Docker? ! It gives us “repeatable”, right out of the box ! ! And it’s fairly straightforward, too! 14

15. Repeatability? ! ! The code you test isn’t necessarily the code that runs in prod 15

16. Repeatability — Docker ! Build an image ! Run tests against the image ! Run the tested image in prod 16

17. Straightforward ! Just one Dockerfile ! Pretty simple syntax ! Immutable images — easy to reason about 17

18. Fault-tolerance? Even if your code is perfect, deployments might fail ! Puppet or Debian repos are down ! Network issues ! The machine dies 18

19. Fault-tolerance — Docker Just one thing has to work — pulling & running an image ! If it fails, just try again ! No machines left in weird states* ! ! *(in theory) 19

20. What more do we need? 20

21. Deploying Docker containers across our entire fleet of servers ! Some way to deal with Docker or container failures ! That’s pretty much it 21

22. Looking around for a Docker platform Several really feature-full offerings ! Pretty opinionated and all-encompassing ! All we need is reliably deploying containers to a fleet of Docker hosts 22

23. Helios 23

24. Makes sure your containers are deployed & running exactly where you want them… ! …and that’s pretty much it ! ! github.com/spotify/helios 24

25. What’s in the box? Helios master frontend that you talk to ! Helios agent runs alongside each Docker instance ! Zookeeper ! Straightforward CLI 25

26. How does it work? Create a job — tell Helios which Docker image to deploy ! Deploy the job — tell Helios where to run the Docker containers 26

27. See it in action 27

28. What’s next? 28

29. Helios is just the beginning ! Brought up plenty of Helios+Docker agents in our private cloud for our devs ! Integrated Helios with JUnit — write JUnit tests run against Docker containers 29

30. Dockerized integration tests 30

31. This week — first Helios+Docker-based service in production ! ! In the future — Containerizing more of our infrastructure 31

32. Give it a go Open source: github.com/spotify/helios ! Try it out: git clone https://github.com/spotify/helios.git && cd helios && vagrant up ! ! ! Come play with us (spotify.com/jobs) 32

Docker at Spotify

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Docker at Spotify (20)

Recently uploaded (20)

Docker at Spotify