ℹ️  This repo contains questions and exercises on various technical topics, sometimes related to DevOps and SRE

📊  There are currently 2619 exercises and questions

⚠️  You can use these for preparing for an interview but most of the questions and exercises don't represent an actual interview. Please read FAQ page for more details

🛑  If you are interested in pursuing a career as DevOps engineer, learning some of the concepts mentioned here would be useful, but you should know it's not about learning all the topics and technologies mentioned in this repository

📝  You can add more exercises by submitting pull requests :) Read about contribution guidelines here

---

DevOps	CI/CD	Git	Ansible	Network
Software Development	Python	Go	Shell Scripting	Kubernetes
Cloud	AWS	Azure	Google Cloud Platform	OpenStack
Operating System	Monitoring	Elastic	Virtualization	DNS
Testing	Databases	Regex	System Design	Hardware
Certificates	Containers	SQL	OpenShift	Storage
Terraform	Mongo	Puppet	Distributed	Questions you can ask
Circle CI	Argo	Kafka		Perl
Linux	Prometheus	Big Data	Soft Skills	Security
Chaos Engineering	Misc

Network

In general, what do you need in order to communicate?

• A common language (for the two ends to understand)
• A way to address who do you want to communicate with
• A Connection (so the content of the communication can reach the recipients)

What is TCP/IP?

A set of protocols that define how two or more devices can communicate with each other.

To learn more about TCP/IP, read here

What is Ethernet?

Ethernet simply refers to the most common type of Local Area Network (LAN) used today. A LAN—in contrast to a WAN (Wide Area Network), which spans a larger geographical area—is a connected network of computers in a small area, like your office, college campus, or even home.

What is a MAC address? What is it used for?

A MAC address is a unique identification number or code used to identify individual devices on the network.

Packets that are sent on the ethernet are always coming from a MAC address and sent to a MAC address. If a network adapter is receiving a packet, it is comparing the packet’s destination MAC address to the adapter’s own MAC address.

When is this MAC address used?: ff:ff:ff:ff:ff:ff

When a device sends a packet to the broadcast MAC address (FF:FF:FF:FF:FF:FF​), it is delivered to all stations on the local network. Ethernet broadcasts are used to resolve IP addresses to MAC addresses (by ARP) at the datalink layer .

What is an IP address?

An Internet Protocol address (IP address) is a numerical label assigned to each device connected to a computer network that uses the Internet Protocol for communication.An IP address serves two main functions: host or network interface identification and location addressing.

Explain subnet mask and given an example

A Subnet mask is a 32-bit number that masks an IP address, and divides the IP address into network address and host address. Subnet Mask is made by setting network bits to all "1"s and setting host bits to all "0"s. Within a given network, out of the total usable host addresses, two are always reserved for specific purposes and cannot be allocated to any host. These are the first address, which is reserved as a network address (a.k.a network ID) and the last address used for network broadcast.

Example

What is a private IP address? In which scenarios/system designs, one should use it?

Private IP addresses are assigned to the hosts in the same network to communicate among one another. As the name "private" suggests, the devices having the private IP addresses assigned can't be reached by the devices from any external network. For example, if I am living in a hostel and I want my hostelmates to join the game server I have hosted, I will ask them to join via my server's private IP address, since the network is local to the hostel.

What is a public IP address? In which scenarios/system designs, one should use it?

A public IP address is the public facing IP address. In the event that you was hosting a game server that you want your friends to join, you will give your friends your public IP address to allow their computers to identify and locate your network and server in order for the connection to take place. One time that you would not need to use a public facing IP address is in the event that you was playing with friends who was connected to the same network as you, in that case, you would use a private ip address. In order for someone to be able to connect to your server that is located internally, you will have to setup a port forward to tell your router to allow traffic from the public domain into your network and vice versa.

Explain the OSI model. What layers there are? What each layer is responsible for?

• Application: user end (HTTP is here)
• Presentation: establishes context between application-layer entities (Encryption is here)
• Session: establishes, manages and terminates the connections
• Transport: transfers variable-length data sequences from a source to a destination host (TCP & UDP are here)
• Network: transfers datagrams from one network to another (IP is here)
• Data link: provides a link between two directly connected nodes (MAC is here)
• Physical: the electrical and physical spec the data connection (Bits are here)

You can read more about the OSI model in penguintutor.com

For each of the following determines to which OSI layer it belongs:

• Error correction
• Packets routing
• Cables and electrical signals
• MAC address
• IP address
• Terminate connections
• 3 way handshake

Error correction - Data link

Packets routing - Network

Cables and electrical signals - Physical

MAC address - Data link

IP address - Network

Terminate connections - Session

3 way handshake - Transport

What delivery schemes are you familiar with?

Unitcast: One to one communication where there is one sender and one receiver.

Broadcast: Sending a message to everyone in the network. The address ff:ff:ff:ff:ff:ff is used for broadcasting. Two common protocols which use broadcast are ARP and DHCP.

Multicast: Sending a message to a group of subscribers. It can be one-to-many or many-to-many.

What is CSMA/CD? Is it used in modern ethernet networks?

CSMA/CD stands for Carrier Sense Multiple Access / Collision Detection. Its primarily focus it to manage access to shared medium/bus where only one host can transmit at a given point of time.

CSMA/CD algorithm:

1. Before sending a frame, it checks whether another host already transmitting a frame.
2. If no one transmitting, it starts transmitting the frame.
3. If two hosts transmitted at the same time, we have a collision.
4. Both hosts stop sending the frame and they send to everyone a 'jam signal' notifying everyone that a collision occurred
5. They are waiting for a random time before sending again
6. Once each host waited for a random time, they try to send the frame again and so the

Describe the following network devices and the difference between them:

• router
• switch
• hub

What is a "Collision Domain"?

A collision domain is a network segment in which devices can potentially interfere with each other by attempting to transmit data at the same time. When two devices transmit data at the same time, it can cause a collision, resulting in lost or corrupted data. In a collision domain, all devices share the same bandwidth, and any device can potentially interfere with the transmission of data by other devices.

What is a "Broadcast Domain"?

A broadcast domain is a network segment in which all devices can communicate with each other by sending broadcast messages. A broadcast message is a message that is sent to all devices in a network rather than a specific device. In a broadcast domain, all devices can receive and process broadcast messages, regardless of whether the message was intended for them or not.

three computers connected to a switch. How many collision domains are there? How many broadcast domains?

Three collision domains and one broadcast domain

How does a router works?

A router is a physical or virtual appliance that passes information between two or more packet-switched computer networks. A router inspects a given data packet's destination Internet Protocol address (IP address), calculates the best way for it to reach its destination and then forwards it accordingly.

What is NAT?

Network Address Translation (NAT) is a process in which one or more local IP address is translated into one or more Global IP address and vice versa in order to provide Internet access to the local hosts.

What is a proxy? How does it works? What do we need it for?

A proxy server acts as a gateway between you and the internet. It’s an intermediary server separating end users from the websites they browse.

If you’re using a proxy server, internet traffic flows through the proxy server on its way to the address you requested. The request then comes back through that same proxy server (there are exceptions to this rule), and then the proxy server forwards the data received from the website to you.

Proxy servers provide varying levels of functionality, security, and privacy depending on your use case, needs, or company policy.

What is TCP? How does it works? What is the 3 way handshake?

TCP 3-way handshake or three-way handshake is a process which is used in a TCP/IP network to make a connection between server and client.

A three-way handshake is primarily used to create a TCP socket connection. It works when:

• A client node sends a SYN data packet over an IP network to a server on the same or an external network. The objective of this packet is to ask/infer if the server is open for new connections.
• The target server must have open ports that can accept and initiate new connections. When the server receives the SYN packet from the client node, it responds and returns a confirmation receipt – the ACK packet or SYN/ACK packet.
• The client node receives the SYN/ACK from the server and responds with an ACK packet.

What is round-trip delay or round-trip time?

From wikipedia: "the length of time it takes for a signal to be sent plus the length of time it takes for an acknowledgement of that signal to be received"

Bonus question: what is the RTT of LAN?

How does SSL handshake work?

SSL handshake is a process that establishes a secure connection between a client and a server.

1. The client sends a Client Hello message to the server, which includes the client's version of the SSL/TLS protocol, a list of the cryptographic algorithms supported by the client, and a random value.
2. The server responds with a Server Hello message, which includes the server's version of the SSL/TLS protocol, a random value, and a session ID.
3. The server sends a Certificate message, which contains the server's certificate.
4. The server sends a Server Hello Done message, which indicates that the server is done sending messages for the Server Hello phase.
5. The client sends a Client Key Exchange message, which contains the client's public key.
6. The client sends a Change Cipher Spec message, which notifies the server that the client is about to send a message encrypted with the new cipher spec.
7. The client sends an Encrypted Handshake Message, which contains the pre-master secret encrypted with the server's public key.
8. The server sends a Change Cipher Spec message, which notifies the client that the server is about to send a message encrypted with the new cipher spec.
9. The server sends an Encrypted Handshake Message, which contains the pre-master secret encrypted with the client's public key.
10. The client and server can now exchange application data.

What is the difference between TCP and UDP?

TCP establishes a connection between the client and the server to guarantee the order of the packages, on the other hand, UDP does not establish a connection between client and server and doesn't handle package order. This makes UDP more lightweight than TCP and a perfect candidate for services like streaming.

Penguintutor.com provides a good explanation.

What TCP/IP protocols are you familiar with?

Explain "default gateway"

A default gateway serves as an access point or IP router that a networked computer uses to send information to a computer in another network or the internet.

What is ARP? How does it works?

ARP stands for Address Resolution Protocol. When you try to ping an IP address on your local network, say 192.168.1.1, your system has to turn the IP address 192.168.1.1 into a MAC address. This involves using ARP to resolve the address, hence its name.

Systems keep an ARP look-up table where they store information about what IP addresses are associated with what MAC addresses. When trying to send a packet to an IP address, the system will first consult this table to see if it already knows the MAC address. If there is a value cached, ARP is not used.

What is TTL? What does it helps to prevent?

What is DHCP? How does it works?

It stands for Dynamic Host Configuration Protocol, and allocates IP addresses, subnet masks and gateways to hosts. This is how it works:

• A host upon entering a network, broadcasts a message in search of a DHCP server (DHCP DISCOVER)
• An offer message is sent back by the DHCP server as a packet containing lease time, subnet mask, IP addresses, etc (DHCP OFFER)
• Depending on which offer accepted, the client sends back a reply broadcast letting all DHCP servers know (DHCP REQUEST)
• Server sends an acknowledgment (DHCP ACK)

Read more here

Can you have two DHCP servers in the same network? How it works?

What is SSL tunneling? How does it works?

What is a socket? Where can you see the list of sockets in your system?

What is IPv6? Why should we consider using it if we have IPv4?

What is VLAN?

What is MTU?

MTU stands for Maximum Transmission Unit. It's the size of the largest PDU (protocol Data Unit) that can be sent in a single transaction.

What happens if you send a packet that is bigger than the MTU?

With IPv4 protocol, router can fragment the PDU then sending all the fragmented PDU through the transaction.

With IPv6 protocol, it issues a error to the user's computer.

True or False?. Ping is using UDP because it doesn't care about reliable connection

What is SDN?

What is ICMP? What is it used for?

What is NAT? How does it work?

NAT stands for network address translation. It’s a way to map multiple local private addresses to a public one before transferring the information. Organizations that want multiple devices to employ a single IP address use NAT, as do most home routers. For example, your computer's private IP could be 192.168.1.100, but your router maps the traffic to it's public IP (e.g. 1.1.1.1). Any device on the internet would see the traffic coming from your public IP (1.1.1.1) instead of your private IP (192.168.1.100).

Which factors affect network performances

Which port number is used in each of the following protocols?:

• SSH
• SMTP
• HTTP
• DNS
• HTTPS
• FTP
• SFTP

• SSH - 22
• SMTP - 25
• HTTP - 80
• DNS - 53
• HTTPS - 443
• FTP - 21
• SFTP - 22

Which factors affect network performances

What is APIPA?

APIPA is a set of it addresses that devices are allocated when the main DHCP server is not reachable

What ip range does APIPA use?

APIPA uses the ip range: 169.254.0.1 - 169.254.255.254.

Control Plane and Data Plane

What "control plane" refers to?

The control plane is the part of the network that decides how to route and forward packets to a different location.

What "data plane" refers to?

The data plane is the part of the network that actually forwards the data/packets.

What "management plane" refers to?

Refers to monitoring and management functions.

To which plane (data, control, ...) is creating routing tables belongs to?

Control Plane.

Explain Spanning Tree Protocol (STP)

What is link aggregation? Why is it used?

What is Asymmetric Routing? How do deal with it?

What overlay (tunnel) protocols are you familiar with?

What is GRE? How does it works?

What is VXLAN? How does it works?

What is SNAT?

Explain OSPF

What is latency?

Latency is the time taken for an information to reach its destination from the source.

What is bandwidth?

Bandwidth is the capacity of a communication channel to measure how much data the latter can handle over a specific time period. More bandwidth would imply more traffic handling and thus more data transfer.

What is throughput?

Throughput refers to the measurement of the real amount of data transferred over a certain period of time across any transmission channel.

When performing a search query, what is more important, latency or throughput? And how to assure that what managing global infrastructure?

Latency. To have a good latency, a search query should be forwarded to the closest datacenter.

When uploading a video, what is more important, latency or throughput? And how to assure that?

Throughput. To have a good throughput, the upload stream should be routed to an underutilized link.

What other considerations (except latency and throughput) are there when forwarding requests?

• Keep caches updated (which means the request could be forwarded not to the closest datacenter)

Explain Spine & Leaf

What is Network Congestion? What can cause it?

What can you tell me about UDP packet format? What about TCP packet format? How is it different?

What is the exponential backoff algorithm? Where is it used?

Using Hamming code, what would be the code word for the following data word 100111010001101?

00110011110100011101

Give examples of protocols found in the application layer

• Hypertext Transfer Protocol (HTTP) - used for the webpages on the internet
• Simple Mail Transfer Protocol (SMTP) - email transmission
• Telecommunications Network - (TELNET) - terminal emulation to allow client access to telnet server
• File Transfer Protocol (FTP) - facilitates transfer of files between any two machines
• Domain Name System (DNS) - domain name translation
• Dynamic Host Configuration Protocol (DHCP) - allocates IP addresses, subnet masks and gateways to hosts
• Simple Network Management Protocol (SNMP) - gathers data of devices on the network

Give examples of protocols found in the network Layer

• Internet Protocol (IP) - assists in routing packets from one machine to another
• Internet Control Message Protocol (ICMP) - lets one know what is going such as error messages and debugging information

What is HSTS?

HTTP Strict Transport Security is a web server directive that informs user agents and web browsers how to handle its connection through a response header sent at the very beginning and back to the browser. This forces connections over HTTPS encryption, disregarding any script's call to load any resource in that domain over HTTP.

Read more [here](https://www.globalsign.com/en/blog/what-is-hsts-and-how-do-i-use-it#:~:text=HTTP%20Strict%20Transport%20Security%20(HSTS,and%20back%20to%20the%20browser.)

Network - Misc

What is the Internet? Is it the same as the World Wide Web?

The internet refers to network of networks, transferring huge amounts of data around the globe.
The World Wide Web is an application running on millions of server, on top of the internet, accessed through what is know as the web browser

What is the ISP?

ISP (Internet Service Provider) is the local internet company provider.

Operating System

Operating System Exercises

Name	Topic	Objective & Instructions	Solution	Comments
Fork 101	Fork	Link	Link
Fork 102	Fork	Link	Link

Operating System - Self Assessment

What is an operating system?

From the book "Operating Systems: Three Easy Pieces":

"responsible for making it easy to run programs (even allowing you to seemingly run many at the same time), allowing programs to share memory, enabling programs to interact with devices, and other fun stuff like that".

Operating System - Process

Can you explain what is a process?

A process is a running program. A program is one or more instructions and the program (or process) is executed by the operating system.

If you had to design an API for processes in an operating system, what would this API look like?

It would support the following:

• Create - allow to create new processes
• Delete - allow to remove/destroy processes
• State - allow to check the state of the process, whether it's running, stopped, waiting, etc.
• Stop - allow to stop a running process

How a process is created?

• The OS is reading program's code and any additional relevant data
• Program's code is loaded into the memory or more specifically, into the address space of the process.
• Memory is allocated for program's stack (aka run-time stack). The stack also initialized by the OS with data like argv, argc and parameters to main()
• Memory is allocated for program's heap which is required for dynamically allocated data like the data structures linked lists and hash tables
• I/O initialization tasks are performed, like in Unix/Linux based systems where each process has 3 file descriptors (input, output and error)
• OS is running the program, starting from main()

True or False? The loading of the program into the memory is done eagerly (all at once)

False. It was true in the past but today's operating systems perform lazy loading which means only the relevant pieces required for the process to run are loaded first.

What are different states of a process?

• Running - it's executing instructions
• Ready - it's ready to run but for different reasons it's on hold
• Blocked - it's waiting for some operation to complete. For example I/O disk request

What are some reasons for a process to become blocked?

• I/O operations (e.g. Reading from a disk)
• Waiting for a packet from a network

What is Inter Process Communication (IPC)?

What is "time sharing"?

Even when using a system with one physical CPU, it's possible to allow multiple users to work on it and run programs. This is possible with time sharing where computing resources are shared in a way it seems to the user the system has multiple CPUs but in fact it's simply one CPU shared by applying multiprogramming and multi-tasking.

What is "space sharing"?

Somewhat the opposite of time sharing. While in time sharing a resource is used for a while by one entity and then the same resource can be used by another resource, in space sharing the space is shared by multiple entities but in a way where it's not being transferred between them.
It's used by one entity until this entity decides to get rid of it. Take for example storage. In storage, a file is yours until you decide to delete it.

What component determines which process runs at a given moment in time?

CPU scheduler

Operating System - Memory

What is "virtual memory" and what purpose it serves?

Virtual memory combines your computer's RAM with temporary space on your hard disk. When RAM runs low, virtual memory helps to move data from RAM to a space called a paging file. Moving data to paging file can free up the RAM so your computer can complete its work. In general, the more RAM your computer has, the faster the programs run. https://www.minitool.com/lib/virtual-memory.html

What is demand paging?

What is copy-on-write?

Copy-on-write (COW) is a resource management concept, with the goal to reduce unnecessary copying of information. It is a concept which is implemented for instance within the POSIX fork syscall, which creates a duplicate process of the calling process.

The idea:

1. If resources are shared between 2 or more entities (for example shared memory segments between 2 processes) the resources don't need to be copied for every entity, but rather every entity has a READ operation access permission on the shared resource. (the shared segements are marked as read-only) (Think of every entity having a pointer to the location of the shared resource which can be dereferenced to read its value)
2. If one entity would perform a WRITE operation on a shared resource a problem would arise since the resource also would be permanently changed for ALL other entities sharing it. (Think of a process modifying some variables on the stack, or allocatingy some data dynamically on the heap, these changes to the shared resource would also apply for ALL other processes, this is definetly an undesirable behaviour)
3. As a solution only if a WRITE operation is about to be performed on a shared resource, this resource gets COPIED first and then the changes are applied.

What is a kernel, and what does it do?

The kernel is part of the operating system and is responsible for tasks like:

• Allocating memory
• Schedule processes
• Control CPU

True or False? Some pieces of the code in the kernel are loaded into protected areas of the memory so applications can't overwritten them

True

What is POSIX?

Explain what is Semaphore and what its role in operating systems

What is cache? What is buffer?

Buffer: Reserved place in RAM which is used to hold data for temporary purposes Cache: Cache is usually used when processes reading and writing to the disk to make the process faster by making similar data used by different programs easily accessible.

Virtualization

What is Virtualization?

Virtualization uses software to create an abstraction layer over computer hardware that allows the hardware elements of a single computer—processors, memory, storage and more - to be divided into multiple virtual computers, commonly called virtual machines (VMs).

What is a hypervisor?

Red Hat: "A hypervisor is software that creates and runs virtual machines (VMs). A hypervisor, sometimes called a virtual machine monitor (VMM), isolates the hypervisor operating system and resources from the virtual machines and enables the creation and management of those VMs."

Read more here

What types of hypervisors are there?

Hosted hypervisors and bare-metal hypervisors.

What are the advantages and disadvantges of bare-metal hypervisor over a hosted hypervisor?

Due to having its own drivers and a direct access to hardware components, a baremetal hypervisor will often have better performances along with stability and scalability.

On the other hand, there will probably be some limitation regarding loading (any) drivers so a hosted hypervisor will usually benefit from having a better hardware compatibility.

What types of virtualization are there?

Operating system virtualization Network functions virtualization Desktop virtualization

Is containerization is a type of Virtualization?

Yes, it's a operating-system-level virtualization, where the kernel is shared and allows to use multiple isolated user-spaces instances.

How the introduction of virtual machines changed the industry and the way applications were deployed?

The introduction of virtual machines allowed companies to deploy multiple business applications on the same hardware while each application is separated from each other in secured way, where each is running on its own separate operating system.

Virtual Machines

Do we need virtual machines in the age of containers? Are they still relevant?

Monitoring

Explain monitoring. What is it? What its goal?

Google: "Monitoring is one of the primary means by which service owners keep track of a system’s health and availability".

What is wrong with the old approach of watching for a specific value and trigger an email/phone alert while value is exceeded?

This approach require from a human to always check why the value exceeded and how to handle it while today, it is more effective to notify people only when they need to take an actual action. If the issue doesn't require any human intervention, then the problem can be fixed by some processes running in the relevant environment.

What types of monitoring outputs are you familiar with and/or used in the past?

Alerts
Tickets
Logging

What is the difference between infrastructure monitoring and application monitoring? (methods, tools, ...)

Application Performance Management

What is Application Performance Management?

• IT metrics translated into business insights
• Practices for monitoring applications insights so we can improve performances, reduce issues and improve overall user experience

Name three aspects of a project you can monitor with APM (e.g. backend)

• Frontend
• Backend
• Infra
• ...

What can be collected/monitored to perform APM monitoring?

• Metrics
• Logs
• Events Traces

Prometheus

What is Prometheus? What are some of Prometheus's main features?

In what scenarios it might be better to NOT use Prometheus?

From Prometheus documentation: "if you need 100% accuracy, such as for per-request billing".

Describe Prometheus architecture and components

Can you compare Prometheus to other solutions like InfluxDB for example?

What is an Alert?

Describe the following Prometheus components:

• Prometheus server
• Push Gateway
• Alert Manager

Prometheus server is responsible for scraping and storing the data
Push gateway is used for short-lived jobs
Alert manager is responsible for alerts ;)

What is an Instance? What is a Job?

What core metrics types Prometheus supports?

What is an exporter? What is it used for?

Which Prometheus best practices are you familiar with?. Name at least three

How to get total requests in a given period of time?

What HA in Prometheus means?

How do you join two metrics?

How to write a query that returns the value of a label?

How do you convert cpu_user_seconds to cpu usage in percentage?

Go

What are some characteristics of the Go programming language?

• Strong and static typing - the type of the variables can't be changed over time and they have to be defined at compile time
• Simplicity
• Fast compile times
• Built-in concurrency
• Garbage collected
• Platform independent
• Compile to standalone binary - anything you need to run your app will be compiled into one binary. Very useful for version management in run-time.

Go also has good community.

What is the difference between var x int = 2 and x := 2?

The result is the same, a variable with the value 2.

With var x int = 2 we are setting the variable type to integer while with x := 2 we are letting Go figure out by itself the type.

True or False? In Go we can redeclare variables and once declared we must use it.

False. We can't redeclare variables but yes, we must used declared variables.

What libraries of Go have you used?

This should be answered based on your usage but some examples are:

• fmt - formatted I/O

What is the problem with the following block of code? How to fix it?

func main() {
    var x float32 = 13.5
    var y int
    y = x
}

The following block of code tries to convert the integer 101 to a string but instead we get "e". Why is that? How to fix it?

package main

import "fmt"

func main() {
    var x int = 101
    var y string
    y = string(x)
    fmt.Println(y)
}

It looks what unicode value is set at 101 and uses it for converting the integer to a string. If you want to get "101" you should use the package "strconv" and replace y = string(x) with y = strconv.Itoa(x)

What is wrong with the following code?:

package main

func main() {
    var x = 2
    var y = 3
    const someConst = x + y
}

Constants in Go can only be declared using constant expressions. But x, y and their sum is variable.
const initializer x + y is not a constant

What will be the output of the following block of code?:

package main

import "fmt"

const (
	x = iota
	y = iota
)
const z = iota

func main() {
	fmt.Printf("%v\n", x)
	fmt.Printf("%v\n", y)
	fmt.Printf("%v\n", z)
}

Go's iota identifier is used in const declarations to simplify definitions of incrementing numbers. Because it can be used in expressions, it provides a generality beyond that of simple enumerations.
x and y in the first iota group, z in the second.
Iota page in Go Wiki

What _ is used for in Go?

It avoids having to declare all the variables for the returns values. It is called the blank identifier.
answer in SO

What will be the output of the following block of code?:

package main

import "fmt"

const (
	_ = iota + 3
	x
)

func main() {
	fmt.Printf("%v\n", x)
}

Since the first iota is declared with the value 3 ( + 3), the next one has the value 4

What will be the output of the following block of code?:

package main

import (
	"fmt"
	"sync"
	"time"
)

func main() {
	var wg sync.WaitGroup

	wg.Add(1)
	go func() {
		time.Sleep(time.Second * 2)
		fmt.Println("1")
		wg.Done()
	}()

	go func() {
		fmt.Println("2")
	}()

	wg.Wait()
	fmt.Println("3")
}

Output: 2 1 3

Aritcle about sync/waitgroup

Golang package sync

What will be the output of the following block of code?:

package main

import (
	"fmt"
)

func mod1(a []int) {
	for i := range a {
		a[i] = 5
	}

	fmt.Println("1:", a)
}

func mod2(a []int) {
	a = append(a, 125) // !

	for i := range a {
		a[i] = 5
	}

	fmt.Println("2:", a)
}

func main() {
	s1 := []int{1, 2, 3, 4}
	mod1(s1)
	fmt.Println("1:", s1)

	s2 := []int{1, 2, 3, 4}
	mod2(s2)
	fmt.Println("2:", s2)
}

Output:
1 [5 5 5 5]
1 [5 5 5 5]
2 [5 5 5 5 5]
2 [1 2 3 4]


In mod1 a is link, and when we're using a[i], we're changing s1 value to. But in mod2, append creats new slice, and we're changing only a value, not s2.

Aritcle about arrays, Blog post about append

What will be the output of the following block of code?:

package main

import (
	"container/heap"
	"fmt"
)

// An IntHeap is a min-heap of ints.
type IntHeap []int

func (h IntHeap) Len() int           { return len(h) }
func (h IntHeap) Less(i, j int) bool { return h[i] < h[j] }
func (h IntHeap) Swap(i, j int)      { h[i], h[j] = h[j], h[i] }

func (h *IntHeap) Push(x interface{}) {
	// Push and Pop use pointer receivers because they modify the slice's length,
	// not just its contents.
	*h = append(*h, x.(int))
}

func (h *IntHeap) Pop() interface{} {
	old := *h
	n := len(old)
	x := old[n-1]
	*h = old[0 : n-1]
	return x
}

func main() {
	h := &IntHeap{4, 8, 3, 6}
	heap.Init(h)
	heap.Push(h, 7)

  fmt.Println((*h)[0])
}

Output: 3

Golang container/heap package

Mongo

What are the advantages of MongoDB? Or in other words, why choosing MongoDB and not other implementation of NoSQL?

MongoDB advantages are as followings:

• Schemaless
• Easy to scale-out
• No complex joins
• Structure of a single object is clear

What is the difference between SQL and NoSQL?

The main difference is that SQL databases are structured (data is stored in the form of tables with rows and columns - like an excel spreadsheet table) while NoSQL is unstructured, and the data storage can vary depending on how the NoSQL DB is set up, such as key-value pair, document-oriented, etc.

In what scenarios would you prefer to use NoSQL/Mongo over SQL?

• Heterogeneous data which changes often
• Data consistency and integrity is not top priority
• Best if the database needs to scale rapidly

What is a document? What is a collection?

What is an aggregator?

What is better? Embedded documents or referenced?

Have you performed data retrieval optimizations in Mongo? If not, can you think about ways to optimize a slow data retrieval?

Queries

Explain this query: db.books.find({"name": /abc/})

Explain this query: db.books.find().sort({x:1})

What is the difference between find() and find_one()?

How can you export data from Mongo DB?

• mongoexport
• programming languages

SQL

SQL Exercises

Name	Topic	Objective & Instructions	Solution	Comments
Functions vs. Comparisons	Query Improvements	Exercise	Solution

SQL Self Assessment

What is SQL?

SQL (Structured Query Language) is a standard language for relational databases (like MySQL, MariaDB, ...).
It's used for reading, updating, removing and creating data in a relational database.

How is SQL Different from NoSQL

The main difference is that SQL databases are structured (data is stored in the form of tables with rows and columns - like an excel spreadsheet table) while NoSQL is unstructured, and the data storage can vary depending on how the NoSQL DB is set up, such as key-value pair, document-oriented, etc.

When is it best to use SQL? NoSQL?

SQL - Best used when data integrity is crucial. SQL is typically implemented with many businesses and areas within the finance field due to it's ACID compliance.

NoSQL - Great if you need to scale things quickly. NoSQL was designed with web applications in mind, so it works great if you need to quickly spread the same information around to multiple servers

Additionally, since NoSQL does not adhere to the strict table with columns and rows structure that Relational Databases require, you can store different data types together.

Practical SQL - Basics

For these questions, we will be using the Customers and Orders tables shown below:

Customers

Customer_ID	Customer_Name	Items_in_cart	Cash_spent_to_Date
100204	John Smith	0	20.00
100205	Jane Smith	3	40.00
100206	Bobby Frank	1	100.20

ORDERS

Customer_ID	Order_ID	Item	Price	Date_sold
100206	A123	Rubber Ducky	2.20	2019-09-18
100206	A123	Bubble Bath	8.00	2019-09-18
100206	Q987	80-Pack TP	90.00	2019-09-20
100205	Z001	Cat Food - Tuna Fish	10.00	2019-08-05
100205	Z001	Cat Food - Chicken	10.00	2019-08-05
100205	Z001	Cat Food - Beef	10.00	2019-08-05
100205	Z001	Cat Food - Kitty quesadilla	10.00	2019-08-05
100204	X202	Coffee	20.00	2019-04-29

How would I select all fields from this table?

Select *
From Customers;

How many items are in John's cart?

Select Items_in_cart
From Customers
Where Customer_Name = "John Smith";

What is the sum of all the cash spent across all customers?

Select SUM(Cash_spent_to_Date) as SUM_CASH
From Customers;

How many people have items in their cart?

Select count(1) as Number_of_People_w_items
From Customers
where Items_in_cart > 0;

How would you join the customer table to the order table?

You would join them on the unique key. In this case, the unique key is Customer_ID in both the Customers table and Orders table

How would you show which customer ordered which items?

Select c.Customer_Name, o.Item
From Customers c
Left Join Orders o
On c.Customer_ID = o.Customer_ID;

Using a with statement, how would you show who ordered cat food, and the total amount of money spent?

with cat_food as (
Select Customer_ID, SUM(Price) as TOTAL_PRICE
From Orders
Where Item like "%Cat Food%"
Group by Customer_ID
)
Select Customer_name, TOTAL_PRICE
From Customers c
Inner JOIN cat_food f
ON c.Customer_ID = f.Customer_ID
where c.Customer_ID in (Select Customer_ID from cat_food);

Although this was a simple statement, the "with" clause really shines when a complex query needs to be run on a table before joining to another. With statements are nice, because you create a pseudo temp when running your query, instead of creating a whole new table.

The Sum of all the purchases of cat food weren't readily available, so we used a with statement to create the pseudo table to retrieve the sum of the prices spent by each customer, then join the table normally.

Which of the following queries would you use?

SELECT count(*)                             SELECT count(*)
FROM shawarma_purchases                     FROM shawarma_purchases
WHERE                               vs.     WHERE
  YEAR(purchased_at) == '2017'              purchased_at >= '2017-01-01' AND
                                            purchased_at <= '2017-31-12'

SELECT count(*)
FROM shawarma_purchases
WHERE
  purchased_at >= '2017-01-01' AND
  purchased_at <= '2017-31-12'

When you use a function (YEAR(purchased_at)) it has to scan the whole database as opposed to using indexes and basically the column as it is, in its natural state.

OpenStack

What components/projects of OpenStack are you familiar with?

Can you tell me what each of the following services/projects is responsible for?:

• Nova
• Neutron
• Cinder
• Glance
• Keystone

• Nova - Manage virtual instances
• Cinder - Block Storage
• Keystone - Authentication service across the cloud

Identify the service/project used for each of the following:

• Copy or snapshot instances
• GUI for viewing and modifying resources
• Block Storage
• Manage virtual instances

• Glance - Images Service. Also used for copying or snapshot instances
• Horizon - GUI for viewing and modifying resources
• Cinder - Block Storage
• Nova - Manage virtual instances

What is a tenant/project?

Determine true or false:

• OpenStack is free to use
• The service responsible for networking is Glance
• The purpose of tenant/project is to share resources between different projects and users of OpenStack

Describe in detail how you bring up an instance with a floating IP

You get a call from a customer saying: "I can ping my instance but can't connect (ssh) it". What might be the problem?

What types of networks OpenStack supports?

How do you debug OpenStack storage issues? (tools, logs, ...)

How do you debug OpenStack compute issues? (tools, logs, ...)

OpenStack Deployment & TripleO

Have you deployed OpenStack in the past? If yes, can you describe how you did it?

Are you familiar with TripleO? How is it different from Devstack or Packstack?

You can read about TripleO right here

OpenStack Compute

Can you describe Nova in detail?

• Used to provision and manage virtual instances
• It supports Multi-Tenancy in different levels - logging, end-user control, auditing, etc.
• Highly scalable
• Authentication can be done using internal system or LDAP
• Supports multiple types of block storage
• Tries to be hardware and hypervisor agnostice

What do you know about Nova architecture and components?

• nova-api - the server which serves metadata and compute APIs
• the different Nova components communicate by using a queue (Rabbitmq usually) and a database
• a request for creating an instance is inspected by nova-scheduler which determines where the instance will be created and running
• nova-compute is the component responsible for communicating with the hypervisor for creating the instance and manage its lifecycle

OpenStack Networking (Neutron)

Explain Neutron in detail

• One of the core component of OpenStack and a standalone project
• Neutron focused on delivering networking as a service
• With Neutron, users can set up networks in the cloud and configure and manage a variety of network services
• Neutron interacts with:
  • Keystone - authorize API calls
  • Nova - nova communicates with neutron to plug NICs into a network
  • Horizon - supports networking entities in the dashboard and also provides topology view which includes networking details

Explain each of the following components:

• neutron-dhcp-agent
• neutron-l3-agent
• neutron-metering-agent
• neutron-*-agtent
• neutron-server

• neutron-l3-agent - L3/NAT forwarding (provides external network access for VMs for example)
• neutron-dhcp-agent - DHCP services
• neutron-metering-agent - L3 traffic metering
• neutron-*-agtent - manages local vSwitch configuration on each compute (based on chosen plugin)
• neutron-server - exposes networking API and passes requests to other plugins if required

Explain these network types:

• Management Network
• Guest Network
• API Network
• External Network

• Management Network - used for internal communication between OpenStack components. Any IP address in this network is accessible only within the datacetner
• Guest Network - used for communication between instances/VMs
• API Network - used for services API communication. Any IP address in this network is publicly accessible
• External Network - used for public communication. Any IP address in this network is accessible by anyone on the internet

In which order should you remove the following entities:

• Network
• Port
• Router
• Subnet

• Port
• Subnet
• Router
• Network

There are many reasons for that. One for example: you can't remove router if there are active ports assigned to it.

What is a provider network?

What components and services exist for L2 and L3?

What is the ML2 plug-in? Explain its architecture

What is the L2 agent? How does it works and what is it responsible for?

What is the L3 agent? How does it works and what is it responsible for?

Explain what the Metadata agent is responsible for

What networking entities Neutron supports?

How do you debug OpenStack networking issues? (tools, logs, ...)

OpenStack - Glance

Explain Glance in detail

• Glance is the OpenStack image service
• It handles requests related to instances disks and images
• Glance also used for creating snapshots for quick instances backups
• Users can use Glance to create new images or upload existing ones

Describe Glance architecture

• glance-api - responsible for handling image API calls such as retrieval and storage. It consists of two APIs: 1. registry-api - responsible for internal requests 2. user API - can be accessed publicly
• glance-registry - responsible for handling image metadata requests (e.g. size, type, etc). This component is private which means it's not available publicly
• metadata definition service - API for custom metadata
• database - for storing images metadata
• image repository - for storing images. This can be a filesystem, swift object storage, HTTP, etc.

OpenStack - Swift

Explain Swift in detail

• Swift is Object Store service and is an highly available, distributed and consistent store designed for storing a lot of data
• Swift is distributing data across multiple servers while writing it to multiple disks
• One can choose to add additional servers to scale the cluster. All while swift maintaining integrity of the information and data replications.

Can users store by default an object of 100GB in size?

Not by default. Object Storage API limits the maximum to 5GB per object but it can be adjusted.

Explain the following in regards to Swift:

• Container
• Account
• Object

• Container - Defines a namespace for objects.
• Account - Defines a namespace for containers
• Object - Data content (e.g. image, document, ...)

True or False? there can be two objects with the same name in the same container but not in two different containers

False. Two objects can have the same name if they are in different containers.

OpenStack - Cinder

Explain Cinder in detail

• Cinder is OpenStack Block Storage service
• It basically provides used with storage resources they can consume with other services such as Nova
• One of the most used implementations of storage supported by Cinder is LVM
• From user perspective this is transparent which means the user doesn't know where, behind the scenes, the storage is located or what type of storage is used

Describe Cinder's components

• cinder-api - receives API requests
• cinder-volume - manages attached block devices
• cinder-scheduler - responsible for storing volumes

OpenStack - Keystone

Can you describe the following concepts in regards to Keystone?

• Role
• Tenant/Project
• Service
• Endpoint
• Token

• Role - A list of rights and privileges determining what a user or a project can perform
• Tenant/Project - Logical representation of a group of resources isolated from other groups of resources. It can be an account, organization, ...
• Service - An endpoint which the user can use for accessing different resources
• Endpoint - a network address which can be used to access a certain OpenStack service
• Token - Used for access resources while describing which resources can be accessed by using a scope

What are the properties of a service? In other words, how a service is identified?

Using:

• Name
• ID number
• Type
• Description

Explain the following: - PublicURL - InternalURL - AdminURL

• PublicURL - Publicly accessible through public internet
• InternalURL - Used for communication between services
• AdminURL - Used for administrative management

What is a service catalog?

A list of services and their endpoints

OpenStack Advanced - Services

Describe each of the following services

• Swift
• Sahara
• Ironic
• Trove
• Aodh
• Ceilometer

• Swift - highly available, distributed, eventually consistent object/blob store
• Sahara - Manage Hadoop Clusters
• Ironic - Bare Metal Provisioning
• Trove - Database as a service that runs on OpenStack
• Aodh - Alarms Service
• Ceilometer - Track and monitor usage

Identify the service/project used for each of the following:

• Database as a service which runs on OpenStack
• Bare Metal Provisioning
• Track and monitor usage
• Alarms Service
• Manage Hadoop Clusters
• highly available, distributed, eventually consistent object/blob store

• Database as a service which runs on OpenStack - Trove
• Bare Metal Provisioning - Ironic
• Track and monitor usage - Ceilometer
• Alarms Service - Aodh
• Manage Hadoop Clusters
• Manage Hadoop Clusters - Sahara
• highly available, distributed, eventually consistent object/blob store - Swift

OpenStack Advanced - Keystone

Can you describe Keystone service in detail?

• You can't have OpenStack deployed without Keystone
• It Provides identity, policy and token services
  • The authentication provided is for both users and services
  • The authorization supported is token-based and user-based.
• There is a policy defined based on RBAC stored in a JSON file and each line in that file defines the level of access to apply

Describe Keystone architecture

• There is a service API and admin API through which Keystone gets requests
• Keystone has four backends:
  • Token Backend - Temporary Tokens for users and services
  • Policy Backend - Rules management and authorization
  • Identity Backend - users and groups (either standalone DB, LDAP, ...)
  • Catalog Backend - Endpoints
• It has pluggable environment where you can integrate with:
  • LDAP
  • KVS (Key Value Store)
  • SQL
  • PAM
  • Memcached

Describe the Keystone authentication process

• Keystone gets a call/request and checks whether it's from an authorized user, using username, password and authURL
• Once confirmed, Keystone provides a token.
• A token contains a list of user's projects so there is no to authenticate every time and a token can submitted instead

OpenStack Advanced - Compute (Nova)

What each of the following does?:

• nova-api
• nova-compuate
• nova-conductor
• nova-cert
• nova-consoleauth
• nova-scheduler

• nova-api - responsible for managing requests/calls
• nova-compute - responsible for managing instance lifecycle
• nova-conductor - Mediates between nova-compute and the database so nova-compute doesn't access it directly

What types of Nova proxies are you familiar with?

• Nova-novncproxy - Access through VNC connections
• Nova-spicehtml5proxy - Access through SPICE
• Nova-xvpvncproxy - Access through a VNC connection

OpenStack Advanced - Networking (Neutron)

Explain BGP dynamic routing

What is the role of network namespaces in OpenStack?

OpenStack Advanced - Horizon

Can you describe Horizon in detail?

• Django-based project focusing on providing an OpenStack dashboard and the ability to create additional customized dashboards
• You can use it to access the different OpenStack services resources - instances, images, networks, ...
  • By accessing the dashboard, users can use it to list, create, remove and modify the different resources
• It's also highly customizable and you can modify or add to it based on your needs

What can you tell about Horizon architecture?

• API is backward compatible
• There are three type of dashboards: user, system and settings
• It provides core support for all OpenStack core projects such as Neutron, Nova, etc. (out of the box, no need to install extra packages or plugins)
• Anyone can extend the dashboards and add new components
• Horizon provides templates and core classes from which one can build its own dashboard

Puppet

What is Puppet? How does it works?

• Puppet is a configuration management tool ensuring that all systems are configured to a desired and predictable state.

Explain Puppet architecture

• Puppet has a primary-secondary node architecture. The clients are distributed across the network and communicate with the primary-secondary environment where Puppet modules are present. The client agent sends a certificate with its ID to the server; the server then signs that certificate and sends it back to the client. This authentication allows for secure and verifiable communication between the client and the master.

Can you compare Puppet to other configuration management tools? Why did you chose to use Puppet?

Explain the following:

• Module
• Manifest
• Node

• Modules - are a collection of manifests, templates, and files
• Manifests - are the actual codes for configuring the clients
• Node - allows you to assign specific configurations to specific nodes

Explain Facter

What is MCollective?

Do you have experience with writing modules? Which module have you created and for what?

Explain what is Hiera

Elastic

What is the Elastic Stack?

The Elastic Stack consists of:

• Elasticsearch
• Kibana
• Logstash
• Beats
• Elastic Hadoop
• APM Server

Elasticserach, Logstash and Kibana are also known as the ELK stack.

Explain what is Elasticsearch

From the official docs:

"Elasticsearch is a distributed document store. Instead of storing information as rows of columnar data, Elasticsearch stores complex data structures that have been serialized as JSON documents"

What is Logstash?

From the blog:

"Logstash is a powerful, flexible pipeline that collects, enriches and transports data. It works as an extract, transform & load (ETL) tool for collecting log messages."

Explain what beats are

Beats are lightweight data shippers. These data shippers installed on the client where the data resides. Examples of beats: Filebeat, Metricbeat, Auditbeat. There are much more.

What is Kibana?

From the official docs:

"Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch indices. You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps."

Describe what happens from the moment an app logged some information until it's displayed to the user in a dashboard when the Elastic stack is used

The process may vary based on the chosen architecture and the processing you may want to apply to the logs. One possible workflow is:

1. The data logged by the application is picked by filebeat and sent to logstash
2. Logstash process the log based on the defined filters. Once done, the output is sent to Elasticsearch
3. Elasticsearch stores the document it got and the document is indexed for quick future access
4. The user creates visualizations in Kibana which based on the indexed data
5. The user creates a dashboard which composed out of the visualization created in the previous step

Elasticsearch

What is a data node?

This is where data is stored and also where different processing takes place (e.g. when you search for a data).

What is a master node?

Par of a master node responsibilites:

• Track the status of all the nodes in the cluster
• Verify replicas are working and the data is available from every data node.
• No hot nodes (no data node that works much harder than other nodes)

While there can be multiple master nodes in reality only of them is the elected master node.

What is an ingest node?

A node which responsible for parsing the data. In case you don't use logstash then this node can recieve data from beats and parse it, similarly to how it can be parsed in Logstash.

What is Coordinating node?

A Coordinating node responsible for routing requests out and in to the cluser (data nodes).

How data is stored in elasticsearch?

• Data is stored in an index
• The index is spread across the cluster using shards

What is an Index?

Index in Elastic is in most cases compared to a whole database from the SQL/NoSQL world.
You can choose to have one index to hold all the data of your app or have multiple indices where each index holds different type of your app (e.g. index for each service your app is running).

The official docs also offer a great explanation (in general, it's really good documentation, as every project should have):

"An index can be thought of as an optimized collection of documents and each document is a collection of fields, which are the key-value pairs that contain your data"

Explain Shards

An index is split into shards and documents are hashed to a particular shard. Each shard may be on a different node in a cluster and each one of the shards is a self contained index.
This allows Elasticsearch to scale to an entire cluster of servers.

What is an Inverted Index?

From the official docs:

"An inverted index lists every unique word that appears in any document and identifies all of the documents each word occurs in."

What is a Document?

Continuing with the comparison to SQL/NoSQL a Document in Elastic is a row in table in the case of SQL or a document in a collection in the case of NoSQL. As in NoSQL a Document is a JSON object which holds data on a unit in your app. What is this unit depends on the your app. If your app related to book then each document describes a book. If you are app is about shirts then each document is a shirt.

You check the health of your elasticsearch cluster and it's red. What does it mean? What can cause the status to be yellow instead of green?

Red means some data is unavailable. Yellow can be caused by running single node cluster instead of multi-node.

True or False? Elasticsearch indexes all data in every field and each indexed field has the same data structure for unified and quick query ability

False. From the official docs:

"Each indexed field has a dedicated, optimized data structure. For example, text fields are stored in inverted indices, and numeric and geo fields are stored in BKD trees."

What reserved fields a document has?

• _index
• _id
• _type

Explain Mapping

What are the advantages of defining your own mapping? (or: when would you use your own mapping?)

• You can optimize fields for partial matching
• You can define custom formats of known fields (e.g. date)
• You can perform language-specific analysis

Explain Replicas

In a network/cloud environment where failures can be expected any time, it is very useful and highly recommended to have a failover mechanism in case a shard/node somehow goes offline or disappears for whatever reason. To this end, Elasticsearch allows you to make one or more copies of your index’s shards into what are called replica shards, or replicas for short.

Can you explain Term Frequency & Document Frequency?

Term Frequency is how often a term appears in a given document and Document Frequency is how often a term appears in all documents. They both are used for determining the relevance of a term by calculating Term Frequency / Document Frequency.

You check "Current Phase" under "Index lifecycle management" and you see it's set to "hot". What does it mean?

"The index is actively being written to". More about the phases here

What this command does? curl -X PUT "localhost:9200/customer/_doc/1?pretty" -H 'Content-Type: application/json' -d'{ "name": "John Doe" }'

It creates customer index if it doesn't exists and adds a new document with the field name which is set to "John Dow". Also, if it's the first document it will get the ID 1.

What will happen if you run the previous command twice? What about running it 100 times?

1. If name value was different then it would update "name" to the new value
2. In any case, it bumps version field by one

What is the Bulk API? What would you use it for?

Bulk API is used when you need to index multiple documents. For high number of documents it would be significantly faster to use rather than individual requests since there are less network roundtrips.

Query DSL

Explain Elasticsearch query syntax (Booleans, Fields, Ranges)

Explain what is Relevance Score

Explain Query Context and Filter Context

From the official docs:

"In the query context, a query clause answers the question “How well does this document match this query clause?” Besides deciding whether or not the document matches, the query clause also calculates a relevance score in the _score meta-field."

"In a filter context, a query clause answers the question “Does this document match this query clause?” The answer is a simple Yes or No — no scores are calculated. Filter context is mostly used for filtering structured data"

Describe how would an architecture of production environment with large amounts of data would be different from a small-scale environment

There are several possible answers for this question. One of them is as follows:

A small-scale architecture of elastic will consist of the elastic stack as it is. This means we will have beats, logstash, elastcsearch and kibana.
A production environment with large amounts of data can include some kind of buffering component (e.g. Reddis or RabbitMQ) and also security component such as Nginx.

Logstash

What are Logstash plugins? What plugins types are there?

• Input Plugins - how to collect data from different sources
• Filter Plugins - processing data
• Output Plugins - push data to different outputs/services/platforms

What is grok?

A logstash plugin which modifies information in one format and immerse it in another.

How grok works?

What grok patterns are you familiar with?

What is `_grokparsefailure?`

How do you test or debug grok patterns?

What are Logstash Codecs? What codecs are there?

Kibana

What can you find under "Discover" in Kibana?

The raw data as it is stored in the index. You can search and filter it.

You see in Kibana, after clicking on Discover, "561 hits". What does it mean?

Total number of documents matching the search results. If not query used then simply the total number of documents.

What can you find under "Visualize"?

"Visualize" is where you can create visual representations for your data (pie charts, graphs, ...)

What visualization types are supported/included in Kibana?

What visualization type would you use for statistical outliers

Describe in detail how do you create a dashboard in Kibana

Filebeat

What is Filebeat?

If one is using ELK, is it a must to also use filebeat? In what scenarios it's useful to use filebeat?

What is a harvester?

Read here

True or False? a single harvester harvest multiple files, according to the limits set in filebeat.yml

False. One harvester harvests one file.

What are filebeat modules?

Elastic Stack

How do you secure an Elastic Stack?

You can generate certificates with the provided elastic utils and change configuration to enable security using certificates model.

Distributed

Explain Distributed Computing (or Distributed System)

According to Martin Kleppmann:

"Many processes running on many machines...only message-passing via an unreliable network with variable delays, and the system may suffer from partial failures, unreliable clocks, and process pauses."

Another definition: "Systems that are physically separated, but logically connected"

What can cause a system to fail?

• Network
• CPU
• Memory
• Disk

Do you know what is "CAP theorem"? (aka as Brewer's theorem)

According to the CAP theorem, it's not possible for a distributed data store to provide more than two of the following at the same time:

• Availability: Every request receives a response (it doesn't has to be the most recent data)
• Consistency: Every request receives a response with the latest/most recent data
• Partition tolerance: Even if some the data is lost/dropped, the system keeps running

What are the problems with the following design? How to improve it?

1. The transition can take time. In other words, noticeable downtime. 2. Standby server is a waste of resources - if first application server is running then the standby does nothing

What are the problems with the following design? How to improve it?

Issues: If load balancer dies , we lose the ability to communicate with the application.

Ways to improve:

• Add another load balancer
• Use DNS A record for both load balancers
• Use message queue

What is "Shared-Nothing" architecture?

It's an architecture in which data is and retrieved from a single, non-shared, source usually exclusively connected to one node as opposed to architectures where the request can get to one of many nodes and the data will be retrieved from one shared location (storage, memory, ...).

Explain the Sidecar Pattern (Or sidecar proxy)

Misc

Name	Topic	Objective & Instructions	Solution	Comments
Highly Available "Hello World"	Exercise	Solution

What happens when you type in a URL in an address bar in a browser?

1. The browser searches for the record of the domain name IP address in the DNS in the following order:

• Browser cache
• Operating system cache
• The DNS server configured on the user's system (can be ISP DNS, public DNS, ...)

2. If it couldn't find a DNS record locally, a full DNS resolution is started.
3. It connects to the server using the TCP protocol
4. The browser sends an HTTP request to the server
5. The server sends an HTTP response back to the browser
6. The browser renders the response (e.g. HTML)
7. The browser then sends subsequent requests as needed to the server to get the embedded links, javascript, images in the HTML and then steps 3 to 5 are repeated.

TODO: add more details!

API

Explain what is an API

I like this definition from blog.christianposta.com:

"An explicitly and purposefully defined interface designed to be invoked over a network that enables software developers to get programmatic access to data and functionality within an organization in a controlled and comfortable way."

What is an API specification?

From swagger.io:

"An API specification provides a broad understanding of how an API behaves and how the API links with other APIs. It explains how the API functions and the results to expect when using the API"

True or False? API Definition is the same as API Specification

False. From swagger.io:

"An API definition is similar to an API specification in that it provides an understanding of how an API is organized and how the API functions. But the API definition is aimed at machine consumption instead of human consumption of APIs."

What is an API gateway?

What are the advantages of using/implementing an API gateway?

What is a Payload in API?

What is Automation? How it's related or different from Orchestration?

Automation is the act of automating tasks to reduce human intervention or interaction in regards to IT technology and systems.
While automation focuses on a task level, Orchestration is the process of automating processes and/or workflows which consists of multiple tasks that usually across multiple systems.

Tell me about interesting bugs you've found and also fixed

What is a Debuggger and how it works?

What services an application might have?

• Authorization
• Logging
• Authentication
• Ordering
• Front-end
• Back-end ...

What is Metadata?

Data about data. Basically, it describes the type of information that an underlying data will hold.

You can use one of the following formats: JSON, YAML, XML. Which one would you use? Why?

I can't answer this for you :)

What's KPI?

What's OKR?

What's DSL (Domain Specific Language)?

Domain Specific Language (DSLs) are used to create a customised language that represents the domain such that domain experts can easily interpret it.

What's the difference between KPI and OKR?

YAML

What is YAML?

Data serialization language used by many technologies today like Kubernetes, Ansible, etc.

True or False? Any valid JSON file is also a valid YAML file

True. Because YAML is superset of JSON.

What is the format of the following data?

{
    applications: [
        {
            name: "my_app",
            language: "python",
            version: 20.17
        }
    ]
}

JSON

What is the format of the following data?

applications:
  - app: "my_app"
    language: "python"
    version: 20.17

YAML

How to write a multi-line string with YAML? What use cases is it good for?

someMultiLineString: |
  look mama
  I can write a multi-line string
  I love YAML

It's good for use cases like writing a shell script where each line of the script is a different command.

What is the difference between someMultiLineString: | to someMultiLineString: >?

using > will make the multi-line string to fold into a single line

someMultiLineString: >
  This is actually
  a single line
  do not let appearances fool you

What are placeholders in YAML?

They allow you reference values instead of directly writing them and it is used like this:

username: {{ my.user_name }}

How can you define multiple YAML components in one file?

Using this: --- For Examples:

document_number: 1
---
document_number: 2

Firmware

Explain what is a firmware

Wikipedia: "In computing, firmware is a specific class of computer software that provides the low-level control for a device's specific hardware. Firmware, such as the BIOS of a personal computer, may contain basic functions of a device, and may provide hardware abstraction services to higher-level software such as operating systems."

Cassandra

When running a cassandra cluster, how often do you need to run nodetool repair in order to keep the cluster consistent?

• Within the columnFamily GC-grace Once a week
• Less than the compacted partition minimum bytes
• Depended on the compaction strategy

HTTP

What is HTTP?

Avinetworks: HTTP stands for Hypertext Transfer Protocol. HTTP uses TCP port 80 to enable internet communication. It is part of the Application Layer (L7) in OSI Model.

Describe HTTP request lifecycle

• Resolve host by request to DNS resolver
• Client SYN
• Server SYN+ACK
• Client SYN
• HTTP request
• HTTP response

True or False? HTTP is stateful

False. It doesn't maintain state for incoming request.

How HTTP request looks like?

It consists of:

• Request line - request type
• Headers - content info like length, enconding, etc.
• Body (not always included)

What HTTP method types are there?

• GET
• POST
• HEAD
• PUT
• DELETE
• CONNECT
• OPTIONS
• TRACE

What HTTP response codes are there?

• 1xx - informational
• 2xx - Success
• 3xx - Redirect
• 4xx - Error, client fault
• 5xx - Error, server fault

What is HTTPS?

Explain HTTP Cookies

HTTP is stateless. To share state, we can use Cookies.

TODO: explain what is actually a Cookie

What is HTTP Pipelining?

You get "504 Gateway Timeout" error from an HTTP server. What does it mean?

The server didn't receive a response from another server it communicates with in a timely manner.

What is a proxy?

What is a reverse proxy?

When you publish a project, you usually publish it with a license. What types of licenses are you familiar with and which one do you prefer to use?

Explain what is "X-Forwarded-For"

Wikipedia: "The X-Forwarded-For (XFF) HTTP header field is a common method for identifying the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer."

Load Balancers

What is a load balancer?

A load balancer accepts (or denies) incoming network traffic from a client, and based on some criteria (application related, network, etc.) it distributes those communications out to servers (at least one).

Why to used a load balancer?

• Scalability - using a load balancer, you can possibly add more servers in the backend to handle more requests/traffic from the clients, as opposed to using one server.
• Redundancy - if one server in the backend dies, the load balancer will keep forwarding the traffic/requests to the second server so users won't even notice one of the servers in the backend is down.

What load balancer techniques/algorithms are you familiar with?

• Round Robin
• Weighted Round Robin
• Least Connection
• Weighted Least Connection
• Resource Based
• Fixed Weighting
• Weighted Response Time
• Source IP Hash
• URL Hash

What are the drawbacks of round robin algorithm in load balancing?

• A simple round robin algorithm knows nothing about the load and the spec of each server it forwards the requests to. It is possible, that multiple heavy workloads requests will get to the same server while other servers will got only lightweight requests which will result in one server doing most of the work, maybe even crashing at some point because it unable to handle all the heavy workloads requests by its own.
• Each request from the client creates a whole new session. This might be a problem for certain scenarios where you would like to perform multiple operations where the server has to know about the result of operation so basically, being sort of aware of the history it has with the client. In round robin, first request might hit server X, while second request might hit server Y and ask to continue processing the data that was processed on server X already.

What is an Application Load Balancer?

In which scenarios would you use ALB?

At what layers a load balancer can operate?

L4 and L7

Can you perform load balancing without using a dedicated load balancer instance?

Yes, you can use DNS for performing load balancing.

What is DNS load balancing? What its advantages? When would you use it?

Load Balancers - Sticky Sessions

What are sticky sessions? What are their pros and cons?

Recommended read:

• Red Hat Article

Cons:

• Can cause uneven load on instance (since requests routed to the same instances) Pros:
• Ensures in-proc sessions are not lost when a new request is created

Name one use case for using sticky sessions

You would like to make sure the user doesn't lose the current session data.

What sticky sessions use for enabling the "stickiness"?

Cookies. There are application based cookies and duration based cookies.

Explain application-based cookies

• Generated by the application and/or the load balancer
• Usually allows to include custom data

Explain duration-based cookies

• Generated by the load balancer
• Session is not sticky anymore once the duration elapsed

Load Balancers - Load Balancing Algorithms

Explain each of the following load balancing techniques

• Round Robin
• Weighted Round Robin
• Least Connection
• Weighted Least Connection
• Resource Based
• Fixed Weighting
• Weighted Response Time
• Source IP Hash
• URL Hash

Explain use case for connection draining?

To ensure that a Classic Load Balancer stops sending requests to instances that are de-registering or unhealthy, while keeping the existing connections open, use connection draining. This enables the load balancer to complete in-flight requests made to instances that are de-registering or unhealthy.

The maximum timeout value can be set between 1 and 3,600 seconds on both GCP and AWS.

Licenses

Are you familiar with "Creative Commons"? What do you know about it?

Explain the differences between copyleft and permissive licenses

In Copyleft, any derivative work must use the same licensing while in permissive licensing there are no such condition. GPL-3 is an example of copyleft license while BSD is an example of permissive license.

Random

How a search engine works?

How auto completion works?

What is faster than RAM?

CPU cache. Source

What is a memory leak?

What is your favorite protocol?

SSH HTTP DHCP DNS ...

What is Cache API?

What is the C10K problem? Is it relevant today?

https://idiallo.com/blog/c10k-2016

Storage

What types of storage are there?

• File
• Block
• Object

Explain Object Storage

• Data is divided to self-contained objects
• Objects can contain metadata

What are the pros and cons of object storage?

Pros:

• Usually with object storage, you pay for what you use as opposed to other storage types where you pay for the storage space you allocate
• Scalable storage: Object storage mostly based on a model where what you use, is what you get and you can add storage as need Cons:
• Usually performs slower than other types of storage
• No granular modification: to change an object, you have re-create it

What are some use cases for using object storage?

Explain File Storage

• File Storage used for storing data in files, in a hierarchical sturcture
• Some of the devices for file storage: hard drive, flash drive, cloud-based file storage
• Files usually organized in directories

What are the pros and cons of File Storage?

Pros:

• Users have full control of their own files and can run variety of operations on the files: delete, read, write and move.
• Security mechanism allows for users to have a better control at things such as file locking

What are some examples of file storage?

Local filesystem Dropbox Google Drive

What types of storage devices are there?

Explain IOPS

Explain storage throughput

What is a filesystem?

Explain Dark Data

Explain MBR

Questions you CAN ask

A list of questions you as a candidate can ask the interviewer during or after the interview. These are only a suggestion, use them carefully. Not every interviewer will be able to answer these (or happy to) which should be perhaps a red flag warning for your regarding working in such place but that's really up to you.

What do you like about working here?

How does the company promote personal growth?

What is the current level of technical debt you are dealing with?

Be careful when asking this question - all companies, regardless of size, have some level of tech debt. Phrase the question in the light that all companies have the deal with this, but you want to see the current pain points they are dealing with

This is a great way to figure how managers deal with unplanned work, and how good they are at setting expectations with projects.

Why I should NOT join you? (or 'what you don't like about working here?')

What was your favorite project you've worked on?

This can give you insights in some of the cool projects a company is working on, and if you would enjoy working on projects like these. This is also a good way to see if the managers are allowing employees to learn and grow with projects outside of the normal work you'd do.

If you could change one thing about your day to day, what would it be?

Similar to the tech debt question, this helps you identify any pain points with the company. Additionally, it can be a great way to show how you'd be an asset to the team.

For Example, if they mention they have problem X, and you've solved that in the past, you can show how you'd be able to mitigate that problem.

Let's say that we agree and you hire me to this position, after X months, what do you expect that I have achieved?

Not only this will tell you what is expected from you, it will also provide big hint on the type of work you are going to do in the first months of your job.

Testing

Explain white-box testing

Explain black-box testing

What are unit tests?

What types of tests would you run to test a web application?

Explain test harness?

What is A/B testing?

What is network simulation and how do you perform it?

What types of performances tests are you familiar with?

Explain the following types of tests:

• Load Testing
• Stress Testing
• Capacity Testing
• Volume Testing
• Endurance Testing

Databases

Name	Topic	Objective & Instructions	Solution	Comments
Message Board Tables	Relational DB Tables	Exercise	Solution

What is a relational database?

• Data Storage: system to store data in tables
• SQL: programming language to manage relational databases
• Data Definition Language: a standard syntax to create, alter and delete tables

What does it mean when a database is ACID compliant?

ACID stands for Atomicity, Consistency, Isolation, Durability. In order to be ACID compliant, the database must meet each of the four criteria

Atomicity - When a change occurs to the database, it should either succeed or fail as a whole.

For example, if you were to update a table, the update should completely execute. If it only partially executes, the update is considered failed as a whole, and will not go through - the DB will revert back to it's original state before the update occurred. It should also be mentioned that Atomicity ensures that each transaction is completed as it's own stand alone "unit" - if any part fails, the whole statement fails.

Consistency - any change made to the database should bring it from one valid state into the next.

For example, if you make a change to the DB, it shouldn't corrupt it. Consistency is upheld by checks and constraints that are pre-defined in the DB. For example, if you tried to change a value from a string to an int when the column should be of datatype string, a consistent DB would not allow this transaction to go through, and the action would not be executed

Isolation - this ensures that a database will never be seen "mid-update" - as multiple transactions are running at the same time, it should still leave the DB in the same state as if the transactions were being run sequentially.

For example, let's say that 20 other people were making changes to the database at the same time. At the time you executed your query, 15 of the 20 changes had gone through, but 5 were still in progress. You should only see the 15 changes that had completed - you wouldn't see the database mid-update as the change goes through.

Durability - Once a change is committed, it will remain committed regardless of what happens (power failure, system crash, etc.). This means that all completed transactions must be recorded in non-volatile memory.

Note that SQL is by nature ACID compliant. Certain NoSQL DB's can be ACID compliant depending on how they operate, but as a general rule of thumb, NoSQL DB's are not considered ACID compliant

What is sharding?

Sharding is a horizontal partitioning.

Are you able to explain what is it good for?

You find out your database became a bottleneck and users experience issues accessing data. How can you deal with such situation?

Not much information provided as to why it became a bottleneck and what is current architecture, so one general approach could be
to reduce the load on your database by moving frequently-accessed data to in-memory structure.

What is a connection pool?

Connection Pool is a cache of database connections and the reason it's used is to avoid an overhead of establishing a connection for every query done to a database.

What is a connection leak?

A connection leak is a situation where database connection isn't closed after being created and is no longer needed.

What is Table Lock?

Your database performs slowly than usual. More specifically, your queries are taking a lot of time. What would you do?

• Query for running queries and cancel the irrelevant queries
• Check for connection leaks (query for running connections and include their IP)
• Check for table locks and kill irrelevant locking sessions

What is a Data Warehouse?

"A data warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of data in support of organisation's decision-making process"

Explain what is a time-series database

What is OLTP (Online transaction processing)?

What is OLAP (Online Analytical Processing)?

What is an index in a database?

A database index is a data structure that improves the speed of operations in a table. Indexes can be created using one or more columns, providing the basis for both rapid random lookups and efficient ordering of access to records.

What data types are there in relational databases?

Explain Normalization

Data that is used multiple times in a database should be stored once and referenced with a foreign key.
This has the clear benefit of ease of maintenance where you need to change a value only in a single place to change it everywhere.

Explain Primary Key and Foreign Key

Primary Key: each row in every table should a unique identifier that represents the row.
Foreign Key: a reference to another table's primary key. This allows you to join table together to retrieve all the information you need without duplicating data.

What types of data tables have you used?

• Primary data table: main data you care about
• Details table: includes a foreign key and has one to many relationship
• Lookup values table: can be one table per lookup or a table containing all the lookups and has one to many relationship
• Multi reference table

What is ORM? What benefits it provides in regards to relational databases usage?

Wikipedia: "is a programming technique for converting data between incompatible type systems using object-oriented programming languages"

In regards to the relational databases:

• Database as code
• Database abstraction
• Encapsulates SQL complexity
• Enables code review process
• Enables usage as a native OOP structure

What is DDL?

Wikipedia: "In the context of SQL, data definition or data description language (DDL) is a syntax for creating and modifying database objects such as tables, indices, and users."

Regex

Given a text file, perform the following exercises

Extract

Extract all the numbers

Extract the first word of each line

Bonus: extract the last word of each line

Extract all the IP addresses

Extract dates in the format of yyyy-mm-dd or yyyy-dd-mm

Extract email addresses

Replace

Replace tabs with four spaces

Replace 'red' with 'green'

System Design

Explain what is a "Single point of failure"?

What is CDN?

CDN (Content Delivery Network) responsible for distributing content geographically. Part of it, is what is known as edge locations, aka cache proxies, that allows users to get their content quickly due to cache features and geographical distribution.

Explain Multi-CDN

In single CDN, the whole content is originated from content delivery network.
In multi-CDN, content is distributed across multiple different CDNs, each might be on a completely different provider/cloud.

What are the benefits of Multi-CDN over a single CDN?

• Resiliency: Relying on one CDN means no redundancy. With multiple CDNs you don't need to worry about your CDN being down
• Flexibility in Costs: Using one CDN enforces you to specific rates of that CDN. With multiple CDNs you can take into consideration using less expensive CDNs to deliver the content.
• Performance: With Multi-CDN there is bigger potential in choosing better locations which more close to the client asking the content
• Scale: With multiple CDNs, you can scale services to support more extreme conditions

Explain "3-Tier Architecture" (including pros and cons)

Explain Mono-repo vs. Multi-repo. What are the cons and pros of each approach?

What are the drawbacks of monolithic architecture?

• Not suitable for frequent code changes and the ability to deploy new features
• Not designed for today's infrastructure (like public clouds)
• Scaling a team to work monolithic architecture is more challenging

What are the advantages of microservices architecture over a monolithic architecture?

• Each of the services individually fail without escalating into an application-wide outage.
• Each service can be developed and maintained by a separate team and this team can choose its own tools and coding language

What's a service mesh?

This article provides a great explanation.

Explain "Loose Coupling"

What is a message queue? When is it used?

Scalability

Explain Scalability

The ability easily grow in size and capacity based on demand and usage.

Explain Elasticity

The ability to grow but also to reduce based on what is required

Explain Disaster Recovery

Explain Fault Tolerance and High Availability

Fault Tolerance - The ability to self-heal and return to normal capacity. Also the ability to withstand a failure and remain functional.

High Availability - Being able to access a resource (in some use cases, using different platforms)

What is the difference between high availability and Disaster Recovery?

wintellect.com: "High availability, simply put, is eliminating single points of failure and disaster recovery is the process of getting a system back to an operational state when a system is rendered inoperative. In essence, disaster recovery picks up when high availability fails, so HA first."

Explain Vertical Scaling

Vertical Scaling is the process of adding resources to increase power of existing servers. For example, adding more CPUs, adding more RAM, etc.

What are the disadvantages of Vertical Scaling?

With vertical scaling alone, the component still remains a single point of failure. In addition, it has hardware limit where if you don't have more resources, you might not be able to scale vertically.

Which type of cloud services usually support vertical scaling?

Databases, cache. It's common mostly for non-distributed systems.

Explain Horizontal Scaling

Horizontal Scaling is the process of adding more resources that will be able handle requests as one unit

What is the disadvantage of Horizontal Scaling? What is often required in order to perform Horizontal Scaling?

A load balancer. You can add more resources, but if you would like them to be part of the process, you have to serve them the requests/responses. Also, data inconsistency is a concern with horizontal scaling.

Explain in which use cases will you use vertical scaling and in which use cases you will use horizontal scaling

Explain Resiliency and what ways are there to make a system more resilient

Explain "Consistent Hashing"

How would you update each of the services in the following drawing without having app (foo.com) downtime?

What is the problem with the following architecture and how would you fix it?

The load on the producers or consumers may be high which will then cause them to hang or crash.
Instead of working in "push mode", the consumers can pull tasks only when they are ready to handle them. It can be fixed by using a streaming platform like Kafka, Kinesis, etc. This platform will make sure to handle the high load/traffic and pass tasks/messages to consumers only when the ready to get them.

Users report that there is huge spike in process time when adding little bit more data to process as an input. What might be the problem?

How would you scale the architecture from the previous question to hundreds of users?

Cache

What is "cache"? In which cases would you use it?

What is "distributed cache"?

What is a "cache replacement policy"?

Take a look here

Which cache replacement policies are you familiar with?

You can find a list here

Explain the following cache policies:

• FIFO
• LIFO
• LRU

Read about it here

Why not writing everything to cache instead of a database/datastore?

Migrations

How you prepare for a migration? (or plan a migration)

You can mention:

roll-back & roll-forward cut over dress rehearsals DNS redirection

Explain "Branch by Abstraction" technique

Design a system

Can you design a video streaming website?

Can you design a photo upload website?

How would you build a URL shortener?

More System Design Questions

Additional exercises can be found in system-design-notebook repository.

Hardware

What is a CPU?

A central processing unit (CPU) performs basic arithmetic, logic, controlling, and input/output (I/O) operations specified by the instructions in the program. This contrasts with external components such as main memory and I/O circuitry, and specialized processors such as graphics processing units (GPUs).

What is RAM?

RAM (Random Access Memory) is the hardware in a computing device where the operating system (OS), application programs and data in current use are kept so they can be quickly reached by the device's processor. RAM is the main memory in a computer. It is much faster to read from and write to than other kinds of storage, such as a hard disk drive (HDD), solid-state drive (SSD) or optical drive.

What is an embedded system?

An embedded system is a computer system - a combination of a computer processor, computer memory, and input/output peripheral devices—that has a dedicated function within a larger mechanical or electronic system. It is embedded as part of a complete device often including electrical or electronic hardware and mechanical parts.

Can you give an example of an embedded system?

Raspberry Pi

What types of storage are there?

Big Data

Explain what is exactly Big Data

As defined by Doug Laney:

• Volume: Extremely large volumes of data
• Velocity: Real time, batch, streams of data
• Variety: Various forms of data, structured, semi-structured and unstructured
• Veracity or Variability: Inconsistent, sometimes inaccurate, varying data

What is DataOps? How is it related to DevOps?

DataOps seeks to reduce the end-to-end cycle time of data analytics, from the origin of ideas to the literal creation of charts, graphs and models that create value. DataOps combines Agile development, DevOps and statistical process controls and applies them to data analytics.

What is Data Architecture?

An answer from talend.com:

"Data architecture is the process of standardizing how organizations collect, store, transform, distribute, and use data. The goal is to deliver relevant data to people who need it, when they need it, and help them make sense of it."

Explain the different formats of data

• Structured - data that has defined format and length (e.g. numbers, words)
• Semi-structured - Doesn't conform to a specific format but is self-describing (e.g. XML, SWIFT)
• Unstructured - does not follow a specific format (e.g. images, test messages)

What is a Data Warehouse?

Wikipedia's explanation on Data Warehouse Amazon's explanation on Data Warehouse

What is Data Lake?

Data Lake - Wikipedia

Can you explain the difference between a data lake and a data warehouse?

What is "Data Versioning"? What models of "Data Versioning" are there?

What is ETL?

Apache Hadoop

Explain what is Hadoop

Apache Hadoop - Wikipedia

Explain Hadoop YARN

Responsible for managing the compute resources in clusters and scheduling users' applications

Explain Hadoop MapReduce

A programming model for large-scale data processing

Explain Hadoop Distributed File Systems (HDFS)

• Distributed file system providing high aggregate bandwidth across the cluster.
• For a user it looks like a regular file system structure but behind the scenes it's distributed across multiple machines in a cluster
• Typical file size is TB and it can scale and supports millions of files
• It's fault tolerant which means it provides automatic recovery from faults
• It's best suited for running long batch operations rather than live analysis

What do you know about HDFS architecture?

HDFS Architecture

• Master-slave architecture
• Namenode - master, Datanodes - slaves
• Files split into blocks
• Blocks stored on datanodes
• Namenode controls all metadata

Ceph

Explain what is Ceph

True or False? Ceph favor consistency and correctness over performances

True

Which services or types of storage Ceph supports?

• Object (RGW)
• Block (RBD)
• File (CephFS)

What is RADOS?

• Reliable Autonomic Distributed Object Storage
• Provides low-level data object storage service
• Strong Consistency
• Simplifies design and implementation of higher layers (block, file, object)

Describe RADOS software components

• Monitor
  • Central authority for authentication, data placement, policy
  • Coordination point for all other cluster components
  • Protect critical cluster state with Paxos
• Manager
  • Aggregates real-time metrics (throughput, disk usage, etc.)
  • Host for pluggable management functions
  • 1 active, 1+ standby per cluster
• OSD (Object Storage Daemon)
  • Stores data on an HDD or SSD
  • Services client IO requests

What is the workflow of retrieving data from Ceph?

What is the workflow of retrieving data from Ceph?

What are "Placement Groups"?

Describe in the detail the following: Objects -> Pool -> Placement Groups -> OSDs

What is OMAP?

What is a metadata server? How it works?

Packer

What is Packer? What is it used for?

In general, Packer automates machine images creation. It allows you to focus on configuration prior to deployment while making the images. This allows you start the instances much faster in most cases.

Packer follows a "configuration->deployment" model or "deployment->configuration"?

A configuration->deployment which has some advantages like:

1. Deployment Speed - you configure once prior to deployment instead of configuring every time you deploy. This allows you to start instances/services much quicker.
2. More immutable infrastructure - with configuration->deployment it's not likely to have very different deployments since most of the configuration is done prior to the deployment. Issues like dependencies errors are handled/discovered prior to deployment in this model.

Release

Explain Semantic Versioning

This page explains it perfectly:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes
MINOR version when you add functionality in a backwards compatible manner
PATCH version when you make backwards compatible bug fixes
Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Certificates

If you are looking for a way to prepare for a certain exam this is the section for you. Here you'll find a list of certificates, each references to a separate file with focused questions that will help you to prepare to the exam. Good luck :)

AWS

• Cloud Practitioner (Latest update: 2020)
• Solutions Architect Associate (Latest update: 2021)
• Cloud SysOps Administration Associate (Latest update: Oct 2022)

Azure

• AZ-900 (Latest update: 2021)

Kubernetes

• Certified Kubernetes Administrator (CKA) (Latest update: 2022)

Additional DevOps and SRE Projects

Credits

Thanks to all of our amazing contributors who make it easy for everyone to learn new things :)

Logos credits can be found here

License