To keep things clear, we create two separate folders with restricted permissions:

• Professor’s folder (~/Desktop/Prof): only the professor can read or write here.
• Student’s folder (~/Desktop/Student): only the student can read or write here.

This setup mimics real-world security: each user protects their private keys in their own secure directory.

1. Create and Protect Workspaces

mkdir ~/Desktop/Prof
chmod 700 ~/Desktop/Prof

mkdir ~/Desktop/Student
chmod 700 ~/Desktop/Student

2. Professor Generates an RSA Key Pair

cd ~/Desktop/Prof

# Generate a 2048-bit private key
openssl genrsa -out PrivateKeyA.pem 2048

# Derive the public key from that private key
openssl rsa -in PrivateKeyA.pem -pubout -out PublicKeyA.pem

# Verify the files exist
ls -l

Example output:

total 12
-rw------- 1 you you 1679 Jun 30 09:00 PrivateKeyA.pem
-rw-r--r-- 1 you you  450 Jun 30 09:00 PublicKeyA.pem

• PrivateKeyA.pem is the secret key (owner-only permissions).
• PublicKeyA.pem is the public key (readable by others).

3. Student Generates Their RSA Key Pair

cd ~/Desktop/Student

openssl genrsa -out PrivateKeyB.pem 2048
openssl rsa -in PrivateKeyB.pem -pubout -out PublicKeyB.pem

ls -l

Example output:

total 12
-rw------- 1 you you 1679 Jun 30 09:05 PrivateKeyB.pem
-rw-r--r-- 1 you you  450 Jun 30 09:05 PublicKeyB.pem

4. Professor Encrypts a Message for the Student

cd ~/Desktop/Prof

# Create a plaintext file
cat > secret.txt
This is a secret message.
^D

# Encrypt it with the student’s public key
openssl rsautl -encrypt \
  -in secret.txt \
  -out secret.enc \
  -inkey ~/Desktop/Student/PublicKeyB.pem \
  -pubin

ls -l

Example output:

total 16
-rw-r--r-- 1 you you   28 Jun 30 09:10 secret.txt
-rw------- 1 you you 1679 Jun 30 09:00 PrivateKeyA.pem
-rw-r--r-- 1 you you  450 Jun 30 09:00 PublicKeyA.pem
-rw-r--r-- 1 you you  256 Jun 30 09:10 secret.enc

To peek at the encrypted file:

cat secret.enc

Example output (binary gibberish):

�����V�j���x�u...�^�

5. Student Decrypts the Message

cd ~/Desktop/Student

# Copy secret.enc from Prof folder (or share it)
# Then decrypt with the student’s private key
openssl rsautl -decrypt \
  -in ~/Desktop/Prof/secret.enc \
  -out secret.dec \
  -inkey PrivateKeyB.pem

ls -l

Example output:

total 20
-rw-r--r-- 1 you you   28 Jun 30 09:05 secret.txt      # if student made one
-rw-r--r-- 1 you you  256 Jun 30 09:10 secret.enc
-rw-r--r-- 1 you you   28 Jun 30 09:10 secret.dec
-rw------- 1 you you 1679 Jun 30 09:05 PrivateKeyB.pem
-rw-r--r-- 1 you you  450 Jun 30 09:05 PublicKeyB.pem

To view the recovered message:

cat secret.dec

Example output:

This is a secret message.

Conclusion

This simple workflow shows the foundation of public-key encryption in data security:

1. Generate a private/public key pair (openssl genrsa + openssl rsa -pubout).
2. Encrypt with the recipient’s public key (openssl rsautl -encrypt).
3. Decrypt with the matching private key (openssl rsautl -decrypt).

Because each private key never leaves its owner’s protected folder, this method secures data even over untrusted networks. Public-key encryption like this is the basic building block for many secure systems, HTTPS websites, secure email, VPNs, and more.

At its core, this example illustrates asymmetric cryptography in its purest form, anyone can encrypt a message using a public key, yet only the holder of the corresponding private key can unlock it. Real-world systems add certificates, use faster symmetric ciphers for large data, and automate key management, but they all still rely on these same three steps.

Try it yourself in a Linux lab, generate your own keys, lock and unlock a file, and you’ll see how public-key cryptography keeps information safe from prying eyes.

In software development environment, the need for efficient artifact management is paramount. As a DevOps Engineer, I recognized the necessity of streamlining the deployment process and ensuring a reliable artifact repository for Java applications. To achieve this, I recently embarked on a journey to set up Artifactory on cloud server. In this blog, I will share the steps I took to accomplish this, detailing the installation and configuration processes along the way.

Purpose

Artifactory management serve as vital components in a DevOps pipeline. They provide a centralized location to store and manage artifacts produced during the software development lifecycle. It is known for its rich feature set, including support for various package types, advanced search capabilities, and user management features.

The Situation:

As a DevOps Engineer, I once faced challenges with a team concerning dependency management and artifact storage, ultimately leading to delays in deployment when locally stored packages were accidentally deleted. The absence of a centralized repository made it difficult to track versions and manage dependencies effectively. To overcome these challenges, I decided to deploy Nexus to manage the artifacts while maven was already used in the environment to build the Java applications.

Let's start simple. First, we will put our Java app on Apache Tomcat, which is a web server. This helps us check if our app works right on a basic website. If it works here, we know it's good to go.

Pre-requisites: Ubuntu Server or any other Linux server

1. Installing Tomcat

Create a Tomcat User

sudo useradd -m -d /opt/tomcat -U -s /bin/false tomcat

Update the Linux Package Manager Cache

sudo apt update

Install Java

sudo apt install openjdk-17-jdk
java -version

Download and Install Tomcat Before downloading, confirm the latest Tomcat build package from the official website.

# Download Tomcat
cd /tmp
wget https://dlcdn.apache.org/tomcat/tomcat-11/v11.0.0-M26/bin/apache-tomcat-11.0.0-M26.tar.gz

# Extract Tomcat files
sudo tar xzvf apache-tomcat-11*tar.gz -C /opt/tomcat --strip-components=1

# Set ownership and permissions
sudo chown -R tomcat:tomcat /opt/tomcat/
sudo chmod -R u+x /opt/tomcat/bin

Configure Users for Tomcat:

• Edit the tomcat-users.xml file to define user roles and access:

sudo nano /opt/tomcat/conf/tomcat-users.xml

Add the following lines before the closing </tomcat-users> tag:

<role rolename="manager-gui" />
<user username="manager" password="xxxxxxx" roles="manager-gui" />

<role rolename="admin-gui" />
<user username="admin" password="xxxxxxx" roles="manager-gui,admin-gui" />

Replace with your chosen password

Allow Remote Access:

• To enable access to the Manager and Host Manager pages, edit the context.xml files:

# Edit Manager context file
sudo nano /opt/tomcat/webapps/manager/META-INF/context.xml

Comment out the Valve definition in this file by adding arrows to the beginning and end of line. Repeat the same for the Host Manager context file:

# Edit Host Manager context file
sudo nano /opt/tomcat/webapps/host-manager/META-INF/context.xml

Setup Systemd for Tomcat:

• Find the Java location:

sudo update-java-alternatives -l

Create the java.sh profile:

sudo nano /etc/profile.d/java.sh
# Add the following lines
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATH

Create the setenv.sh file for Tomcat:

sudo nano /opt/tomcat/bin/setenv.sh
# Add the following lines
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export JRE_HOME=/usr/lib/jvm/java-17-openjdk-amd64

Create the Tomcat Service File (to automatically start on every system reboot)

sudo nano /etc/systemd/system/tomcat.service

Add the following configuration:

[Unit]
Description=Tomcat
After=network.target

[Service]
Type=forking
User=tomcat
Group=tomcat

Environment="JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64"
Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom"
Environment="CATALINA_BASE=/opt/tomcat"
Environment="CATALINA_HOME=/opt/tomcat"
Environment="CATALINA_PID=/opt/tomcat/temp/tomcat.pid"
Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"

ExecStart=/opt/tomcat/bin/startup.sh
ExecStop=/opt/tomcat/bin/shutdown.sh

RestartSec=10
Restart=always

[Install]
WantedBy=multi-user.target

Start Tomcat

sudo systemctl daemon-reload
sudo systemctl start tomcat
sudo systemctl enable tomcat

Allow access to Tomcat on port 8080:

sudo ufw allow 8080

Test Tomcat Access the Tomcat web interface by navigating to http://<your-server-ip>:8080 in your web browser.

2. Downloading and Building Your Java Application with Maven

Install Maven

sudo apt update
sudo apt install maven
mvn -version

Clone A sample Application from GitHub

cd /home/ubuntu
git clone https://github.com/balogsun/maven-java-builds.git

Edit the pom.xml File

cd maven-java-builds

Change the <finalName> tag in your pom.xml to a name of your choice and save:

<finalName>My_Maven_Webapp</finalName>

Lets modify index.jsp file to what we would like to see on our browser when launced

nano /home/ubuntu/maven-java-builds/src/main/webapp/index.jsp

Look for below section and change it to whatwever suits you

<table border="0" cellpadding="0" cellspacing="0" width="100%" style="max-width: 600px;">
    <tr>
        <td bgcolor="#ffffff" align="center" valign="top" style="padding: 40px 20px 20px 20px; border-radius: 4px 4px 0px 0px; color: #11111
            <h1 style="font-size: 28px; font-weight: 400; margin: 2;">My Builds with Maven</h1>
            <img src=" https://img.icons8.com/clouds/100/000000/handshake.png" width="125" height="120" style="display: block; border: 0px;"
            <p style="font-size:15px">Maven Build.</p>
            <p style="font-size:20px">Cool!!!</p>
        </td>

Build Your Application

mvn package

Deploy the WAR File to Tomcat

sudo cp /home/ubuntu/maven-java-builds/target/My_Maven_Webapp.war /opt/tomcat/webapps

Access your application at http://<your-server-ip>:8080/My_Maven_Webapp.

3. Installing Nexus Repository Manager

Download and Install Nexus Repository

cd /opt
sudo wget https://download.sonatype.com/nexus/3/latest-unix.tar.gz
sudo tar -zxvf latest-unix.tar.gz
sudo mv nexus-3.* nexus

Create Nexus User

sudo adduser nexus
sudo echo 'nexus ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers

Change Ownership

sudo chown -R nexus:nexus /opt/nexus
sudo chown -R nexus:nexus /opt/sonatype-work

Configure Nexus to Run as a Service:

• Edit the nexus.rc file:

sudo nano /opt/nexus/bin/nexus.rc
# Add the following line
run_as_user="nexus"

To modify config options, open the /opt/nexus/bin/nexus.vmoptions file, you can modify the config path as shown below

sudo nano /opt/nexus/bin/nexus.vmoptions

In the below settings, the directory is changed from ../sonatype-work to /opt/sonatype-work

-Xms1024m
-Xmx1024m
-XX:MaxDirectMemorySize=1024m

-XX:LogFile=/opt/sonatype-work/nexus3/log/jvm.log
-XX:-OmitStackTraceInFastThrow
-Djava.net.preferIPv4Stack=true
-Dkaraf.home=.
-Dkaraf.base=.
-Dkaraf.etc=etc/karaf
-Djava.util.logging.config.file=/etc/karaf/java.util.logging.properties
-Dkaraf.data=/opt/sonatype-work/nexus3
-Dkaraf.log=/opt/sonatype-work/nexus3/log
-Djava.io.tmpdir=/opt/sonatype-work/nexus3/tmp

Create a Systemd Service for Nexus

sudo nano /etc/systemd/system/nexus.service

Add the following configuration:

[Unit]
Description=nexus service
After=network.target

[Service]
Type=forking
LimitNOFILE=65536
ExecStart=/opt/nexus/bin/nexus start
ExecStop=/opt/nexus/bin/nexus stop
User=nexus
Restart=on-abort

[Install]
WantedBy=multi-user.target

Start Nexus

sudo systemctl start nexus
sudo systemctl enable nexus
sudo systemctl status nexus

Allow access to Nexus on port 8081:

sudo ufw allow 8081/tcp

Access Nexus Web Interface Navigate to http://<your-server-ip>:8081 in your web browser.

Login to Nexus

1. Use the default credentials:

   • Username: admin

   • Password: (this password is generated during the installation; find it in the file /opt/nexus/sonatype-work/nexus3/admin.password).

Change the Admin Password

You will be prompted to change the password upon first login.

4. Creating Repositories in Nexus

Create a Maven Hosted Repository

1. In the Nexus web interface, click on wheel-like icon.

2. Click on repository and Create repository.

3. Select maven2 (hosted) and click Next.

   

4. Configure the repository settings (like name, version policy, etc.), and click Create repository.

5. Configure the repository settings, and click Create repository.

Test a manual upload to artifactory.

First get the url of your repository and copy it out for the next coomand below

cd /home/ubuntu/maven-java-builds/target
curl -v -u admin:pass --upload-file /root/maven-java-builds/target/My_Maven_Webapp.war http://<server-ip-address>:8081/repository/My_Maven_Webapp/

• Check back at your repository in the browse section to find you artifacts uploaded.

• Likewise, artifacts can also be donwloaded manually.

curl -X GET http://admin:pass@<server-ip-address>:8081/repository/My_Maven_Webapp/My_Maven_Webapp.war --output My_Maven_Webapp.war

Integrating Nexus with Maven Builds

Let's configure your Maven project to work with Nexus.

• Navigate to your project directory:

cd /home/ubuntu/maven-java-builds

Configuring pom.xml

Your pom.xml file is the heart of your Maven project. Here's a sample configuration:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>myapp</groupId>
  <artifactId>My_Maven_Webapp</artifactId>
  <packaging>war</packaging>
  <version>1.0.0</version>
  <name>My sample Maven Webapp</name>
  <url>http://maven.apache.org</url>

  <!-- Properties for Java version and encoding -->
  <properties>
    <maven.compiler.release>17</maven.compiler.release>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <!-- Dependencies -->
  <dependencies>
    <dependency>
       <groupId>javax.servlet</groupId>
       <artifactId>javax.servlet-api</artifactId>
       <version>4.0.1</version>
       <scope>provided</scope>
    </dependency>
  </dependencies>

  <!-- Build configuration -->
  <build>
    <finalName>My_Maven_Webapp</finalName>
    <plugins>
      <!-- Compiler plugin -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.10.1</version>
        <configuration>
          <release>17</release>
        </configuration>
      </plugin>
      <!-- WAR plugin -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-war-plugin</artifactId>
        <version>3.3.2</version>
      </plugin>
      <!-- Deploy plugin -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-deploy-plugin</artifactId>
        <version>3.1.0</version>
      </plugin>
    </plugins>
  </build>

  <!-- Distribution Management for Nexus -->
  <distributionManagement>
    <repository>
      <id>nexus</id>
      <name>My_Maven_Webapp</name>
      <url>http://<Ip-address>:8081/repository/My_Maven_Webapp/</url> <!-- Your repository URL here -->
    </repository>
  </distributionManagement>
</project>

Setting Up Nexus

Configure Maven to use Nexus by creating a settings.xml file:

mkdir -p ~/.m2 (this should already exist following initial deploymenets)
sudo nano ~/.m2/settings.xml

Add the following content:

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd">
    <servers>
        <server>
            <id>nexus</id>
            <username>username</username>
            <password>password</password>
        </server>
    </servers>
</settings>

Ensure proper permissions:

chmod 755 ~/.m2/settings.xml
chown username:groupname ~/.m2/settings.xml

Deploying Your Application

To deploy your application to Nexus, run:

mvn clean deploy

This command will compile your code, run tests, package your application, and upload it to Nexus.

Check back at your repository and find the newly uploaded artifacts

Updating Your Application

To update your application:

1. Modify your pom.xml to increment the version number.

2. Update your application code as needed.

3. Run mvn clean deploy again to deploy the new version.

Troubleshooting

If you encounter a 401 Unauthorized error, ensure that:

1. Your settings.xml is in the correct location (~/.m2/settings.xml or /root/.m2/settings.xml if running as root).

2. The credentials in settings.xml match those in your Nexus server.

Java Compatibility

Ensure you're using Java 17 or later to set up your environment:

Conclusion

With these detailed steps, you would have successfully accomplished three key objectives: setting up Tomcat, deploying a Java application, and integrating Nexus Repository into the workflow. This process not only streamlines the development process but will also enhance collaboration within a team. The ability to manage dependencies efficiently and deploy applications reliably will definitely increase one's confidence in delivering high-quality software on time. Furthermore, these manual builds can be integrated into CI/CD pipelines, for full automation. Overall, this implementation significantly improves development lifecycle, positioning a team for more efficient and effective project execution.

Think of APIs (Application Programming Interfaces) as the secret sauce behind seamless interactions between different apps and services. They’re like the unsung heroes that keep everything connected, whether it’s pulling in social media feeds or checking the weather.

🚦 What’s an API Gateway, Anyway?

Imagine an API Gateway as the traffic manager for your data. It’s the hub where all API requests pass through, ensuring they’re handled efficiently. This gateway takes care of important tasks like security, routing requests, and even translating data formats so that everything runs smoothly.

🍽️ Let’s Break Down APIs

Picture this scenario:

• You (the client) make a request to a waiter (the API) for something on the menu.

• The waiter takes your order to the kitchen (the system) and gets your meal (the response) prepared.

• The waiter then delivers the meal back to you.

That’s the essence of an API, facilitating communication between systems and delivering the right information.

📲 What About API Requests?

An API Request is like making an order at your favorite restaurant. When you use an app to check the weather, for instance:

• Your phone sends a request to the weather service via the API.

• This request specifies what you need (like the temperature or forecast) and includes any necessary authentication details.

The service processes your request and sends back the information you asked for in an API response.

🍕 Real-World Example: Ordering Pizza with an App

Think of a food delivery app, like Uber Eats:

1. Multiple APIs at Work:

   • User API for your account info.

   • Restaurant API for listing restaurants.

   • Payment API for processing transactions.

   • Delivery API for tracking your order.

2. How the API Gateway Fits In:

   • Central Hub: Instead of each service interacting directly with the app, all requests go through the API Gateway.

   • Routing: The Gateway directs requests to the appropriate service, like User or Payment APIs.

   • Security: Manages authentication and permissions.

   • Traffic Control: Balances request flow to prevent overload.

   • Data Transformation: Ensures data is in the right format for each service.

   • Response Aggregation: Collects responses from different services and sends a unified update back to the app.

🔍 Why an API Gateway Matters

• Streamlines Requests: Centralizes the process, making it simpler and more efficient.

• Enhances Security: Manages access control and authentication.

• Controls Traffic: Prevents any one service from getting overwhelmed by requests.

• Monitors Performance: Provides insights and tracking for better troubleshooting.

APIs and API Gateways might not always be in the spotlight, but they’re essential for the smooth operation of today’s tech landscape.

We will walk you through integrating Datadog for monitoring, setting up automated recovery scripts, and leveraging Node.js and webhooks to create a reliable self-healing system (with a focus on disk management).

By the end of this guide, you'll have a fully automated setup that can proactively manage system issues, ensuring smooth and uninterrupted operations.

Prerequisites

• A Datadog Account: Create an active Datadog account for monitoring and alerts.

• A Linux Server: This can be either on-premises or a cloud-based instance.

• Internet Access: Required for installing software, setting up integrations, and accessing Datadog.

1. Create a Datadog Account

1. Sign Up for Datadog:

   • Visit Datadog's sign-up page and create a free trial account by entering your email and setting a password.

   • Complete the registration process, create a name for your organization, and verify your email if required.

   • Log in to your Datadog dashboard.

2. Deploy Your First Datadog Agent

Guides to install on your preferred operating system are listed in the integration --> agents section.

For a Local Ubuntu Server:

1. Install the Datadog Agent:

   • Obtain Your Datadog API Key:

     • Navigate to Organization settings > API Keys to find your API key.

• Install the Agent:

  • Run the following command on your Ubuntu server to install the Datadog Agent:

      DD_API_KEY=8axxxxxxxxxxxxxxxxxxxxxxxxxxxf43 DD_SITE="datadoghq.com" bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"
    

• This command sets up the agent and automatically configures it with your API key.

2. Verify Agent Installation:

   • Check the agent status to ensure it is running:

       systemctl status datadog-agent
       sudo datadog-agent status
     

   • You should see output indicating that the agent is running and sending data.

     

   • To check logs, use:

       tail -f /var/log/datadog/agent.log
     

For Ubuntu Server Managing a Kubernetes Cluster with kubectl and helm already installed:

1. Create a Datadog API Key Secret:

   • Execute the following command to create a Kubernetes secret containing your Datadog API key:

       kubectl create secret generic datadog-secret --from-literal api-key=8axxxxxxxxxxxxxxxxxxxxxxxxxxxf43
     

2. Deploy the Datadog Agent in the Cluster:

   • Add the Datadog Helm repository and update:

       curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
       chmod 700 get_helm.sh
       ./get_helm.sh
     
       helm repo add datadog https://helm.datadoghq.com
       helm repo update
     

   • Create and configure datadog-values.yaml:

       nano datadog-values.yaml
     

     Add the following content:

       datadog:
         apiKeyExistingSecret: datadog-secret
     

   • Deploy the Datadog Agent:

       helm install datadog-agent -f datadog-values.yaml datadog/datadog
     

     

   • Confirm agents are running:

       kubectl get all
     

     

3. Prepare a Disk for Monitoring

An alert will be triggered when the disk capacity reaches a determined threshold.

1. Add a New Disk to the Server:

   • In your VMware or AWS cloud instance, add a new 2GB disk.

Adding a 2GB Disk in VMware Workstation

1. Open VMware Workstation:

   • Launch VMware Workstation and select your virtual machine.

2. Open VM Settings:

   • Right-click on the virtual machine and select Settings.

     

3. Add a New Disk:

   • Click Add to open the Add Hardware Wizard.

   • Choose Hard Disk and click Next.

   • Select SCSI (recommended) or IDE and click Next.

   • Choose Create a new virtual disk and click Next.

   • Specify the disk size as 2 GB.

   • Choose the location to store the virtual disk file and click Next.

   • Click Finish to create the disk.

Adding a 2GB Disk in AWS

1. Log in to AWS Management Console:

   • Navigate to AWS Management Console.

   • Log in with your credentials.

2. Navigate to EC2 Dashboard:

   • Go to Services > EC2.

3. Create a New EBS Volume:

   • In the left sidebar, click on Volumes under Elastic Block Store.

   • Click Create Volume.

   • Configure the volume:

     • Volume Type: Choose General Purpose SSD (gp3), Provisioned IOPS SSD (io1), etc.

     • Size: Enter 2 GiB.

     • Availability Zone: Select the same availability zone as your EC2 instance.

   • Click Create Volume.

     

4. Attach the EBS Volume to an EC2 Instance:

   • Go back to Volumes.

   • Select the volume you created.

   • Click Actions > Attach Volume.

   • Choose the instance you want to attach the volume to from the drop-down list.

   • Click Attach.

5. Log in to Your EC2 or VMware Instance.

6. Prepare the Volume for Use:

   • Verify Disk:

       lsblk
     

     The new disk should be listed as /dev/xvdf or similar.

     

7. Install LVM: LVM (Logical Volume Management) is used to manage disk volumes because it offers flexibility, efficiency, and scalability. It allows dynamic resizing of partitions, easy addition and removal of disks, and improved storage utilization through aggregation and thin provisioning. LVM enhances performance with striping, supports snapshots for backups, simplifies administration, and can be combined with mirroring or RAID for high availability, making it an ideal choice for environments with dynamic storage needs.

   • Install LVM tools:

       sudo apt update
       sudo apt install -y lvm2
     

8. Set Up LVM:

   • Create a Physical Volume (PV):

       sudo pvcreate /dev/sdb
     

   • Create a Volume Group (VG):

       sudo vgcreate demoVG /dev/sdb
     

   • Create a Logical Volume (LV) using all available space:

       sudo lvcreate -n demoLV -l 100%FREE demoVG
     

9. Format and Mount the Logical Volume:

   • Format the LV with ext4 filesystem:

       sudo mkfs.ext4 /dev/demoVG/demoLV
     

   • Create a mount point and mount the LV:

       sudo mkdir /demo
       sudo mount /dev/demoVG/demoLV /demo
     

   • Verify the Mount:

       df -h /demo
     

     

10. Configure Automatic Mounting:

    • Add the following entry to /etc/fstab for automatic mounting at boot:

        echo '/dev/demoVG/demoLV /demo ext4 defaults 0 2' | sudo tee -a /etc/fstab
      

    • Verify fstab Configuration:

        cat /etc/fstab
      

4. Set Up the Webhook HTTPS Listener Using Node.js

If you prefer not to use Node.js, you may explore python Flask service, or Datadog’s serverless functions (if using a cloud provider like AWS) to trigger the script directly via AWS Lambda or an equivalent service, but the below method gives direct control on the infrastructure.

**First create the script, that would be triggered by Datadog’s Webhook Integration that clears/move log files in /demo directory**.

Example script (purge_demo.sh):

nano /tmp/purge_demo.sh

#!/bin/bash

LOGFILE="/tmp/purge_demo.log"

echo "Running purge script at $(date)" >> $LOGFILE

# Directory to purge
TARGET_DIR="/demo"

# Check if the directory exists
if [ -d "$TARGET_DIR" ]; then
    echo "Purging all files in $TARGET_DIR..." >> $LOGFILE
    rm -rf ${TARGET_DIR}/* >> $LOGFILE 2>&1
    echo "All files in $TARGET_DIR have been purged." >> $LOGFILE
else
    echo "Directory $TARGET_DIR does not exist." >> $LOGFILE
fi

• Make the Script Executable:

    chmod +x /path/to/purge_demo.sh
  

1. Install Node.js:

   • Install the Node.js package repository and Node.js:

       curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
       sudo apt install -y nodejs
     

   • Verify Installation:

       node -v
       npm -v
     

     

2. Create a Simple Node.js Webhook Listener:

   • Create a directory for the webhook listener and navigate to it:

       mkdir ~/webhook_listener
       cd ~/webhook_listener
     

   • Initialize a new Node.js project:

       npm init -y
     

   • Install Express:

       npm install express
     

   • Create the webhook_listener.js file:

       nano webhook_listener.js
     

     

     • Add the following code to webhook_listener.js:

         const express = require('express');
         const { exec } = require('child_process');
       
         const app = express();
         const port = 6060;
       
         app.post('/purge', (req, res) => {
             // Execute the purge script
             exec('/tmp/purge_demo.sh', (error, stdout, stderr) => {
                 if (error) {
                     console.error(`Error executing script: ${error.message}`);
                     res.status(500).send('Internal Server Error');
                     return;
                 }
                 if (stderr) {
                     console.error(`Script stderr: ${stderr}`);
                 }
                 console.log(`Script output: ${stdout}`);
                 res.send('Purge script executed');
             });
         });
       
         app.listen(port, () => {
             console.log(`Webhook listener running at http://localhost:${port}`);
         });
       

       

   • The service will be actively listening for incoming HTTP POST requests on port 6060. When Datadog triggers the webhook, it will send an HTTP or HTTPS POST request to this specific URL. This request will prompt the execution of the purge script.

3. Make the Listener Persistent with PM2 (the service will continuously run in background):

   • Install PM2:

       sudo npm install -g pm2
     

   • Start the webhook listener with PM2:

       pm2 start webhook_listener.js
     

   • Verify PM2 Process:

       pm2 list
       pm2 stop webhook_listener.js
       pm2 restart webhook_listener.js
     

     

Step 4: Test the Webhook Listener

1. Simulate a Webhook Request:

   You can use curl to simulate a POST request to your webhook:

    curl -X POST http://localhost:6060/purge
   

   

   If everything is set up correctly, the Node.js script should execute /tmp/purge_demo.sh and return a confirmation message.

5. [Optional] Expose the Local Server with a VPN Tunnel, Just in case it is not a Linux cloud instance, it will need a temporary internet access through a vpn tunnel.

1. Install Localtunnel:

   • Install Localtunnel:

       sudo npm install -g localtunnel
     

2. Start a Tunnel:

   • Start a Localtunnel to expose your local webhook listener:

       lt --port 6060 --subdomain trigger-xxxx
     

   • This screen output URL will look like https://trigger-xxxx.loca.lt.

3. Get Tunnel Password:

   • Access the tunnel password (if needed) for first-time access:

   • Open the URL in a browser and you may be requested to enter the tunnel password.

       wget -q -O - https://loca.lt/mytunnelpassword
     

   • The private IP displayed will be your password.

6. Configure Datadog Webhook

1. Create a Webhook in Datadog:

   • Log in to Datadog and navigate to Integrations > Search for Webhooks.

   • Click New Webhook and configure it:

     • Name: Run_Purge_Script

     • URL: https://trigger-xxxxx.loca.lt/purge #for tunnel URL

     • OR

     • URL: https://server_domainIP/purge #for cloud instance

     • Additional Options: Set as needed. [optional]

   • Click Save.

     

     

2. Test that datadog can send a POST test request and Set Up a Datadog Monitor to Trigger the Webhook:

   • On the monitoring page, navigate to synthetic monitoring and testing > New Test.

   • Click New API test, HTTP, URL: POST, https://server_domainIP/purge, > send.

   • You should get a success response as the screenshot below

• Create a Monitor:

  • Navigate to Infrastructure in Datadog page.

  • Hover your mouse on the host and click on view host dashboard

    

  • A graphic display of some metrics you can monitor will be shown here.

  • Click on the metrics you want to monitor (e.g. disk usage by device), click Create Monitor.

• Configure the Monitor:

  • Set the query to trigger an alert when disk usage exceeds 90%:

      max(last_5m):max:system.disk.in_use{device:/dev/mapper/demoVG-demoLV} by {device} > 0.9
    

    

  • Set the alert message:

      Alert: Disk usage on /demo has exceeded 90%. Triggering purge script.
    

  • Add recipients:

      @your_email@domain.com @webhook-Run_Purge_Script
    

    Here, your webhook will also be a recipient, by simply typing the @ key, in the message tab, a list of recipients will pop up for you to select.

    

• Save and Activate the Monitor:

  • Click Save to activate the monitor.

• Navigate to monitor section to see a list of your configured monitors.

  

7. Verify the Self-Healing Process

1. Populate the /demo Directory:

   • Open a separate session to the server.

   • Copy files to /demo directory until it reaches 95% capacity to simulate a full disk:

       dd if=/dev/zero of=/demo/testfile bs=1M count=1900
     

2. Monitor Disk Usage: (open a new session to the server)

   • Run a continuous loop on your server to monitor the /demo directory:

       while true; do date && ls -l && pwd && du -ms; sleep 2; done
     

     • This command shows me the real time date, list of files, size of files and current working directory of /demo partition.

       

   • Ensure that the disk usage reaches 90% and triggers the Datadog monitor.

3. Check Webhook Execution:

   • Verify that the webhook is called and the purge script executes as expected.

     

   • An email will be automatically sent about the filled-up disk, and the /demo partition will be cleaned up.

     

     You will notice that the time that the email was triggered and the time the disk was full, are the same. Within seconds, the script has been executed and any upcoming disruption would have been averted.

   • Check the /demo directory to ensure that files are deleted when the threshold is crossed.

     

   • Another email will be received to inform that the alert has been treated and closed.

     

By following these detailed steps, you'll establish a realistic self-healing system. While the script provided focuses on clearing logs, it can be adapted to perform other actions, such as restarting a service, scaling an instance, or any other task. With Datadog monitoring disk usage and triggering a Node.js webhook, the system will automatically execute the necessary script, ensuring responsive and efficient management of your infrastructure. Please feel free to leave a like, comment, or ask a question if you need clarity on any of the steps. Happy Learning!

If you want to optimize costs in setting up and managing Kubernetes in a cloud environment with integrated CI/CD workflows, this guide provides practical strategies to help you achieve that.

It offers a detailed approach to installing and configuring essential tools for setting up Kubernetes and integrating it with CI/CD on a Windows/Desktop environment. You'll find step-by-step instructions for setting up Chocolatey, Docker, Git, Minikube, kubectl, CircleCI, and ArgoCD.
By following these steps, you’ll establish a robust development workflow that leverages these tools effectively while minimizing cloud costs

1. Prerequisites: Create accounts on the following websites if you don’t already have them:

   • GitHub or GitLab

   • Docker Hub

   • CircleCI

Install Chocolatey for Windows

Chocolatey is a package manager for Windows, designed to make the installation, upgrading, and management of software easier on the Windows platform. It's similar to package managers on other operating systems, such as apt on Debian-based systems or yum on Red Hat-based systems.

1. Search for and run the PowerShell application as an Administrator.

2. Run the command below in the PowerShell terminal:

    Get-ExecutionPolicy
   

   If it returns Restricted, then run:

    Set-ExecutionPolicy AllSigned
   

   or

    Set-ExecutionPolicy Bypass -Scope Process
   

3. Run the following command to install Chocolatey:

    Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
   

   If you don't see any errors, you are ready to use Chocolatey! You can also visit the Chocolatey website for an extensive guide.

Install Docker Desktop

1. Follow the instructions at Docker's official documentation.

Install Git

1. Download and install Git from Git SCM.

Follow the instructions at Kubernetes official documentation.

To install kubectl using Chocolatey:

    choco install kubernetes-cli

Check to ensure the version installed is up-to-date:

    kubectl version --client

Minikube for Windows

Minikube is local Kubernetes, focusing on making it easy to learn and develop for Kubernetes.

All you need is Docker (or similarly compatible) container or a Virtual Machine environment, and Kubernetes is a single command away: minikube start

You may visit Minikube's start guide for setup.

Requirements:

• 2 CPUs or more

• 2-4GB of free memory

• 20GB of free disk space

• Internet connection

• Virtual machine manager (e.g., Docker Desktop, QEMU, Hyperkit, Hyper-V, Podman, VirtualBox, VMware Fusion/Workstation)

Install minikube with chocolatey and start your cluster:

    choco install minikube
    minikube start

If that returns an error related to the virtual environment, try:

    docker context ls
    docker context use desktop-linux
    minikube start --driver=docker --docker-env="desktop-linux"
    OR
    minikube start --driver=hyperv --docker-env="desktop-linux"

Additional Minikube commands:

    minikube version # Display minikube version
    minikube pause # Tthis will not free up resources or stop the cluster, it will only make the service and cluster unavailable or unreachable
    minikube unpause # Resume cluster services
    minikube stop # Shuts down the virtual machine
    minikube delete # Destroys and clean up the VM data from disk.

For more information, visit Minikube's documentation where you can find basic sample deployments you can try your hands on.

Open the GitHub URL Hotel-Booking and fork the repository.

• Follow the instructions to complete cloning the repository to your own github account so that you can work with the copy you have created.

  

Set Up CircleCI

To set up and configure CircleCI for continuous integration, follow these detailed steps:

1. Sign Up and Connect Your Repository

• Go to the CircleCI website and sign up for a free account using GitHub or Bitbucket.

• Connect an organization

• Give any name of your choice as an organization

• In the CircleCI dashboard, select Projects from the sidebar.

• Click Create Project and choose the repository you want to connect.

• What would you like to do, select Build, test, and deploy a software application

  

• Give your project a name.

• Next, setup pipeline

• Name your pipeline

  

• Choose a repo, select either github, gitlab or gitbucket as repo source.

• Authorize CircleCI to access your GitHub or Bitbucket account.

CircleBot, will prepare a custom starter config file to build and test your code.

CircleCI Configuration

• A .circleci/config.yml file will be created automatically if it doesnt already exist. Review it and click submit and run. You may not want to run the one that is suggested for you, you may simply select to run a simlpe hello world option, to proceed, then go to your github repo and change it to a working config below that will build and store your codes as images in your docker hub container repository.

Configure Project Settings

• By default. a trigger is already created for you, you can check if it exists or you create one.

  

  

Environment Variables

• To prevent authentication errors to your docker hub environment, you need to set it in your environmental variables.

• In the CircleCI dashboard, go to Project Settings > Environment Variables.

• Add any necessary environment variables (e.g., DOCKER_USER, DOCKER_PASS for DockerHub authentication).

• Put in you credentials for your dockerhub account user here.

  

I have setup below a Config File that will securely integrate the hotel booking application for CircleCI

Note, I have deliberately instructed the security tools to bypass any vulnerability just for demo purposes only. The pipeline will fail with an exit code if there are any vulnerabilities found in the code or docker images built. For production purpose, remove the string || true in the config file.

    version: 2.1

    executors:
      default:
        docker:
          - image: cimg/node:22.6.0
        working_directory: ~/project

    jobs:
      code_security_scan:
        executor: default
        steps:
          - checkout

          - run:
              name: Install dependencies
              command: npm install

          - run:
              name: Run npm audit (ignore failures)
              command: npm audit --audit-level=high || true

      build_and_scan:
        executor: default
        steps:
          - checkout

          - setup_remote_docker:
              version: default  # Ensure Docker is available

          - run:
              name: Docker login
              command: |
                echo $DOCKER_PASS | docker login -u $DOCKER_USER --password-stdin

          - run:
              name: Build Docker Image
              command: docker build -t <githubuser>/hotel:v0 .

          - run:
              name: Install Trivy
              command: |
                curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /tmp/trivy v0.54.1
                sudo ln -s /tmp/trivy/trivy /usr/local/bin/trivy

          - run:
              name: Scan Docker Image for Vulnerabilities
              command: |
                trivy image --severity HIGH,CRITICAL <githubuser>/hotel:v0 || true

          - run:
              name: Push Docker Image
              command: docker push <githubuser>/hotel:v0

          - run:
              name: Remove Docker Image
              command: docker rmi <githubuser>/hotel:v0

    workflows:
      version: 2
      build_and_deploy:
        jobs:
          - code_security_scan
          - build_and_scan:
              requires:
                - code_security_scan

3. Commit and Push Changes

• Commit the .circleci/config.yml file to your repository: This step is taken when you save the config file.

5. Trigger a Build

• Push a commit to your github repository to trigger the first build. You can make any small modification to the config.yml file and commit it to initiate a trigger. This step will be mentioned again when we deploy argoCD.

• Monitor the build process in the CircleCI dashboard under the Pipelines section.

6. Monitor and Manage

• Use the CircleCI dashboard --> Click on Pipelines to view build logs, test results, and deployment status.

• Failed builds are automatically sent to your email used to register a CircleCI account, sometime in junk/spam folder, you may add it to safe sender list.

  

  

By following these steps, you can set up CircleCI to automate your project's build and test processes, integrating seamlessly with your existing GitHub or Bitbucket repositories.

Lets now Install ArgoCD

1. Create the namespace and install ArgoCD:

    kubectl create namespace argocd
    kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
   

    kubectl -n argocd get all
    kubectl get svc -n argocd
   

2. Change ArgoCD server service type to NodePort:

   • Agrocd-server service is using “ClusterIP”. We can change it to NodePort” to access the agrocd UI from your local browser.

        kubectl edit svc argocd-server -n argocd
        OR
        kubectl edit svc argocd-server -n argocd -o yaml

A notepad will be automatically opened, scroll down and change from ClusterIP to NodePort. Now service will be changed to “NodePort”

3. Note the Minikube Control Plane IP Address and ArgoCD service port that will be used to access the argoCD URL:

    kubectl get node -o wide
    kubectl get svc -n argocd
   

   

   

4. Download and install Argo CD CLI:

   • Visit ArgoCD releases for the latest version. Explore the GitHub repo for newer release if necessary.

   • Run ArgoCD CLI commands from the Windows command prompt, Open windows CMD as administrator, change directory to location where you downloaded the argocd exe file. Do not double click on the exe file and try to run directly from windows, it may be flagged.

       argocd login $ARGOCD_SERVER --username admin --password $ARGO_PWD --<insecure/secure>
     
       #Sample command below:
       argocd-windows-amd64.exe login <MinikubeIpAddress>:ArgoCDServicePortNo> --username admin --password xxxxxx –-insecure
     

     

5. Access ArgoCD UI:

   Visit http://ControlPlaneNodeIP:ArgocdServicePort.

6. Retrieve the initial admin password: After reaching the UI for the first time, you can login with username: admin and the random password generated during the installation. You can find the password by running:

    kubectl get secret -n argocd
    kubectl describe secret argocd-initial-admin-secret -n argocd
    kubectl get secret -n argocd argocd-initial-admin-secret -o yaml
   

   

The password is still encrypted so you have to decrypt it.

Windows may not support native base64 decoding, you can use an online website at https://www.base64decode.org/, insert the values and decode it.

• We can use this password to login. After login it is recommended to change the password.

• Update password in the GUI, User Info Section --> update password --> Save

7. You should delete the initial secret afterwards:

    kubectl get secret -n argocd
    kubectl -n argocd delete secret argocd-initial-admin-secret
   

• In the ArgoCD web interface, click on Settings -> Repositories.

• On the next page, click on Connect Repo

• Fill out as below. Scroll up and click on Connect.

Now we have connected our GitHub repository with ArgoCD. Next is to create an application.

• Click on the Applications page and click on Create New App.

• Fill in the details as below.

• Scroll down the page and fill in the source and destination details. The deployment YAML for our case repo is inside the K8S path; we need to put that as our path.

• Select the cluster URL and namespace. Now click on Create. It will create the app.

  

  

• Upon completion of the creation, argoCD will attempt to automatically deploy the hotel app using the K8S path that we have defined, this is where the deployment.yml file was placed. Upon successful deployment, We will find the hotel app deployed in the minikube cluster.

    kubectl get all
  

  

• We can access the hotel app using the minikube controlplane NodeIP and hotel service port no.

    kubectl get nodes -o wide

From the checks above, URL will be http://172.27.217.12:30537

• Now we can make a change/update to the booking app by modifying a content of the source codes, here i have modified some details in the src\routes\home\home.jsx path. Once the file is saved and committed, CircleCI previously configured will automatically detect this change and run the pipeline in .CircleCI/config.yml, to scan and build a new image, then push it to Docker Hub repository, ready for deployment by argoCD.

  

• Update that image details in your K8S/deployment.yml file in your repo and click on Sync in the argoCD app. This means that if CircleCI builds an image with a tag of v2, then you have to update the image tag in K8S/deployment.yml to v2 also.

The hotel-deployment in you cluster will be automatically updated with the new modification you have made.

• When you click on Sync with argoCD, you have some options tick boxes to select from. If using Argo CD for the first time in a development environment, here are some basic options you might consider selecting:

• 

Options

• Revision: The revision is set to HEAD, which means the latest commit in the default branch will be used.

• DRY RUN: This is a safe option to start with as it allows you to simulate the synchronization process without making any actual changes. It helps you verify what changes would be applied.

Sync Options

• AUTO-CREATE NAMESPACE: Useful if you want Argo CD to automatically create the namespace for your application if it doesn’t exist. This can simplify setup in a development environment.

• APPLY OUT OF SYNC ONLY: This option ensures that only resources that are out of sync with the Git repository are updated, which can be useful for incremental updates.

Additional Considerations

• PRUNE: You might want to use this cautiously. In a development environment, it can be useful to remove resources not defined in your Git repository, but ensure you understand its impact first.

• RETRY: Consider enabling this if you want Argo CD to automatically retry synchronization in case of failure, which can be helpful during development when changes are frequent.

These options provide a balance between safety and functionality, allowing you to manage your applications effectively while minimizing risks.

Prune Propagation Policy

• FOREGROUND: Deletes resources in the foreground, blocking until the resource is fully deleted.

Additional Options

• REPLACE: Replaces resources instead of updating them, which may lead to downtime.

This configuration is used to manage how Argo CD synchronizes application manifests from a Git repository to a Kubernetes cluster. Each option provides control over how resources are applied and managed during the sync process.

In Conclusion

By following this guide, you’ll have set up a budget-friendly CI/CD system running kubernetes on your Windows/Desktop using tools like Chocolatey, Docker, Git, Minikube, kubectl, CircleCI, and ArgoCD. This setup makes your development process smoother and cuts down on cloud costs.

You’ll discover how to set up and use each tool, connect them together, and automate your development tasks. With Minikube, you can run Kubernetes locally, and with CircleCI+ArgoCD, you can integrate/deploy and manage your apps.

Using these tools will help you manage your code, build projects, and deploy apps more easily and effectively, making your development work more reliable and efficient.

In today's fast-paced tech environment, launching a new product often requires setting up a secure and scalable infrastructure, quickly and efficiently. As a Cloud platform or DevOps engineer, your role is critical in ensuring that these servers are not only operational but also secure against potential threats. This guide takes you through a real-world scenario of securing new set of web servers from scratch, covering everything from initial server hardening to advanced automation and monitoring techniques. By the end, you'll gain knowledge of how to have a robust, repeatable process for securing and managing Linux web servers, ready to be scaled across your infrastructure.

Let's walk through creating a repeatable process that can be applied to multiple servers, ensuring they are all configured securely and consistently. I will be hardening the system, setting up file integrity monitoring, system audit, central logging solution and finally automating the process with ansible.

Below is a bash script for basic system hardening of an Ubuntu server. It can definitely be applied to some other linux flavous as needed. The script covers some security measures to ensure the server is suitably protected. After the script, I'll explain each stage in detail.

Bash Script: nano system_hardening.sh

#!/bin/bash

# Ensure the script is run as root
if [ "$(id -u)" -ne 0 ]; then
    echo "This script must be run as root. Exiting."
    exit 1
fi

# Update and upgrade the system
echo "Updating and upgrading the system..."
apt update && apt upgrade -y

# 1. Disable root login via SSH
echo "Disabling root login via SSH..."
sed -i 's/^PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config


# 2. Enable logging of sudo commands
echo "Enabling logging of sudo commands..."
echo "Defaults logfile=/var/log/sudo.log" >> /etc/sudoers

# 3. Install and configure UFW (Uncomplicated Firewall)
echo "Installing and configuring UFW..."
apt install -y ufw
ufw default deny incoming
ufw default allow outgoing
ufw allow OpenSSH
ufw enable

# 4. Disable unused filesystems
echo "Disabling unused filesystems..."
echo "install cramfs /bin/true" >> /etc/modprobe.d/disable-filesystems.conf
echo "install freevxfs /bin/true" >> /etc/modprobe.d/disable-filesystems.conf
echo "install jffs2 /bin/true" >> /etc/modprobe.d/disable-filesystems.conf
echo "install hfs /bin/true" >> /etc/modprobe.d/disable-filesystems.conf
echo "install hfsplus /bin/true" >> /etc/modprobe.d/disable-filesystems.conf
echo "install udf /bin/true" >> /etc/modprobe.d/disable-filesystems.conf

# 5. Set strong password policies
echo "Setting strong password policies..."
apt install -y libpam-pwquality
sed -i 's/^# minlen.*/minlen = 12/' /etc/security/pwquality.conf
sed -i 's/^# dcredit.*/dcredit = -1/' /etc/security/pwquality.conf
sed -i 's/^# ucredit.*/ucredit = -1/' /etc/security/pwquality.conf
sed -i 's/^# lcredit.*/lcredit = -1/' /etc/security/pwquality.conf
sed -i 's/^# ocredit.*/ocredit = -1/' /etc/security/pwquality.conf

# 6. Disable IPv6 if not needed
echo "Disabling IPv6..."
sed -i 's/^#net.ipv6.conf.all.disable_ipv6 = 1/net.ipv6.conf.all.disable_ipv6 = 1/' /etc/sysctl.conf
sysctl -p

# 7. Secure shared memory
echo "Securing shared memory..."
echo "tmpfs /run/shm tmpfs defaults,noexec,nosuid 0 0" >> /etc/fstab

# 8. Remove unnecessary packages
echo "Removing unnecessary packages..."
apt-get autoremove -y --purge

# 9. Install and configure rkhunter
echo "Installing and configuring rkhunter..."
apt install -y rkhunter
rkhunter --update
rkhunter --propupd
rkhunter --check --sk

# 10. Set up log file permissions
echo "Setting up log file permissions..."
chmod -R go-rwx /var/log/*

# 11. Configure login banner
echo "Configuring login banner..."
echo "Authorized access only. All activity may be monitored and reported." > /etc/issue.net
sed -i 's/^#Banner.*/Banner \/etc\/issue.net/' /etc/ssh/sshd_config

# 12. Set up limits on user processes
echo "Setting up limits on user processes..."
echo "* hard nproc 100" >> /etc/security/limits.conf

# 13. Restrict cron jobs to authorized users
echo "Restricting cron jobs to authorized users..."
touch /etc/cron.allow
chmod 600 /etc/cron.allow

# 14. Restrict access to su command
echo "Restricting access to su command..."

# Ensure required package is installed.
apt install -y libpam-modules

# Configure PAM to restrict access to su command.
if ! grep -q "auth required pam_wheel.so use_uid" /etc/pam.d/su; then
    echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su
fi

# Create the 'wheel' group if it does not exist
if ! getent group wheel > /dev/null; then
    echo "Creating 'wheel' group..."
    groupadd wheel
fi

# Add the 'ubuntu' user to the 'wheel' group
echo "Adding 'ubuntu' user to 'wheel' group..."
usermod -aG wheel ubuntu

echo "su command access restricted to 'wheel' group members, including 'ubuntu' user."

# 15. Disable core dumps
echo "Disabling core dumps..."
echo "* hard core 0" >> /etc/security/limits.conf
echo "fs.suid_dumpable = 0" >> /etc/sysctl.conf
sysctl -p

# 16. Enable ASLR (Address Space Layout Randomization)
echo "Enabling ASLR..."
echo "kernel.randomize_va_space = 2" >> /etc/sysctl.conf
sysctl -p

# Restart SSH to apply changes
echo "Restarting SSH service..."
systemctl restart ssh

echo "System hardening completed."

Make the script executable and run it.

chmod 764 system_hardening.sh

./chmod 764 system_hardening.sh

Snippet of the script running

Summary of Hardening Steps

1. Update the server packages.

2. Disable root login via SSH: Prevents direct root login, reducing the risk of brute force attacks on the root account.

3. Enable logging of sudo commands: Tracks all commands executed with sudo, enhancing accountability and auditability.

4. Install and configure UFW (Uncomplicated Firewall): Sets up a simple firewall to manage incoming and outgoing traffic.

5. Disable unused filesystems: Prevents the loading of unnecessary and potentially vulnerable filesystems.

6. Set strong password policies: Enforces strong password requirements to reduce the risk of password-based attacks. These settings enforce a minimum password length of 12 characters with at least one digit, uppercase letter, lowercase letter, and special character.

7. Disable IPv6 if not needed: Disables IPv6 to reduce the attack surface if it's not in use.

8. Secure shared memory: Protects shared memory by mounting it with restrictive options.

9. Remove unnecessary packages: Reduces the attack surface by removing unused packages.

10. Install and configure rkhunter: Checks for rootkits and other security issues on the server.

11. Set up log file permissions: Ensures that log files are protected from unauthorized access.

12. Configure login banner: Displays a warning message to users before login, indicating monitoring and restrictions.

13. Set up limits on user processes: Limits the number of processes a user can run, preventing fork bomb attacks.

14. Restrict cron jobs to authorized users: Only allows authorized users to create and edit cron jobs, reducing the risk of malicious cron tasks.

15. Restrict access to su command: Limits access to the su command to authorized users, preventing unauthorized privilege escalation.

16. Disable core dumps: Prevents the creation of core dumps, which could contain sensitive information.

17. Enable ASLR (Address Space Layout Randomization): Randomizes memory addresses, making it more difficult for attackers to exploit vulnerabilities.

This script now focuses on securing a Linux server by implementing a set of essential hardening measures without adding new tools or services beyond those already included.

This script is designed to cover a wide range of basic security measures that are generally applicable to many environments. For your specific production environment, additional hardening steps might be needed depending on the specific use case.

Now lets test some of the functionalities that the script has implemented

• vagrant user is not a member of wheel and was not able to perform su functions, unlike the ubuntu user.

  

• vagrant user tried to change password that did not meet up with strong password policies and that attempt failed.

  

• display banner is shown whenever a user logs in.

• root user could no longer make a direct ssh login

Next we can install and setup Fail2ban

A security tool designed to protect servers from brute-force attacks, protecting services such as SSH, HTTP, and FTP by monitoring log files for suspicious activity and automatically banning offending IP addresses. It works by defining "jails" for specific services like SSH, HTTP, and FTP, where it monitors for failed login attempts or other malicious behavior. When a threshold of failures is reached, Fail2ban modifies firewall rules to block the IP for a set period. The tool is highly configurable, allowing custom filters and flexible ban durations, and it automatically unbans IPs after the ban period expires.

Below is a script that will install and setup a basic configuration that provides a strong level of security, but you can further tailor it to meet specific needs by creating custom jails and filters.

#!/bin/bash

# Fail2ban installation and configuration script

echo "Updating package list..."
sudo apt update

echo "Installing Fail2ban..."
sudo apt install -y fail2ban

# Create a local copy of the jail configuration file for custom settings if not already present
echo "Creating local configuration file..."
if [ ! -f /etc/fail2ban/jail.local ]; then
    sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
fi

# Backup existing jail.local before modifying
echo "Backing up existing jail.local..."
sudo cp /etc/fail2ban/jail.local /etc/fail2ban/jail.local.bak

# Append new configuration settings to jail.local
echo "Configuring Fail2ban..."
sudo bash -c 'echo -e "[DEFAULT]\n# Whitelist your own IPs\nignoreip = 127.0.0.1/8 ::1 <ipaddress>/24 <ipaddress>/24\n\n# Ban time in seconds (e.g., 10 minutes)\nbantime = 600\n\n# Time frame for counting failures in seconds\nfindtime = 600\n\n# Max retry attempts before banning\nmaxretry = 5\n\n[sshd]\nenabled = true\nport = ssh\nlogpath = %(sshd_log)s\nmaxretry = 5" | sudo tee /etc/fail2ban/jail.local > /dev/null'

# Restart and enable Fail2ban service
echo "Starting and enabling Fail2ban service..."
sudo systemctl restart fail2ban
sudo systemctl enable fail2ban

# Display the status of the Fail2ban service
echo "Checking Fail2ban status..."
sudo systemctl status fail2ban

# Optional: Monitor Fail2ban logs
echo "You can monitor Fail2ban logs with the following command:"
echo "sudo tail -f /var/log/fail2ban.log"

echo "Fail2ban installation and configuration completed."

How to Use the Script

2. Make the Script Executable: Run the following command to make the script executable:

    chmod +x install_fail2ban.sh
   

3. Run the Script: Execute the script with sudo to install and configure Fail2ban:

    sudo ./install_fail2ban.sh
   

What This Script Does

• Installs Fail2ban using the package manager.

• Creates a local configuration file (jail.local) to customize settings without affecting the default jail.conf.

• Configures Fail2ban with some basic settings:

  • Whitelisting local IPs.

  • Setting ban time, find time, and maximum retries.

  • Enabling and configuring the SSH jail.

• Starts and enables the Fail2ban service to run at boot.

• Provides the status of the Fail2ban service for verification.

• Offers a command to monitor Fail2ban logs for real-time information.

Other commands to try

List Banned IPs

sudo fail2ban-client status ssh

This will show the number of currently banned IPs and the list of banned IPs.

If you need to unban an IP address manually:

sudo fail2ban-client set sshd unbanip <IP_ADDRESS>

• Replace <IP_ADDRESS> with the actual IP address you want to unban.

This script automates the entire process of installing and configuring Fail2ban, making it easy to secure your server against brute-force attacks.

Now this same server has an alternate IP, i attempted to make several failed attempts to it and here are the logs showing the IP being banned, and future connections from that IP were no longer permitted.

Next, Lets install and configure a basic web server Apache

To install and configure a web server on Ubuntu, you can use Apache or nginx. I have created a custom html document (a simple pizza booking website) for this purpose and here's a step-by-step guide to install and configure Apache:

Bash Script for Installing and Configuring Apache

nano apache.sh

#!/bin/bash

# Web server installation and configuration script

echo "Installing Apache..."
sudo apt install -y apache2

# Ensure Apache is running and enabled on boot
echo "Starting and enabling Apache service..."
sudo systemctl start apache2
sudo systemctl enable apache2

# Configure firewall to allow HTTP and HTTPS traffic
echo "Configuring firewall..."
sudo ufw allow 'Apache Full'

# Create a simple HTML file for the custom site

echo "Creating a pizza ordering web page..."
echo '<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Order Your Pizza</title>
    <style>
        body {
            background-color: #f4f4f9;
            font-family: Arial, sans-serif;
            margin: 0;
            padding: 0;
            color: #333;
            text-align: center;
        }
        header {
            background-color: #808080; /* Grey */
            color: white;
            padding: 20px;
        }
        h1 {
            margin: 0;
            font-size: 2.5em;
        }
        h2 {
            font-size: 1.5em;
            margin-top: 0.5em;
            color: #555;
        }
        h3 {
            font-size: 1.2em;
            margin-top: 1em;
            color: #333;
        }
        .container {
            max-width: 800px;
            margin: 20px auto;
            padding: 20px;
            background-color: white;
            border-radius: 8px;
            box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);
        }
        .form-group {
            margin-bottom: 15px;
            text-align: left;
        }
        .form-group input {
            width: 100%;
            padding: 10px;
            border: 1px solid #ddd;
            border-radius: 4px;
        }
        .toppings {
            display: flex;
            justify-content: center;
            flex-wrap: wrap;
        }
        .toppings div {
            margin: 10px;
            font-size: 1.1em;
        }
        .button {
            background-color: #4CAF50; /* Green */
            border: none;
            color: white;
            padding: 15px 25px;
            text-align: center;
            text-decoration: none;
            display: inline-block;
            font-size: 16px;
            margin: 10px;
            border-radius: 5px;
            cursor: pointer;
            transition: background-color 0.3s ease;
        }
        .button:hover {
            background-color: #45a049;
        }
        .button.reset {
            background-color: #e7e7e7; /* Gray */
            color: black;
        }
        .button.reset:hover {
            background-color: #d5d5d5;
        }
        img {
            max-width: 100%;
            height: auto;
            margin-top: 20px;
            border-radius: 8px;
        }
        footer {
            margin-top: 20px;
            font-size: 14px;
            color: #777;
        }
    </style>
</head>
<body>
    <header>
        <h1>Your One Stop Shop for Great Pizzas</h1>
    </header>
    <div class="container">
        <h2>Kindly Make Your Order by Filling the Details Below</h2>
        <form>
            <div class="form-group">
                <label for="name">Name:</label>
                <input type="text" id="name" placeholder="Your Name" required>
            </div>
            <div class="form-group">
                <label for="email">Email:</label>
                <input type="email" id="email" placeholder="Your Email" required>
            </div>
            <div class="form-group">
                <label for="phone">Phone No:</label>
                <input type="tel" id="phone" placeholder="Your Phone Number" required>
            </div>
            <h3>Select Pizza Toppings</h3>
            <div class="toppings">
                <div><input type="checkbox" id="chicken"><label for="chicken"> Chicken</label></div>
                <div><input type="checkbox" id="pepperoni"><label for="pepperoni"> Pepperoni</label></div>
                <div><input type="checkbox" id="sausage"><label for="sausage"> Sausage</label></div>
                <div><input type="checkbox" id="mushrooms"><label for="mushrooms"> Mushrooms</label></div>
            </div>
            <button type="submit" class="button">Submit</button>
            <button type="reset" class="button reset">Reset</button>
        </form>
    </div>
    <img src="https://upload.wikimedia.org/wikipedia/commons/thumb/9/91/Pizza-3007395.jpg/1280px-Pizza-3007395.jpg" alt="Pizza">
    <footer>
        <h4>All Rights Reserved</h4>
    </footer>
</body>
</html>' | sudo tee /var/www/html/index.html > /dev/null

# Remove the default site configuration

echo "Removing the default site configuration..."
sudo a2dissite 000-default.conf

# Create and enable custom site configuration

echo "Creating custom site configuration..."
sudo tee /etc/apache2/sites-available/custom-site.conf > /dev/null <<EOF
<VirtualHost *:80>
    DocumentRoot /var/www/html
    <Directory /var/www/html>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
EOF

echo "Enabling the custom site configuration..."
sudo a2ensite custom-site.conf

# Restart Apache to apply any configuration changes

echo "Restarting Apache..."
sudo systemctl restart apache2

# Display Apache status

echo "Checking Apache status..."
sudo systemctl status apache2

echo "Apache installation and configuration completed. You can access your web server at http://<your-server-ip>."

Save the Script: Save the script as install_configure_apache.sh. Make it Executable:

chmod +x apache.sh

Run the Script:

sudo ./apache.sh

A snippet of the script when run

Accessing the Web Server

Once the script completes, you can access the web server by navigating to http://<your-server-ip> in your web browser. You should see the simple HTML page you created.

Also ensure to set up SSL for Apache for secure communication between users and the application. Proper SSL/TLS configuration will encrypt data in transit, protect against man-in-the-middle attacks, and build trust with your users by validating your server’s identity.

Next lets setup File Integrity Monitoring with AIDE (Advanced Intrusion Detection Environment), to help you monitor file integrity effectively and receive daily reports

To implement File Integrity Monitoring with AIDE (Advanced Intrusion Detection Environment), follow these steps: We will also be setting up mail services using gmail.

One Needs to Have 2-Step Verification Enabled for Google mail Account as Less Secure Apps has been deprecated.

1. Navigate to https://myaccount.google.com/apppasswords.

2. Name and Create your app, then click the Create button.

3. A new app password will be generated for you. Copy this password.

1. Install AIDE

# !/bin/bash

# Install AIDE

echo "Installing AIDE..."
sudo apt-get install -y aide

# Backup the existing AIDE configuration file

echo "Backing up AIDE configuration..."
sudo cp /etc/aide/aide.conf /etc/aide/aide.conf.bkup

# Add exclusions to AIDE configuration file

echo "Configuring AIDE exclusions..."
echo "!/tmp" | sudo tee -a /etc/aide/aide.conf > /dev/null
echo "!/var/spool" | sudo tee -a /etc/aide/aide.conf > /dev/null

# Initialize AIDE database

echo "Initializing AIDE database. This may take a while..."
sudo aide -c /etc/aide/aide.conf --init

# Move the new AIDE database to the correct location

echo "Moving AIDE database..."
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

# Run an AIDE check against the initialized database

echo "Running AIDE check. This may take a while..."
sudo aide -c /etc/aide/aide.conf --check

# Install Postfix

echo "Installing Postfix..."
sudo apt-get install -y postfix

# Configure Postfix for Gmail relay

echo "Configuring Postfix..."
sudo sed -i '/^relayhost =/d' /etc/postfix/main.cf
echo "relayhost = [smtp.gmail.com]:587" | sudo tee -a /etc/postfix/main.cf > /dev/null
echo "smtp_use_tls = yes" | sudo tee -a /etc/postfix/main.cf > /dev/null
echo "smtp_sasl_auth_enable = yes" | sudo tee -a /etc/postfix/main.cf > /dev/null
echo "smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd" | sudo tee -a /etc/postfix/main.cf > /dev/null
echo "smtp_sasl_security_options = noanonymous" | sudo tee -a /etc/postfix/main.cf > /dev/null

# Create sasl_passwd file for Postfix

echo "Creating SASL password file..."
sudo touch /etc/postfix/sasl_passwd
echo "[smtp.gmail.com]:587 <user@gmail.com>:gmailpass" | sudo tee -a /etc/postfix/sasl_passwd > /dev/null

# Secure the SASL password file and update Postfix lookup table

echo "Securing SASL password file..."
sudo chmod 600 /etc/postfix/sasl_passwd
sudo postmap /etc/postfix/sasl_passwd

# Allow ports 25 and 587 on firewall

echo "Allowing ports 25 and 587 on firewall..."
sudo ufw allow out 25
sudo ufw allow out 587

# Restart Postfix to apply changes

echo "Restarting Postfix..."
sudo systemctl restart postfix

# Test the Postfix configuration

echo "Testing Postfix configuration by sending a test email..."
echo "If you get this mail, it means the mail application is working" | mail -s "Postfix mail from <ServerA@domain.com>" <user@gmail.com>

# Print mail logs

echo "Printing mail logs..."
tail /var/log/mail.log

# Set up a cron job for daily AIDE checks and email reports

echo "Setting up daily AIDE checks cron job..."
(crontab -l 2>/dev/null; echo "0 2 ** * /usr/bin/aide -c /etc/aide/aide.conf --check | mail -s 'AIDE Daily Report' <user@gmail.com>") | crontab -

echo "Setup complete. Check your email for the test message and AIDE reports."

Screenshot showing Email sent successfully from the analysis of AIDE Daily Report

Steps for Conducting a Security Audit with Lynis

1. Install Lynis

   • Open your terminal and update the package list:

       sudo apt-get update
     

   • Install Lynis:

       sudo apt-get install -y lynis
     

2. Run a System Audit

   • Execute the Lynis system audit: The audit process will take some time and will generate output directly in the terminal. To save the report to a file for easier review, you can redirect the output:

       sudo lynis audit system > /var/log/lynis-audit.log
     

3. Review the Lynis Report

   • Open and review the Lynis report, from the saved log file:

       sudo tail /var/log/lynis-audit.log
     

     

     

     

Next, we will be reviewing the Lynis report, and address at least 5 medium or high-risk findings, and then document the changes made to fix the findings with justifications.

1. Weak File Permissions on Critical Directories

• Finding: Directories such as /etc/cron.d, /etc/cron.daily, /etc/cron.hourly, /etc/cron.weekly, and /etc/cron.monthly have permissive permissions.

• Impact: Loose file permissions on these directories could allow unauthorized users to modify scheduled tasks, potentially leading to privilege escalation or system compromise.

• Action: Tighten permissions on these directories.

• Steps:

  1. Check Current Permissions:

      ls -ld /etc/cron.*
     

     • Review the permissions to ensure they are not overly permissive.

  2. Set Secure Permissions:

      sudo chmod 600 /etc/cron.d
      sudo chmod 600 /etc/cron.daily
      sudo chmod 600 /etc/cron.hourly
      sudo chmod 600 /etc/cron.weekly
      sudo chmod 600 /etc/cron.monthly
     

sudo chmod 600 /etc/cron.yearly 3. **Verify the Change**: bash ls -ld /etc/cron.* ``` - Ensure the permissions are set to 600 (i.e., -rw-------), meaning only the root user can read and write.

• Justification: Restricting access to cron directories prevents unauthorized users from tampering with scheduled tasks, which is critical for maintaining system integrity.

2. Weak File Permissions on Sensitive Files

• Finding: Some files, such as /etc/ssh/sshd_config, and directories like /etc/cron.d, have weak permissions.

• Impact: Inadequate file permissions can allow unauthorized users to modify critical configuration files or execute malicious cron jobs.

• Action: Adjust file and directory permissions to ensure they are secure.

• Steps:

  1. Check Current Permissions:

      ls -l /etc/ssh/sshd_config
     

     • Review the permissions to ensure they are not overly permissive.

  2. Set Secure Permissions:

      sudo chmod 600 /etc/ssh/sshd_config
     

  3. Verify the Change:

      ls -l /etc/ssh/sshd_config
     

     • Ensure the permissions are set to 600 (i.e., -rw-------), meaning only the root user can read and write.

• Justification: Proper permissions help protect sensitive files and configurations from unauthorized access and modification.

3. Missing Password Aging Configurations

• Finding: User password aging settings are disabled (/etc/login.defs).

• Impact: Lack of password aging increases the risk of password-related vulnerabilities as passwords may not be changed regularly.

• Action: Enable and configure password aging settings in /etc/login.defs to enforce regular password changes.

• Steps:

  1. Edit the Login Definitions File:

      sudo nano /etc/login.defs
     

  2. Configure Password Aging Settings:

     • Set the minimum number of days between password changes:

         PASS_MIN_DAYS 7
       

     • Set the maximum number of days a password is valid:

         PASS_MAX_DAYS 90
       

     • Set the number of days before password expiration that the user is warned:

         PASS_WARN_AGE 10
       

  3. Apply the Changes:

     • Apply these settings to all user accounts:

         sudo chage --mindays 7 --maxdays 90 --warndays 10 ubuntu
       

  4. Verify the Settings:

      sudo chage -l ubuntu
     

     • Ensure the correct password aging policies are applied.

• Justification: Enforcing password aging helps ensure that passwords are updated regularly, reducing the risk of compromised accounts.

4. Lack of Sticky Bit on /tmp Directory

• Finding: The sticky bit is not set on the /tmp directory.

• Impact: Without the sticky bit, users can delete or modify files in /tmp that are owned by other users, which could lead to privilege escalation or data tampering.

• Action: Set the sticky bit on the /tmp directory.

• Steps:

  1. Check Current Permissions:

      ls -ld /tmp
     

     • This command will show the current permissions of the /tmp directory. Look for a t at the end of the permission string (e.g., drwxrwxrwt), which indicates the sticky bit is set.

  2. Set the Sticky Bit:

      sudo chmod +t /tmp
     

  3. Verify the Change:

      ls -ld /tmp
     

     • Ensure the sticky bit is now set (drwxrwxrwt).

• Justification: Applying the sticky bit restricts file deletions in /tmp to the file's owner, thereby improving security and preventing unauthorized file manipulations.

5. Unrestricted NTP Service Access

• Finding: NTP (Network Time Protocol) service is running without any access control.

• Impact: An unrestricted NTP service can be abused for DDoS amplification attacks or for manipulating the system clock.

• Action: Configure the NTP service to restrict access.

• Steps:

  1. Edit the NTP Configuration File:

      sudo nano /etc/ntp.conf
     

  2. Add Access Control Restrictions:

     • Add the following line to restrict default access:

         restrict default kod nomodify notrap nopeer noquery
       

  3. Restart the NTP Service:

      sudo systemctl restart ntp
     

  4. Verify the Configuration:

      ntpq -p
     

     • This command will show the status of the NTP peers and the current configuration.

• Justification: Restricting NTP access helps prevent abuse of the service and ensures the system clock's integrity.

These steps will help address the findings identified in the security audit, ensuring the system is better protected against various threats and vulnerabilities.

Understanding CIS Benchmarks

CIS (Center for Internet Security) Benchmarks are consensus-based, best-practice security configuration guides. They are designed to help organizations secure their IT systems and data against cyber threats. These benchmarks are widely recognized as industry standards for hardening systems and improving their security posture.

Using Tools Like oscap (OpenSCAP) or InSpec to Apply and Check CIS Benchmarks

To run InSpec against a Linux Ubuntu host, follow these steps to install InSpec, run a pre-built CIS benchmark profile, and generate a report.

1. Install InSpec on the Ubuntu Host

First, you'll need to install InSpec on your Ubuntu system.

A. Install via Omnitruck Script (Recommended)

This is the easiest method to install InSpec.

curl https://omnitruck.chef.io/install.sh | sudo bash -s -- -P inspec

B. Install via APT Repository (Alternative Method)

1. Add the Chef APT repository:

    echo "deb [arch=amd64] https://packages.chef.io/repos/apt/stable $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/chef-stable.list
   

2. Add the GPG key:

    curl https://packages.chef.io/chef.asc | sudo apt-key add -
   

3. Update the package list and install InSpec:

    sudo apt-get update
    sudo apt-get install inspec
   

2. Obtain a CIS Benchmark Profile

You can use a pre-built CIS benchmark profile. One such profile is provided by the DevSec Project.

A. Download the CIS Benchmark Profile

For example, to download and execute the CIS benchmark for Linux:

inspec supermarket exec dev-sec/linux-baseline

This command downloads the profile and executes it on your local machine.

3. Run the InSpec Profile Against the Ubuntu Host

Once you have the InSpec profile, you can run it on the host machine.

A. Run the Profile Locally on the Ubuntu Host

inspec exec https://github.com/dev-sec/linux-baseline
git clone https://github.com/dev-sec/linux-baseline.git
cd linux-baseline
inspec exec .

This command will execute the CIS benchmark checks against your Ubuntu system.

B. Run the Profile and Generate a Report

You can generate a report in various formats, such as JSON or HTML, for further analysis.

• Generate an HTML Report:

    inspec exec https://github.com/dev-sec/linux-baseline --reporter html:report.html
    inspec exec . --reporter html:report.html
  

• Generate a JSON Report:

    inspec exec dev-sec/linux-baseline --reporter json:report.json
  

4. Review the Report

• If you generated an HTML report, you can ship out the report to your local machine and then open it in a web browser:

• If you generated a JSON report, you can parse it or use it as input for other tools or automation processes.

  

  

5. (Optional) Schedule Regular Scans

You can automate InSpec scans by scheduling them with cron.

A. Edit the Crontab

crontab -e

B. Add a Cron Job for Regular Scans

For example, to run the scan every day at midnight and generate an HTML report:

0 0 * * * /usr/bin/inspec exec /path/to/linux-baseline --reporter html:/path/to/report-$(date +\%Y-\%m-\%d).html

This cron job will create a new report file each day, named with the current date.

Steps to Apply and Check CIS Benchmarks Using OpenSCAP (Alternative to Inspec)

1. Install OpenSCAP: sudo apt-get install openscap-scanner openscap-utils

Check OpenSCAP version

oscap -V

Install SCAP security guide

sudo apt install ssg-debderived ssg-base

The SSG policies we just installed are located in the /usr/share/xml/scap/ssg/content directory.

Download the latest Scap Security Guide from

sudo wget https://github.com/ComplianceAsCode/content/releases/download/v0.1.74/scap-security-guide-0.1.74.zip

Install unzip if not already pre-installed

sudo apt install unzip

Unzip Scap Security Guide

sudo unzip scap-security-guide-0.1.74.zip
cd scap-security-guide-0.1.74

You should now see the SSG security policies for various linux flavors, but we will work with ssg-ubuntu2204-xccdf.xml in our scenario.

Display a list of available Profiles in the selected security policy

oscap info /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-xccdf.xml

Lets work with xccdf_org.ssgproject.content_profile_cis_level1_server which is tailored for CIS benchmarks Level 1 server hardening.

Run the evaluation scan that will output the report in html format

sudo oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_cis_level1_server --report report.html /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-xccdf.xml

After the evaluation has completed, you will see a long list of rules and whether your system passed or failed those rules. Copy the result.html file to your local machine and open with web browser.

By following these steps, you'll be able to use InSpec to run CIS benchmark checks against your Ubuntu system, generate reports, and even automate regular compliance checks. By integrating CIS Benchmarks and tools like OpenSCAP into your security practices, you can significantly enhance your system's security posture, ensure compliance with industry standards, and automate the process of securing your IT infrastructure.

Next, let us put evrything we have previously done into automation. On your chosen primary server.

Here is a bash script containing the steps to install and configure Ansible on an Ubuntu system. I have 1 primary server and 2 other hosts, and ansible will be configured with ssh keys to authenticate to the host servers. Run this script on only the primary server where you want ansible to be installed.

sudo nano install_ansible.sh

#!/bin/bash

# Install Ansible
echo "Installing Ansible..."
sudo apt install ansible -y

# Verify Ansible installation
echo "Verifying Ansible installation..."
ansible --version

# Create and configure the inventory file
echo "Configuring Ansible inventory file..."
sudo mkdir /etc/ansible/
sudo touch /etc/ansible/hosts
sudo tee /etc/ansible/hosts > /dev/null <<EOL
[ServerA]
192.168.x.x

[ServerB]
192.168.x.x

[ServerC]
192.168.x.x
EOL

# Edit the Ansible configuration file
echo "Editing Ansible configuration file..."
sudo tee /etc/ansible/ansible.cfg > /dev/null <<EOL
[defaults]
inventory = /etc/ansible/hosts
remote_user = ubuntu

[privilege_escalation]
become = True
become_method = sudo
become_user = root
ask_sudo_pass = False
become_ask_pass = True
EOL

# Test Ansible configuration
#echo "Testing Ansible configuration..."
#ansible all -m ping
#
echo "Ansible installation and configuration complete!"

1. Save the Script: name it install_ansible.sh.

2. Make the Script Executable:

    chmod +x install_ansible.sh
   

3. Run the Script:

    ./install_ansible.sh
   

Then we run the ansible script to configure ansible hosts and authentication keys

# !/bin/bash
# Define variables

SSH_KEY_PATH="$HOME/.ssh/id_rsa"
ANSIBLE_HOSTS="/etc/ansible/hosts"
REMOTE_USER="ubuntu"

# Generate SSH key pair if it doesn't exist

if [ ! -f "$SSH_KEY_PATH" ]; then
  echo "Generating SSH key pair..."
  ssh-keygen -t rsa -b 4096 -f "$SSH_KEY_PATH" -N "" -C "$REMOTE_USER"
else
  echo "SSH key pair already exists at $SSH_KEY_PATH."
fi

# Extract host IPs from Ansible inventory file

HOSTS=$(grep -E '^\[Server[A-C]\]' "$ANSIBLE_HOSTS" -A 10 | grep -v '^\[' | awk '{print $1}')

# Copy the SSH key to the target hosts

for HOST in $HOSTS; do
  echo "Copying SSH key to $REMOTE_USER@$HOST..."

  ssh-copy-id -i "$SSH_KEY_PATH.pub" "$REMOTE_USER@$HOST"

  if [ $? -eq 0 ]; then
    echo "Successfully copied SSH key to $HOST."
  else
    echo "Failed to copy SSH key to $HOST."
  fi
done

# Test SSH access

for HOST in $HOSTS; do
  echo "Testing SSH access to $REMOTE_USER@$HOST..."

  ssh -o PasswordAuthentication=no "$REMOTE_USER@$HOST" "echo 'SSH to $HOST successful.'"

  if [ $? -eq 0 ]; then
    echo "SSH access to $HOST confirmed."
  else
    echo "SSH access to $HOST failed."
  fi
done

echo "SSH key setup complete."

Below hardening yaml script includes both package installation and configuration, you may choose to separate installation and configuration commands into separate tasks, or separate yaml files

run hardening_script.yml to harden the rest of the ansible hosts

---
- name: Run System Hardening Script
  hosts: all
  become: true
  tasks:

    - name: Ensure the hardening script is present on the target host
      copy:
        dest: /tmp/system_hardening.sh
        content: |
          #!/bin/bash
          echo "Updating the system..."
          apt update
          echo "Disabling root login via SSH..."
          sed -i 's/^PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
          echo "Enabling logging of sudo commands..."
          echo "Defaults logfile=/var/log/sudo.log" >> /etc/sudoers
          echo "Installing and configuring UFW..."
          apt install -y ufw
          ufw default deny incoming
          ufw default allow outgoing
          ufw allow OpenSSH
          ufw enable
          echo "Disabling unused filesystems..."
          echo "install cramfs /bin/true" >> /etc/modprobe.d/disable-filesystems.conf
          echo "install freevxfs /bin/true" >> /etc/modprobe.d/disable-filesystems.conf
          echo "install jffs2 /bin/true" >> /etc/modprobe.d/disable-filesystems.conf
          echo "install hfs /bin/true" >> /etc/modprobe.d/disable-filesystems.conf
          echo "install hfsplus /bin/true" >> /etc/modprobe.d/disable-filesystems.conf
          echo "install udf /bin/true" >> /etc/modprobe.d/disable-filesystems.conf
          echo "Setting strong password policies..."
          apt install -y libpam-pwquality
          sed -i 's/^# minlen.*/minlen = 12/' /etc/security/pwquality.conf
          sed -i 's/^# dcredit.*/dcredit = -1/' /etc/security/pwquality.conf
          sed -i 's/^# ucredit.*/ucredit = -1/' /etc/security/pwquality.conf
          sed -i 's/^# lcredit.*/lcredit = -1/' /etc/security/pwquality.conf
          sed -i 's/^# ocredit.*/ocredit = -1/' /etc/security/pwquality.conf
          echo "Disabling IPv6..."
          sed -i 's/^#net.ipv6.conf.all.disable_ipv6 = 1/net.ipv6.conf.all.disable_ipv6 = 1/' /etc/sysctl.conf
          sysctl -p
          echo "Securing shared memory..."
          echo "tmpfs /run/shm tmpfs defaults,noexec,nosuid 0 0" >> /etc/fstab
          echo "Removing unnecessary packages..."
          apt-get autoremove -y --purge
          echo "Installing and configuring rkhunter..."
          apt install -y rkhunter
          rkhunter --update
          rkhunter --propupd
          rkhunter --check --sk
          echo "Setting up log file permissions..."
          chmod -R go-rwx /var/log/*
          echo "Configuring login banner..."
          echo "Authorized access only. All activity may be monitored and reported." > /etc/issue.net
          sed -i 's/^#Banner.*/Banner \/etc\/issue.net/' /etc/ssh/sshd_config
          echo "Setting up limits on user processes..."
          echo "* hard nproc 100" >> /etc/security/limits.conf
          echo "Restricting cron jobs to authorized users..."
          touch /etc/cron.allow
          chmod 600 /etc/cron.allow
          echo "Restricting access to su command..."
          apt install -y libpam-modules
          if ! grep -q "auth required pam_wheel.so use_uid" /etc/pam.d/su; then
              echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su
          fi
          if ! getent group wheel > /dev/null; then
              echo "Creating 'wheel' group..."
              groupadd wheel
          fi
          echo "Adding 'ubuntu' user to 'wheel' group..."
          usermod -aG wheel ubuntu
          echo "Disabling core dumps..."
          echo "* hard core 0" >> /etc/security/limits.conf
          echo "fs.suid_dumpable = 0" >> /etc/sysctl.conf
          sysctl -p
          echo "Enabling ASLR..."
          echo "kernel.randomize_va_space = 2" >> /etc/sysctl.conf
          sysctl -p
          echo "Restarting SSH service..."
          systemctl restart ssh
          echo "System hardening completed."
        mode: '0755'

    - name: Run the hardening script
      command: /tmp/system_hardening.sh

Step 2: Deploy and Execute the Playbook

1. Save the Playbook:

   Save the playbook as run_hardening_script.yml in your Ansible project directory.

2. Run the Playbook:

   Execute the playbook using the ansible-playbook command:

    ansible-playbook run_hardening_script.yml -vv
   

To create an Ansible playbook that will install and configure AIDE and Postfix

nano install_aide_postfix.yml

---
- name: Install and Configure AIDE and Postfix
  hosts: all
  become: true
  tasks:

    - name: Ensure AIDE and Postfix installation and configuration script is present on the target host
      copy:
        dest: /tmp/install_aide_postfix.sh
        content: |
          #!/bin/bash

          # Install AIDE
          echo "Installing AIDE..."
          apt-get install -y aide

          # Backup the existing AIDE configuration file
          echo "Backing up AIDE configuration..."
          cp /etc/aide/aide.conf /etc/aide/aide.conf.bkup

          # Add exclusions to AIDE configuration file
          echo "Configuring AIDE exclusions..."
          echo "!/tmp" | tee -a /etc/aide/aide.conf > /dev/null
          echo "!/var/spool" | tee -a /etc/aide/aide.conf > /dev/null

          # Initialize AIDE database
          echo "Initializing AIDE database. This may take a while..."
          aide -c /etc/aide/aide.conf --init

          # Move the new AIDE database to the correct location
          echo "Moving AIDE database..."
          mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

          # Run an AIDE check against the initialized database
          echo "Running AIDE check. This may take a while..."
          aide -c /etc/aide/aide.conf --check

          # Install Postfix
          echo "Installing Postfix..."
          apt-get install -y postfix

          # Configure Postfix for Gmail relay
          echo "Configuring Postfix..."
          sed -i '/^relayhost =/d' /etc/postfix/main.cf
          echo "relayhost = [smtp.gmail.com]:587" | tee -a /etc/postfix/main.cf > /dev/null
          echo "smtp_use_tls = yes" | tee -a /etc/postfix/main.cf > /dev/null
          echo "smtp_sasl_auth_enable = yes" | tee -a /etc/postfix/main.cf > /dev/null
          echo "smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd" | tee -a /etc/postfix/main.cf > /dev/null
          echo "smtp_sasl_security_options = noanonymous" | tee -a /etc/postfix/main.cf > /dev/null

          # Create sasl_passwd file for Postfix
          echo "Creating SASL password file..."
          touch /etc/postfix/sasl_passwd
          echo "[smtp.gmail.com]:587 user@gmail.com:gmailpass" | tee -a /etc/postfix/sasl_passwd > /dev/null

          # Secure the SASL password file and update Postfix lookup table
          echo "Securing SASL password file..."
          chmod 600 /etc/postfix/sasl_passwd
          postmap /etc/postfix/sasl_passwd

          # Allow ports 25 and 587 on firewall
          echo "Allowing ports 25 and 587 on firewall..."
          ufw allow out 25
          ufw allow out 587

          # Restart Postfix to apply changes
          echo "Restarting Postfix..."
          systemctl restart postfix

          # Test the Postfix configuration
          echo "Testing Postfix configuration by sending a test email..."
          echo "If you get this mail, it means the mail application is working" | mail -s "Postfix mail from ServerA@domain.com" user@gmail.com

          # Print mail logs
          echo "Printing mail logs..."
          tail /var/log/mail.log

          # Set up a cron job for daily AIDE checks and email reports
          echo "Setting up daily AIDE checks cron job..."
          (crontab -l 2>/dev/null; echo "0 2 * * * /usr/bin/aide -c /etc/aide/aide.conf --check | mail -s 'AIDE Daily Report' user@gmail.com") | crontab -

          echo "Setup complete. Check your email for the test message and AIDE reports."
      mode: '0755'

    - name: Run the AIDE and Postfix installation script
      command: /tmp/install_aide_postfix.sh

Insert your Gmail credentials in the yml above, save and run the Playbook::

ansible-playbook install_aide_postfix.yml

Below is an Ansible playbook that will apply specific security configurations based on CIS Benchmarks

nano secure_ubuntu.yml

---

- name: Apply Security Configurations for Ubuntu Server
  hosts: all
  become: yes

  vars:
    sensitive_files:
      - /etc/passwd
      - /etc/shadow
      - /etc/ssh/sshd_config
      - /etc/gshadow
    unnecessary_services:
      - cups
      - avahi-daemon
      - nfs-server
      - rpcbind
      - vsftpd
    ntp_servers:
      - time.google.com
      - time.cloudflare.com
    secure_delete_tool: "secure-delete"
    files_to_secure_delete:
      - /tmp/abc.log  # Add the path to the file you want to securely delete

  tasks:

  - name: Restrict permissions on sensitive files
      file:
        path: "{{ item }}"
        owner: root
        group: root
        mode: 0600
      loop: "{{ sensitive_files }}"
      tags: restrict_permissions

  - name: Disable IPv6 (if not needed)
      sysctl:
        name: net.ipv6.conf.all.disable_ipv6
        value: '1'
        sysctl_set: yes
        state: present
        reload: yes
      tags: disable_ipv6

  - name: Disable uncommon network protocols
      lineinfile:
        path: /etc/modprobe.d/blacklist.conf
        line: "blacklist {{ item }}"
        create: yes
      loop:
    - dccp
    - sctp
    - rds
    - tipc
      tags: disable_protocols

  - name: Stop and disable unnecessary services
      service:
        name: "{{ item }}"
        enabled: no
        state: stopped
      loop: "{{ unnecessary_services }}"
      tags: disable_services

  - name: Install and configure chrony for NTP synchronization
      apt:
        name: chrony
        state: present
      tags: configure_ntp

  - name: Configure chrony with NTP servers
      lineinfile:
        path: /etc/chrony/chrony.conf
        line: "server {{ item }} iburst"
        create: yes
        state: present
      loop: "{{ ntp_servers }}"
      notify:
    - restart chrony
      tags: configure_ntp

  - name: Install secure deletion tool
      apt:
        name: "{{ secure_delete_tool }}"
        state: present
      tags: secure_delete

  - name: Securely delete files
      command: "srm -v {{ item }}"
      loop: "{{ files_to_secure_delete }}"
      when: files_to_secure_delete | length > 0
      tags: secure_delete

  handlers:
  - name: restart chrony
      service:
        name: chrony
        state: restarted

1. Playbook Structure:

   • The playbook is organized into tasks, each of which performs a specific security function.

   • vars are used to define variables for sensitive files, unnecessary services, NTP servers, and files to be securely deleted.

2. Save and run the Playbook:

    ansible-playbook secure_ubuntu.yml
   

3. Customize:

   • Modify the sensitive_files, unnecessary_services, and ntp_servers lists according to your environment's requirements.

   • Define the files_to_secure_delete list with the paths of files you wish to securely delete.

Scaling and Monitoring

Let's set up an Ansible playbook to deploy secure web servers on 2 additional identical servers

Playbook to Installs and configures my custom (pizza booking) web server

nano apache.yml

---
- name: Install and Configure Apache Web Server
  hosts: all
  become: yes

  tasks:
    - name: Install Apache
      apt:
        name: apache2
        state: present
      tags: install_apache

    - name: Ensure required Apache modules are enabled
      apache2_module:
        name: "{{ item }}"
        state: present
      loop:
        - ssl
        - rewrite
        - headers
        - expires
      tags: apache_modules

    - name: Configure firewall to allow HTTP and HTTPS traffic
      ufw:
        rule: allow
        name: 'Apache Full'
      tags: configure_firewall

    - name: Remove the default site configuration
      command: a2dissite 000-default.conf
      notify:
        - reload apache
      tags: remove_default_site

    - name: Create a custom HTML file for the site
      copy:
        content: |
          <!DOCTYPE html>
          <html lang="en">
          <head>
              <meta charset="UTF-8">
              <meta http-equiv="X-UA-Compatible" content="IE=edge">
              <meta name="viewport" content="width=device-width, initial-scale=1.0">
              <title>Order Your Pizza</title>
              <style>
                  body { background-color: #f4f4f9; font-family: Arial, sans-serif; margin: 0; padding: 0; color: #333; text-align: center; }
                  header { background-color: #808080; color: white; padding: 20px; }
                  h1 { margin: 0; font-size: 2.5em; }
                  h2 { font-size: 1.5em; margin-top: 0.5em; color: #555; }
                  h3 { font-size: 1.2em; margin-top: 1em; color: #333; }
                  .container { max-width: 800px; margin: 20px auto; padding: 20px; background-color: white; border-radius: 8px; box-shadow: 0 0 10px rgba(0, 0, 0, 0.1); }
                  .form-group { margin-bottom: 15px; text-align: left; }
                  .form-group input { width: 100%; padding: 10px; border: 1px solid #ddd; border-radius: 4px; }
                  .toppings { display: flex; justify-content: center; flex-wrap: wrap; }
                  .toppings div { margin: 10px; font-size: 1.1em; }
                  .button { background-color: #4CAF50; border: none; color: white; padding: 15px 25px; text-align: center; text-decoration: none; display: inline-block; font-size: 16px; margin: 10px; border-radius: 5px; cursor: pointer; transition: background-color 0.3s ease; }
                  .button:hover { background-color: #45a049; }
                  .button.reset { background-color: #e7e7e7; color: black; }
                  .button.reset:hover { background-color: #d5d5d5; }
                  img { max-width: 100%; height: auto; margin-top: 20px; border-radius: 8px; }
                  footer { margin-top: 20px; font-size: 14px; color: #777; }
              </style>
          </head>
          <body>
              <header>
                  <h1>Your One Stop Shop for Great Pizzas</h1>
              </header>
              <div class="container">
                  <h2>Kindly Make Your Order by Filling the Details Below</h2>
                  <form>
                      <div class="form-group">
                          <label for="name">Name:</label>
                          <input type="text" id="name" placeholder="Your Name" required>
                      </div>
                      <div class="form-group">
                          <label for="email">Email:</label>
                          <input type="email" id="email" placeholder="Your Email" required>
                      </div>
                      <div class="form-group">
                          <label for="phone">Phone No:</label>
                          <input type="tel" id="phone" placeholder="Your Phone Number" required>
                      </div>
                      <h3>Select Pizza Toppings</h3>
                      <div class="toppings">
                          <div><input type="checkbox" id="chicken"><label for="chicken"> Chicken</label></div>
                          <div><input type="checkbox" id="pepperoni"><label for="pepperoni"> Pepperoni</label></div>
                          <div><input type="checkbox" id="sausage"><label for="sausage"> Sausage</label></div>
                          <div><input type="checkbox" id="mushrooms"><label for="mushrooms"> Mushrooms</label></div>
                      </div>
                      <button type="submit" class="button">Submit</button>
                      <button type="reset" class="button reset">Reset</button>
                  </form>
              </div>
              <img src="https://upload.wikimedia.org/wikipedia/commons/thumb/9/91/Pizza-3007395.jpg/1280px-Pizza-3007395.jpg" alt="Pizza">
              <footer>
                  <h4>All Rights Reserved</h4>
              </footer>
          </body>
          </html>
        dest: /var/www/html/index.html
        mode: '0644'
      tags: configure_website

    - name: Enable the custom site configuration
      copy:
        content: |
          <VirtualHost *:80>
              DocumentRoot /var/www/html
              <Directory /var/www/html>
                  Options Indexes FollowSymLinks
                  AllowOverride All
                  Require all granted
              </Directory>
              ErrorLog ${APACHE_LOG_DIR}/error.log
              CustomLog ${APACHE_LOG_DIR}/access.log combined
          </VirtualHost>
        dest: /etc/apache2/sites-available/custom-site.conf
        mode: '0644'
      notify:
        - reload apache
      tags: configure_custom_site

    - name: Enable the custom site configuration
      command: a2ensite custom-site.conf
      notify:
        - reload apache
      tags: enable_custom_site

    - name: Restart Apache to apply any configuration changes
      service:
        name: apache2
        state: restarted
      tags: restart_apache

    - name: Display Apache status
      command: systemctl status apache2
      register: apache_status
      changed_when: false
      tags: apache_status

  handlers:
    - name: reload apache
      service:
        name: apache2
        state: reloaded

Run the Playbook:

Execute the playbook using the ansible-playbook command:

ansible-playbook apache.yml -K

• You notice a failure on one of the servers, all I had to do was integrate the apt-get update command into the ansible script to make it successful.

Monitoring with ELK stack

• To implement a central logging solution using the ELK stack (Elasticsearch, Logstash, and Kibana) and create a dashboard or report to overview the security status of all servers, follow these detailed steps. This guide will cover setting up the ELK stack on a central logging server and configuring other servers to send their logs.

1. Central Logging Server Setup

1.1 Java Environment Packages

Elasticsearch and logstash requires Java to run. Ensure its comes pre-installed once they are installed. Keep below commands and run it to check java version after ELK installation.

/usr/share/elasticsearch/jdk/bin/java -version
/usr/share/logstash/jdk/bin/java -version

1.2 Install and Configure Elasticsearch

sudo apt-get install apt-transport-https

• Import the Elasticsearch PGP Key

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

Installing from the APT repository:

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

sudo apt-get update && sudo apt-get install elasticsearch

Configure Elasticsearch

Edit the /etc/elasticsearch/elasticsearch.yml file

cluster.name: my-application
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: <ipaddress>
http.port: 9200
discovery.seed_hosts: ["<>"]
cluster.initial_master_nodes: ["node-1"]
xpack.security.enabled: false

Start and enable Elasticsearch

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

Check Elasticsearch logs

tail /var/log/elasticsearch/my-application.log

1.3 Install and Configure Logstash

# Install Logstash
sudo apt install -y logstash

# Create Logstash configuration file for Filebeat input
sudo tee /etc/logstash/conf.d/filebeat.conf > /dev/null <<EOF
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://<ipaddress>:9200"]
    index => "security-logs-%{+YYYY.MM.dd}"
  }
}
EOF

Start and enable Logstash

sudo systemctl start logstash
sudo systemctl enable logstash
sudo systemctl status logstash

1.4 Install and Configure Kibana

Install Kibana

sudo apt install -y kibana
sudo apt-get update && sudo apt-get install kibana

Configure Kibana, make modifications to /etc/kibana/kibana.yml file

Uncomment and set server.port

sudo sed -i 's/#server.port: 5601/server.port: 5601/' /etc/kibana/kibana.yml

Uncomment and modify elasticsearch.hosts

sudo sed -i 's/#elasticsearch.hosts: \["http:\/\/localhost:9200"\]/elasticsearch.hosts: \["http:\/\/<ipaddress>:9200"\]/' /etc/kibana/kibana.yml

Uncomment and modify server.host

sudo sed -i 's/#server.host: "localhost"/server.host: "<ipaddress>"/' /etc/kibana/kibana.yml

server.name: "ServerA"
server.publicBaseUrl: "http://<ipaddress>:5601"

Start and enable Kibana

sudo systemctl start kibana
sudo systemctl enable kibana
sudo systemctl status kibana

Configure firewall on Central Logging Server

# Allow necessary ports
sudo ufw allow 9200/tcp   # Elasticsearch
sudo ufw allow 5601/tcp   # Kibana
sudo ufw allow 5044/tcp   # Logstash
sudo ufw enable

Install and Configure Filebeat on All Servers that will feed logs to kibana

# Install Filebeat
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

sudo apt update
sudo apt install -y filebeat

Edit the Filebeat configuration file /etc/filebeat/filebeat.yml on each server:

• setting up filebeat to ship logs to elastic search,

output.elasticsearch:
  hosts: ["<ipaddress>:9200"]
  username: "kibana_user"
  password: "passw"

setup.kibana:
    host: "<ipaddress>:5601"
    username: "kibana_user"  
    password: "passw"

But if setting up filebeat to ship logs through logstash, uncomment and modify the output.logstash section: but note that output.logstash and output.elasticsearch cannot be enabled at same time.

# output.logstash
# The Logstash hosts
  #hosts: ["<ipaddress>:5044"]

List the available modules:

cd /usr/share/filebeat/bin
filebeat modules list

I have installed apache previously so we’re going to configure Filebeat to ship apache logs, system logs and authentication logs.

cd /etc/filebeat/modules.d
filebeat modules enable apache

nano /etc/filebeat/modules.d/apache.yml

- module: apache

# Access logs

  access:
    enabled: true
    var.paths:
      - /var/log/apache2/access.log*

# Error logs

  error:
    enabled: true
    var.paths:
      - /var/log/apache2/error.log*

filebeat modules enable system

nano /etc/filebeat/modules.d/system.yml

- module: system

# Syslog

  syslog:
    enabled: true
    var.paths:
      - /var/log/syslog*

# Authorization logs

  auth:
    enabled: true
    var.paths:
      - /var/log/auth.log*

Below can be used if it were nginx server that was installed.

cd /usr/share/filebeat/bin
filebeat modules enable nginx

nano /etc/filebeat/modules.d/nginx.yml

- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log*"]

Test that the configuration is okay

$ sudo filebeat test config
sudo filebeat test config -e
Config OK

2.3 Start and Enable Filebeat on All Servers

# Start and enable Filebeat
sudo systemctl start filebeat
sudo systemctl enable filebeat
sudo systemctl status filebeat

2.4 Adjust Firewall on All Servers if enabled

• Ensure UFW allows traffic to and from the Central Logging Server:

# Allow traffic to the Central Logging Server
sudo ufw allow 5044/tcp   # Logstash
sudo ufw enable

• Test for connectivity amongst all filebeats host servers

curl -X GET "<ipaddress>:9200/"
curl -I <http://<ipaddress>:5601> [firewall]

Create Dashboards and Reports in Kibana [Run this on central logging server only]

cd /usr/share/filebeat/bin
sudo filebeat setup --dashboards

• Enable default dashboard

sudo filebeat setup -e

4.1 Access Kibana

• Open your web browser and navigate to http://<ipaddress>:5601.

• In the side navigation, click Discover. To see Filebeat data, make sure the predefined filebeat-* data view is selected.

• If you don’t see data in Kibana, try changing the time filter to a larger range. By default, Kibana shows the last 15 minutes.

• In the side navigation, click Dashboard, then select the dashboard that you want to open.

  syslog dashboard

• 

  ssh login dashboard

• 

  sudo commands dashboard

• 

• Optionally, Configure your visualization to display the status data you are interested in (e.g., number of failed login attempts), select a pre-set dashboard and Add visualizations, Click Save.

In wrapping up, the journey of securing and automating cloud servers is a critical aspect of modern DevOps practices. Through meticulous server hardening, continuous monitoring, and strategic use of automation tools like Ansible, we've crafted a resilient and scalable infrastructure. This process not only fortifies your servers against vulnerabilities but also enhances efficiency and consistency across deployments. By adopting these best practices, your infrastructure would be positioned to maintain a secure, reliable, and scalable environment, enabling you to confidently support your company's growth and technological advancements.

Network security skills apply directly to cloud security measures. Understanding containerization, microservices, and Infrastructure as Code (IaC) is crucial. Network troubleshooting, edge computing, compliance, cost optimization, and cloud-native networking all benefit from a strong foundation in traditional networking principles.

Lets picture a scenario where an organization is opening a new branch office and needs to set up a secure network infrastructure that integrates with the main office. As a Cloud Platform Engineer, you’re tasked with designing, implementing, and automating this setup. I will be running through a step by step process of how to achieve this using a vmware worksation. The concept remains the same whether you use on-prem or cloud resources.

Deploying 3 VMs Using VMware Workstation

Network Architecture Diagram:

A Simple Network Architecture

Diagram Details:

• Main Office VPN Gateway: Connects to the Branch Router via a VPN tunnel.

• Branch Router: Provides internet access and routes traffic between the branch office and the main office via the VPN tunnel.

• Branch Switch: Central hub connecting all devices within the branch office.

• Peripheral Devices: Connected directly to the Branch Switch.

• Branch Server: Manages DNS, VPN, DHCP, and NTP services.

• Client Desktops: Connect to the Branch Switch to access network resources and services.

Using VMware Workstation, I took the steps below to deploy 3 VMs and achieve the network configuration:

Create Three VMs

Create three VMs using VMware Workstation and a predefined VMDK disk (located at OSBoxes),https://www.osboxes.org/ubuntu/#ubuntu-24-04-vmware follow these steps:

1. Open VMware Workstation

2. Start VMware Workstation on your machine.

3. Create a New Virtual Machine

4. For each of the three VMs (Main Office Server, Branch Office Server, and Client VM), follow these steps:

• Select File > New Virtual Machine…

• Specify Disk File

• Select I will install the operating system later and click next.

• Select a guest operating system Linux and version ubuntu

• Choose the location where you have kept the custom .vmdk file Complete the setup and click finish

• Name the Virtual Machine

• Give a name to the virtual machine, e.g., “Main Office Server,” “Branch Office Server,” or “Client VM.” Specify the location where you want to store the VM files.

• Finish

• Click “Finish” to complete the VM creation process.

• Repeat these steps for each of the three VMs.

  • Now select a virtual machine, click on VM then settings, this opens the VM properties, click Add, select hard disk, scsi, use an existing virtual disk, browse for location of disk and click finish.

Now you can start the VM with the imported disk [.vmdk file] you have selected.

Network Configuration

Create Virtual Networks

1. Create Virtual Networks in VMware Workstation:

MainOfficeNet:

1. Open VMware Workstation.

2. Go to Edit > Virtual Network Editor.

3. Click Add Network and select a network (e.g., VMnet2).

4. Set the network to Host-only.

5. Set the subnet IP to 192.168.1.0 and the subnet mask to 255.255.255.0.

6. Disable the DHCP for this network.

7. Rename VMnet3 to MainOfficeNet.

• BranchOfficeNet:

1. Follow the same steps to create another network (e.g., VMnet3).

2. Set the network to Host-only.

3. Set the subnet IP to 192.168.5.0 and the subnet mask to 255.255.255.0.

4. Disable the DHCP for this network.

5. Rename VMnet3 to BranchOfficeNet.

1. Assign Networks to VMs:

• Main Office Server:

1. Go to the settings of the Main Office Server VM.

2. Under Network Adapter, add the network adapter.

3. Attach the adapter to VMnet2 (MainOfficeNet).

• Branch Office Server:

1. Go to the settings of the Branch Office Server VM.

2. Under Network Adapter, add the network adapters.

3. Attach the adapter to VMnet3 (BranchOfficeNet).

• Client VM:

1. Go to the settings of the Client VM.

2. Under Network Adapter, attach the adapter to VMnet3 (BranchOfficeNet).

3. Power On the VMs

4. Power on each VM by right-clicking each VM and selecting “Power On.”

Configure Network Interfaces

Updating or Replacing Existing Connections

1. Identify and Modify Existing Connections

• Delete Existing Connections:

sudo nmcli con delete netplan-ens33 
sudo nmcli con delete 'Wired connection 1'

• Add New Connections:

• Add MainOfficeNet to ens33

sudo nmcli con add type ethernet ifname ens33 con-name MainOfficeNet ip4 192.168.1.10/24

1. Apply and Bring Up Connections

• Bring Up the Connection:

• sudo nmcli con up MainOfficeNet

1. Restart NetworkManager

2. Restart NetworkManager to apply the changes:

sudo systemctl restart NetworkManager

Configuration for Branch Office Server

Configure Network Interfaces:

• Delete Existing Connections:

sudo nmcli con delete netplan-ens33 
sudo nmcli con delete 'Wired connection 1'

• Add New Connections:

• # Add BranchOfficeNet to ens33

sudo nmcli con add type ethernet ifname ens33 con-name BranchOfficeNet ip4 192.168.5.10/24

1. Bring Up the Connections:

• sudo nmcli con up BranchOfficeNet

1. Verify the configuration:

nmcli con show 
nmcli con show --active 
ip addr show 
ifconfig -a

Configuration for Client VM

1. Delete Existing Connections:

• sudo nmcli con delete 'Wired connection 1'

Add DHCP Configuration:

• nmcli con add type ethernet ifname ens33 con-name BranchOfficeNet ipv4.method auto

Bring Up the Connection:

• sudo nmcli con up BranchOfficeNet

Verify Configuration

• From the Client VM, ensure it gets an IP address via DHCP and check connectivity:

• nmcli device show ens33 ip route ip addr

Setting Up DNS:

Setting up DNS with BIND9 involves configuring both the main and branch servers to handle local domain resolution and forward external queries appropriately. Below are the detailed steps for setting this up.

Main Server (main.abc.local)

1. Install BIND9

sudo apt update
sudo apt install bind9 bind9utils bind9-doc -y

2. Configure BIND9

Edit the BIND9 configuration files to set up the master server:

• Edit named.conf.local

• sudo nano /etc/bind/named.conf.local

Add the following:

zone "abc.local" {
    type master;
    file "/etc/bind/db.abc.local";
};

zone "5.168.192.in-addr.arpa" {
    type master;
    file "/etc/bind/db.192.168.5";
};

Create Zone Files

• Create the forward zone file for abc.local:

• sudo nano /etc/bind/db.abc.local

Add the following content:

$TTL    604800
@       IN      SOA     main.abc.local. admin.abc.local. (
                             2         ; Serial
                        604800         ; Refresh
                         86400         ; Retry
                       2419200         ; Expire
                        604800 )       ; Negative Cache TTL
;
@       IN      NS      main.abc.local.
main    IN      A       192.168.1.10
branch  IN      A       192.168.5.10
client  IN      A       192.168.5.15

• Create the reverse zone file for 192.168.5.x:

• sudo nano /etc/bind/db.192.168.5

• Add the following content:

$TTL    604800
@       IN      SOA     main.abc.local. admin.abc.local. (
                             2         ; Serial
                        604800         ; Refresh
                         86400         ; Retry
                       2419200         ; Expire
                        604800 )       ; Negative Cache TTL
;
@       IN      NS      main.abc.local.
10      IN      PTR     branch.abc.local.
15      IN      PTR     client.abc.local.

3. Configure Forwarders

Edit named.conf.options to include forwarders:

sudo nano /etc/bind/named.conf.options

Add the following within the options block:

options {
    directory "/var/cache/bind";

    forwarders {
        8.8.8.8;  // Google's DNS
        8.8.4.4;  // Google's DNS
    };

    dnssec-validation auto;

    listen-on-v6 { any; };
};

4. Restart BIND9

sudo systemctl restart bind9
systemctl status bind9
systemctl restart named.service

Branch Server (branch.abc.local)

1. Install BIND9

sudo apt update
sudo apt install bind9 bind9utils bind9-doc -y

Configure iptables to permit incoming dns queries from client systems.

Allow incoming DNS traffic on port 53 (UDP)

sudo iptables -A INPUT -p udp - dport 53 -j ACCEPT

Allow incoming DNS traffic on port 53 (TCP)

sudo iptables -A INPUT -p tcp - dport 53 -j ACCEPT

Save the iptables rules

sudo sh -c “iptables-save > /etc/iptables/rules.v4”

2. Configure BIND9 as Slave

Edit the BIND9 configuration files to set up the branch server:

• Edit named.conf.local

• sudo nano /etc/bind/named.conf.local

• Add the following:

zone "abc.local" {
    type slave;
    file "/var/cache/bind/db.abc.local";
    masters { 192.168.1.10; };
};

zone "5.168.192.in-addr.arpa" {
    type slave;
    file "/var/cache/bind/db.192.168.5";
    masters { 192.168.1.10; };
};

• Configure Forwarding

• Edit named.conf.options to forward unknown/external queries to the main server:

• sudo nano /etc/bind/named.conf.options

• Add the following within the options block:

options {
    directory "/var/cache/bind";

    forwarders {
        192.168.1.10;  // Main server's IP
    };

    dnssec-validation auto;

    listen-on-v6 { any; };
};

3. Restart and enable BIND9 service at every system reboot

sudo systemctl restart bind9
sudo systemctl enable bind9

Check Zone Transfer Logs on Branch Server

sudo systemctl status bind9

Test DNS Resolution

On the branch server, use dig to test:

dig main.abc.local @localhost
dig branch.abc.local @localhost
dig client.abc.local @localhost
dig google.com @localhost

The dig output indicates that the DNS server running on localhost successfully resolved the domain main.abc.local to the IP address 192.168.1.10, but it issued a warning because .local is reserved for mDNS. The query was handled correctly with an authoritative answer and the response took 4 milliseconds.

The dig output indicates that the DNS server running on localhost successfully resolved google.com to the IP address 142.251.32.78. The query was handled correctly with an authoritative answer, and the response took 1 millisecond. The result shows that your local DNS server is capable of resolving external domain names as well, as it is properly configured to forward queries or perform DNS resolution.

By following these steps, your branch server will be configured to resolve local domain queries from its own zone files and forward any unknown or external domain queries to the main server. The main server is configured to forward external queries to public DNS servers (like Google’s DNS).

Client Configuration (192.168.5.15)

Ensure that the client machine is configured to use the branch office DNS server for its DNS queries.

Setup # operation for /etc/resolv.conf.

nameserver 192.168.5.10 nameserver 127.0.0.53 options edns0 trust-ad search branch.company.local

1. Configure/etc/netplan/01-netcfg.yaml

2. Edit the netplan configuration to use the branch server for DNS:

• sudo nano /etc/netplan/01-netcfg.yaml

1. Ensure it contains:

network:
  version: 2
  ethernets:
    ens33:
      dhcp4: yes
      routes:
        - to: default
          via: 192.168.5.10
      nameservers:
        addresses:
          - 192.168.5.10
          - 8.8.8.8

Apply the Netplan configuration:

sudo netplan apply

Verify the Setup

1. Check DNS Resolution

2. On the client, verify that DNS resolution works:

ping main.abc.local 
ping branch.abc.local  
nslookup main.abc.local 
nslookup branch.abc.local  
nslookup client.abc.local

External DNS Resolution:

• nslookup google.com

1. Test External DNS Resolution

2. On the branch server, test that external domain resolution works:

• dig google.com

1. Check BIND9 Status

2. Ensure that BIND9 is running correctly on both the main and branch servers:

• sudo systemctl status bind9

By following these steps, you should have a working DNS setup where the branch office can resolve both local and external domains through the main office DNS server.

Steps to Deploy Tinc VPN

Tinc VPN is a flexible and powerful VPN daemon that supports full-mesh routing and dynamic links between nodes. Here’s how to deploy Tinc VPN on both the main server and branch server.

Prerequisites

• Two Linux servers (main and branch) with root or sudo access.

• Ensure that both servers have open network ports for Tinc (default is 655 for TCP and UDP).

Step 1: Install Tinc VPN

On Both Servers [main and branch]:

1. Update the package list and install Tinc:

• sudo apt update sudo apt install tinc

Step 2: Create Tinc Configuration Directories

On Both Servers:

1. Create the main configuration directory for Tinc:

• sudo mkdir -p /etc/tinc/vpn/hosts

1. Navigate to the Tinc directory:

• cd /etc/tinc/vpn

Step 3: Generate Tinc Configuration Files

On Both Servers:

1. Create thetinc.conffile:

• sudo nano tinc.conf

1. Add the following configuration (replaceMainServerandBranchServerwith appropriate hostnames):

2. Main Server (main):

Name = main  
AddressFamily = ipv4  
Interface = tun0

1. Branch Server (branch):

Name = branch  
AddressFamily = ipv4  
Interface = tun0  
ConnectTo = main

1. Create thetinc-upscript:

• sudo nano tinc-up

1. Add the following content (adjust IP addresses as needed):

2. Main Server:

• #!/bin/sh ifconfig $INTERFACE 10.0.0.1 netmask 255.255.255.0

1. Branch Server:

• #!/bin/sh ifconfig $INTERFACE 10.0.0.2 netmask 255.255.255.0

1. Make thetinc-upscript executable:

• sudo chmod +x tinc-up

1. Create thetinc-downscript:

• sudo nano tinc-down

1. Add the following content:

• #!/bin/sh ifconfig $INTERFACE down

1. Make thetinc-downscript executable:

• sudo chmod +x tinc-down

Step 4: Configure Host Files

On Both Servers:

1. Create a host configuration file for each server:

2. Main Server:

• sudo nano hosts/main

1. Branch Server:

• sudo nano hosts/branch

1. Add the following content:

Main Server: [hosts/main]

• Address = 192.168.1.10 Subnet = 10.0.0.1/32

Branch Server: [hosts/branch]

• Address = 192.168.5.10 Subnet = 10.0.0.2/32

Step 5: Generate Tinc Keys

On Both Servers:

1. Generate the Tinc RSA key pair:

sudo tincd -n vpn -K4096

This will generatersa_key.privandhosts/<hostname>files.
Share the contents of these files between the servers:

Main Server:

• sudo cat /etc/tinc/vpn/hosts/main

Branch Server:

• sudo cat /etc/tinc/vpn/hosts/branch

1. Copy the public key portion (the lines starting with-----BEGIN RSA PUBLIC KEY-----to-----END RSA PUBLIC KEY-----) from each server and add it to the corresponding host file on the other server:

2. On Main Server (/etc/tinc/vpn/hosts/branch):

• -----BEGIN RSA PUBLIC KEY----- (Branch Server public key here) -----END RSA PUBLIC KEY-----

1. On Branch Server (/etc/tinc/vpn/hosts/main):

• -----BEGIN RSA PUBLIC KEY----- (Main Server public key here) -----END RSA PUBLIC KEY-----

1. Ensure the hosts files, main and branch exists on both servers. [copy them all to each other]

Step 6: Start Tinc VPN

On Both Servers:

1. Enable and start the Tinc service:

• sudo systemctl enable tinc@vpn sudo systemctl start tinc@vpn sudo systemctl status tinc@vpn

1. Check the status of the Tinc service:

• sudo systemctl status tinc@vpn

Step 7: Verify the Connection

1. Verify that thetun0interface is up and configured correctly on both servers:

• ifconfig tun0

1. Check the connectivity between the servers:

• ping 10.0.0.2 # From Main Server ping 10.0.0.1 # From Branch Server ssh osboxes@10.0.0.2 # From Main Server ifconfig tun0

Setup DHCP on Branch Office:

1. Install DHCP (isc-dhcp-server):

• sudo apt update sudo apt install isc-dhcp-server

1. Configure DHCP Server:

2. Edit the DHCP server configuration file:

• sudo nano /etc/dhcp/dhcpd.conf

1. Add or modify the following lines to configure the DHCP server:

# Optionally specify a domain name
option domain-name "company.local";

# Specify the default lease time (in seconds)
default-lease-time 600;

# Specify the maximum lease time (in seconds)
max-lease-time 7200;

# Specify the network and subnet
subnet 192.168.5.0 netmask 255.255.255.0 {
    range 192.168.5.50 192.168.5.100;  # IP range to be assigned to clients
    option routers 192.168.5.10;       # Gateway
    option domain-name-servers 192.168.5.10, 8.8.8.8;  # DNS servers

1. Specify Network Interface for DHCP:

2. Edit the file /etc/default/isc-dhcp-server to specify the network interface that the DHCP server should listen on. For example:

INTERFACESv4="ens33"

Replace ens33 with the name of the network interface connected to your local network.

1. Restart DHCP Server:

2. Activate and start the DHCP server to apply the changes:

sudo systemctl start isc-dhcp-server
systemctl status isc-dhcp-server
systemctl enable isc-dhcp-server

On the Client Server

1. Configure Netplan for DHCP

Edit the Netplan configuration file to use DHCP for obtaining an IP address:

1. Edit Netplan Configuration:

sudo nano /etc/netplan/01-netcfg.yaml

1. Modify the Configuration to Use DHCP:

2. Update the file to use DHCP for the ens33 interface:

network:
version: 2
ethernets:
 ens33:
   dhcp4: yes
   routes:
     - to: default
       via: 192.168.5.10
   nameservers:
     addresses:
       - 8.8.8.8
       - 8.8.4.4

1. Apply the Netplan configuration:

• sudo netplan apply

1. A new network will be created and will then need to be attached to ens33, then reboot so that it can pickup an IP.

1. Or you can use below command to configure the network to be auto assigned an IP.

sudo nmcli con add type ethernet ifname ens33 con-name netplan-ens33 ipv4.method auto

Verify the Setup

1. Check DHCP Lease on Client:

nmcli con show 
sudo nmcli con up netplan-ens33 
nmcli device show ens33 
nmcli device status 
nmcli con show netplan-ens33

Once the client server is powered up, it should automatically obtain an IP address from the branch server. Verify the IP address on the client:

ip addr show ens33

You should see an IP address within the range specified in the DHCP server configuration (e.g., 192.168.5.15 to 192.168.5.40).

By following these steps, your branch server will act as a DHCP server, and the client server will automatically receive its IP address from the branch server when powered up.

Provide Internet Access from Branch Office Server to Client System

To provide internet access from your branch office server (which has internet access) to your client system (which does not), you can set up Network Address Translation (NAT) and configure IP forwarding on the branch office server. Here’s a step-by-step guide to achieve this on your Ubuntu server:

1. Enable IP Forwarding

You need to enable IP forwarding on the branch office server to allow it to forward packets between the client system and the internet.

Open the /etc/sysctl.conf file with a text editor:

sudo nano /etc/sysctl.conf

Find the line:

#net.ipv4.ip_forward=1

Uncomment it (remove the #), so it reads:

net.ipv4.ip_forward=1

Apply the changes:

sudo sysctl -p

2. Configure NAT (Network Address Translation)

Use iptables to configure NAT on the branch office server. This will allow the client system to use the branch office server’s internet connection.

Run the following commands:

sudo iptables -t nat -A POSTROUTING -o ens38 -j MASQUERADE
sudo iptables -A FORWARD -i ens38 -o ens33 -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A FORWARD -i ens33 -o ens38 -j ACCEPT

Replace <internet_interface> with the name of the network interface connected to the internet (e.g., ens33) and <client_interface> with the name of the network interface [BranchOfficeNet]connected to the client system (e.g., ens34).

3. Save theiptables Configuration

To ensure that your iptables rules persist after a reboot, you need to save them. On Ubuntu, you can use the iptables-save command and store the rules in a file.

Save the rules:

Install iptables-persistent to load the rules at boot [if not already installed]:

sudo apt-get install iptables-persistent

sudo sh -c "iptables-save > /etc/iptables/rules.v4"
OR
sudo iptables-save | sudo tee /etc/iptables/rules.v4

Check that the NAT rule is in place:

sudo iptables -t nat -L -v

4. Configure the Client System

Ensure that the client system has its default gateway set to the branch office server’s IP address on the local network.

On the client system, you can set the default gateway by editing the /etc/netplan/01-netcfg.yaml file or using the ip route command. For example:

sudo nano /etc/netplan/01-netcfg.yaml
network:
version: 2
ethernets: ens33:
dhcp4: yes
routes:

network:
  version: 2
  ethernets:
    ens33:
      dhcp4: yes
      routes:
        - to: default
          via: 192.168.5.10
      nameservers:
        addresses:
          - 192.168.5.10
          - 8.8.8.8

sudo netplan apply

5. Test the Configuration

Test the internet connectivity from the client system by pinging an external website or using a web browser.

ping google.com

NTP setup

I will be using chrony set up the NTP service, where the client system receives time sychronization from the branch server, following these steps:

On the Branch Server

1. Install Chrony:

sudo apt-get update
sudo apt-get install chrony -y

1. Configure Chrony:

Edit the Chrony configuration file:

sudo nano /etc/chrony/chrony.conf

Add the following lines to allow the client server to get time from the branch server:

allow 192.168.5.0/24

This allow directive permits the specified network to access the time service. Adjust the network range if necessary.

Configure IPtables to permit NTP service syncronization on port 123 from client systems:

sudo iptables -A INPUT -p udp --dport 123 -j ACCEPT
sudo iptables -A OUTPUT -p udp --sport 123 -j ACCEPT

sudo iptables-save | sudo tee /etc/iptables/rules.v4
sudo iptables -L -v -n

1. Start Chrony:

sudo systemctl start chrony
sudo systemctl enable chrony

1. Verify Chrony Status:

sudo systemctl status chrony

On the Client Server

1. Install Chrony:

sudo apt-get update
sudo apt-get install chrony -y

1. Configure Chrony:

Edit the Chrony configuration file:

sudo nano /etc/chrony/chrony.conf

Add the branch server’s IP address as the NTP server, remove all other default ntp sources by hashing it out:

server 192.168.5.10 iburst #Branch server IP address

1. Start Chrony:

sudo systemctl start chrony

1. Verify Chrony Status:

sudo systemctl status chrony

Verification

On the brannch server, run ‘netplan apply’

To ensure that the client server is correctly synchronizing its time from the branch server, you can use the chronyc command.

On the Client Server:

1. Check Chrony Sources:

chronyc sources

You should see the branch server (192.168.5.10 or its hostname) listed as a source.

1. Other chrony commands are listed below:

chronyc tracking
sudo chronyc -a makestep
chronyc activity
chronyc serverstats
chronyc sources -v

To set up Snort to monitor for suspicious activity on your servers, follow these steps:

1. Install Snort

On Debian/Ubuntu:

sudo apt update
sudo apt install snort

2. Configure Snort

Initial Configuration:

Snort’s main configuration file is located at /etc/snort/snort.conf. Open this file to configure it according to your network setup.

sudo nano /etc/snort/snort.conf

Configure Network Variables:

Set your HOME_NET and EXTERNAL_NET variables. set up Snort to monitor three interfaces with different subnet, set it as follows:

for Main server:

var HOME_NET [192.168.1.0/24,10.0.0.0/24,192.168.79.0/24]
var EXTERNAL_NET !$HOME_NET

for branch server:

var HOME_NET [192.168.5.0/24,10.0.0.0/24,192.168.79.0/24] 
var EXTERNAL_NET !$HOME_NET

for client:

var HOME_NET 192.168.5.0/24 
var EXTERNAL_NET !$HOME_NET

Include Rule Files:

Ensure the rule paths are correctly specified:

echo 'include $RULE_PATH/local.rules' >> /etc/snort/snort.conf

4. Create Local Rules

You can create custom rules specific to your network in the local.rules file:

sudo nano /etc/snort/rules/local.rules

Add some basic rules:

# Alert on any ICMP traffic
alert icmp any any -> any any (msg:"ICMP Traffic Detected"; sid:1000001; rev:1;)

# Alert on any TCP traffic to port 123 (NTP)
alert tcp any any -> any 123 (msg:"TCP Traffic to NTP port 123 detected"; sid:1000002; rev:1;)

# Alert on any TCP traffic to port 53 (DNS)
alert tcp any any -> any 53 (msg:"TCP Traffic to DNS port 53 detected"; sid:1000003; rev:1;)

# Alert on any UDP traffic to port 53 (DNS)
alert udp any any -> any 53 (msg:"UDP Traffic to DNS port 53 detected"; sid:1000004; rev:1;)

# Alert on any TCP traffic to port 22 (SSH)
alert tcp any any -> any 22 (msg:"TCP Traffic to SSH port 22 detected"; sid:1000005; rev:1;)

5. Test Snort Configuration

Test the configuration to ensure there are no syntax errors:

sudo snort -T -c /etc/snort/snort.conf

6. Run Snort

Run Snort in IDS mode:

To monitor multiple network interfaces with Snort, You can run Snort multiple times, each time specifying a different interface:

snort -A console -c /etc/snort/snort.conf -i <network-interface>
snort -A console -c /etc/snort/snort.conf -i ens33 &
snort -A console -c /etc/snort/snort.conf -i eth1 &

Replace <network-interface> with your network interface, for example, ens33.

• Sample output of result

7. Automate Snort Startup

To ensure Snort starts on boot, create a systemd service file.

Create Systemd Service File:

sudo nano /etc/systemd/system/snort33.service

Add the following content:

[Unit]
Description=Snort NIDS
After=network.target

[Service]
ExecStart=/usr/sbin/snort -c /etc/snort/snort.conf -i ens33
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target

Create another Systemd Service File for vpn tunnel [connecting branch and main servers]:

sudo nano /etc/systemd/system/tun.service

Add the following content:

[Unit]
Description=Snort tunnel NIDS
After=network.target

[Service]
ExecStart=/usr/sbin/snort -c /etc/snort/snort.conf -i tun0
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target

Start the services

systemctl daemon-reload
systemctl enable tun.service
systemctl start tun.service

systemctl enable snort33.service
systemctl start snort33.service
systemctl status snort33.service

8. Monitor Snort Logs

Snort logs its alerts to /var/log/snort/snort.alert.fast by default. You can monitor this file for suspicious activity.

tail -f /var/log/snort/snort.alert.fast

By following these steps, Snort will monitor your network traffic for suspicious activity and log alerts based on the rules defined. You can optionally adjust and expand the rules and configuration to match the specific requirements and threats relevant to your environment.

Configure NMAP:

To use nmap for network scanning and to ensure that all expected services are running and accessible, follow these steps:

1. Install Nmap on branch server

First, make sure nmap is installed on your system. You can install it using the package manager for your distribution:

For Debian/Ubuntu-based systems:

sudo apt-get update
sudo apt-get install nmap

2. Basic Network Scan

To scan an entire network for live hosts and open ports, use:

nmap -sP 192.168.5.0/24

or

nmap -sn 192.168.5.0/24

This will perform a “ping scan” to determine which hosts are online.

3. Service Detection

To detect services running on specific hosts or an entire network, use the following command:

nmap -sV 192.168.5.0/24

This performs a service/version detection scan to determine what services and versions are running on open ports.

4. Port Scanning

To scan for open ports on a specific host:

nmap -p- 192.168.5.10

This scans all 65535 ports. You can specify a range of ports:

Command below scans for port 22 and port 80 on the server

nmap -p 22,80,443 192.168.5.15

5. Aggressive Scan

An aggressive scan performs a detailed scan including service detection, OS detection, and more:

nmap -A 192.168.5.15

This will give you extensive information about the host, including open ports, services, OS, and more.

7. Scan Multiple Hosts

To scan multiple hosts or subnets, list them in the command:

sudo nmap -p 80,22 192.168.5.10 192.168.5.15 192.168.1.10

8. Output Formats

For easier analysis, you can save the results in various formats:

• Normal Output:

sudo nmap -oN scan_results.txt 192.168.5.15 192.168.5.10

9. Schedule Regular Scans

To ensure services are consistently monitored, you can set up a cron job to run nmap regularly.

Edit the crontab with:

crontab -e

Add an entry to run nmap at a regular interval, e.g., daily at midnight:

0 0 * * * /usr/bin/nmap -sP 192.168.5.0/24 > /var/log/nmap_daily_scan.log

These steps will help ensure all expected services are running and accessible on your network.

Capture network packets withtcpdump

Capturing and analyzing network traffic using tcpdump and Wireshark can help you verify that your VPN is working correctly and identify any potential issues. Below are the steps to perform this task on both the main and branch servers.

Installtcpdump if not already pre-installed.

For Debian/Ubuntu-based systems:

sudo apt-get update
sudo apt-get install tcpdump

Capture Traffic

You need to capture traffic on the interfaces involved in the VPN connection. For example, my VPN setup interface is tun0, I can use the following command:

sudo tcpdump -i tun0 -w vpn_traffic.pcap

• -i tun0: Specifies the interface to capture traffic on (replace tun0 with your own actual VPN interface).

• -w vpn_traffic.pcap: Writes the captured packets to a file named vpn_traffic.pcap.

You can also filter the traffic by IP address or port if you want to narrow down the capture:

sudo tcpdump -i tun0 host 10.0.0.1 -w vpn_traffic.pcap

Capture on Both Servers

Run the above tcpdump commands on both the main and branch servers to capture the relevant traffic.

I will also run a ping and ssh commands across both servers in another session so that ICMP and SSH packets can be captured in the tcpdump process.

ping 10.0.0.2 # From the main server 
ping 10.0.0.1 # From the branch server 
ssh user@10.0.0.2 # From the main server 
ssh user@10.0.0.1 # From the branch server

Step 2: Transfer the Capture Files

After capturing the traffic, transfer the .pcap files to your local machine for analysis. You can use scp (secure copy) to do this:

scp user@branch_server:/path/to/vpn_traffic.pcap /local/path/
scp user@main_server:/path/to/vpn_traffic.pcap /local/path/

Step 3: Analyze Traffic with Wireshark

Install Wireshark

Wireshark is available for Windows, macOS, and Linux. You can download it from the official Wireshark website.

Open captured files in Wireshark and analyze:

1. Open Wireshark.

2. Click File -> Open and select the transferred .pcap files to open.

3. Use Wireshark’s display filters to focus on specific traffic. For example, to display only traffic between two IP addresses, enter below strings in the search bar display fileter, and click on the blue arrow to the right to apply.

• ip.addr == 10.0.0.1 && ip.addr == 10.0.0.2

4. Inspect VPN Traffic: Look at the traffic on the tun0 interface (or your VPN interface) to ensure packets are being transmitted and received correctly.

5. Check for Anomalies: Look for retransmissions, malformed packets, or any other unusual activity.

By following these steps, you can capture and analyze the network traffic on both your main and branch servers, verify the VPN functionality, and troubleshoot any potential issues using tcpdump and Wireshark.

AUTOMATION

Configure Ansible Playbooks

I will also install ansible and Based on the manual steps I have used to initially setup the configuration, I will create Ansible playbooks to automate the setup for DHCP, DNS, NTP, and basic firewall rules.

Here is a code file containing the steps to install and configure Ansible on an Ubuntu system for this environment.

#!/bin/bash

# Update package index
echo "Updating package index..."
sudo apt update

# Install Ansible
echo "Installing Ansible..."
sudo apt install ansible -y

# Optionally, install the latest version of Ansible via PPA
echo "Adding Ansible PPA..."
sudo add-apt-repository ppa:ansible/ansible -y
sudo apt update
sudo apt install ansible -y

# Verify Ansible installation
echo "Verifying Ansible installation..."
ansible --version

# Create and configure the inventory file
echo "Configuring Ansible inventory file..."
sudo tee /etc/ansible/hosts > /dev/null <<EOL
[main_server]
192.168.1.10

[branch_server]
192.168.5.10

[client_server]
192.168.5.15
EOL

# Edit the Ansible configuration file
echo "Editing Ansible configuration file..."
sudo tee /etc/ansible/ansible.cfg > /dev/null <<EOL
[defaults]
inventory = /etc/ansible/hosts
remote_user = your_user
private_key_file = /path/to/private/key
EOL

# Test Ansible configuration
echo "Testing Ansible configuration..."
ansible all -m ping

echo "Ansible installation and configuration complete!"

Instructions for Using the Script

Save the Script:

Save the above script to a file, e.g., install_ansible.sh.

Make the Script Executable:

chmod +x install_ansible.sh

Run the Script:

./install_ansible.sh

This script handles the installation of Ansible, adds the Ansible PPA if desired, sets up the inventory file, and configures the Ansible configuration file. Adjust the remote_user and private_key_file in the Ansible configuration file according to your setup.

Below are the Ansible playbooks for each service:

1. DHCP Configuration

Playbook:dhcp.yml

---
- name: Configure DHCP Server
  hosts: branch_server
  become: yes
  tasks:
    - name: Install DHCP server
      apt:
        name: isc-dhcp-server
        state: present
        update_cache: yes

    - name: Configure DHCP server
      copy:
        dest: /etc/dhcp/dhcpd.conf
        content: |
          option domain-name "company.local";
          default-lease-time 600;
          max-lease-time 7200;

          subnet 192.168.5.0 netmask 255.255.255.0 {
              range 192.168.5.50 192.168.5.100;
              option routers 192.168.5.10;
              option domain-name-servers 192.168.5.10, 8.8.8.8;
          }

    - name: Specify network interface for DHCP
      lineinfile:
        path: /etc/default/isc-dhcp-server
        regexp: '^INTERFACESv4='
        line: 'INTERFACESv4="ens33"'

    - name: Restart DHCP server
      service:
        name: isc-dhcp-server
        state: restarted
        enabled: yes

2. DNS Configuration

Playbook:dns.yml

---
- name: Configure DNS Server
  hosts: main_server
  become: yes
  tasks:
    - name: Install BIND9
      apt:
        name: "{{ item }}"
        state: present
        update_cache: yes
      with_items:
        - bind9
        - bind9utils
        - bind9-doc

    - name: Configure BIND9 named.conf.local
      copy:
        dest: /etc/bind/named.conf.local
        content: |
          zone "abc.local" {
              type master;
              file "/etc/bind/db.abc.local";
          };

          zone "5.168.192.in-addr.arpa" {
              type master;
              file "/etc/bind/db.192.168.5";
          };

    - name: Create forward zone file for abc.local
      copy:
        dest: /etc/bind/db.abc.local
        content: |
          $TTL    604800
          @       IN      SOA     main.abc.local. admin.abc.local. (
                                   2         ; Serial
                              604800         ; Refresh
                               86400         ; Retry
                             2419200         ; Expire
                              604800 )       ; Negative Cache TTL
          ;
          @       IN      NS      main.abc.local.
          main    IN      A       192.168.1.10
          branch  IN      A       192.168.5.10
          client  IN      A       192.168.5.15

    - name: Create reverse zone file for 192.168.5.x
      copy:
        dest: /etc/bind/db.192.168.5
        content: |
          $TTL    604800
          @       IN      SOA     main.abc.local. admin.abc.local. (
                                   2         ; Serial
                              604800         ; Refresh
                               86400         ; Retry
                             2419200         ; Expire
                              604800 )       ; Negative Cache TTL
          ;
          @       IN      NS      main.abc.local.
          10      IN      PTR     branch.abc.local.
          15      IN      PTR     client.abc.local.

    - name: Configure BIND9 forwarders
      lineinfile:
        path: /etc/bind/named.conf.options
        insertafter: '{'
        line: |
          forwarders {
              8.8.8.8;  // Google's DNS
              8.8.4.4;  // Google's DNS
          };

    - name: Restart BIND9
      service:
        name: bind9
        state: restarted
        enabled: yes

- name: Configure DNS Server
  hosts: branch_server
  become: yes
  tasks:
    - name: Install BIND9
      apt:
        name: "{{ item }}"
        state: present
        update_cache: yes
      with_items:
        - bind9
        - bind9utils
        - bind9-doc

    - name: Configure BIND9 named.conf.local as slave
      copy:
        dest: /etc/bind/named.conf.local
        content: |
          zone "abc.local" {
              type slave;
              file "/var/cache/bind/db.abc.local";
              masters { 192.168.1.10; };
          };

          zone "5.168.192.in-addr.arpa" {
              type slave;
              file "/var/cache/bind/db.192.168.5";
              masters { 192.168.1.10; };
          };

    - name: Configure BIND9 forwarders
      lineinfile:
        path: /etc/bind/named.conf.options
        insertafter: '{'
        line: |
          forwarders {
              192.168.1.10;  // Main server's IP
          };

    - name: Restart BIND9
      service:
        name: bind9
        state: restarted
        enabled: yes

3. NTP Configuration (using chrony) with iptables rules

Playbook:ntp.yml

---
- name: Configure NTP with Chrony
  hosts: branch_server
  become: yes
  tasks:
    - name: Install chrony
      apt:
        name: chrony
        state: present
        update_cache: yes

    - name: Configure chrony to allow network
      lineinfile:
        path: /etc/chrony/chrony.conf
        line: 'allow 192.168.5.0/24'
        state: present

    - name: Restart chrony
      service:
        name: chrony
        state: restarted
        enabled: yes

    - name: Add iptables rules for chrony
      iptables:
        chain: INPUT
        protocol: udp
        destination_port: 123
        jump: ACCEPT

    - name: Add iptables rules for chrony output
      iptables:
        chain: OUTPUT
        protocol: udp
        source_port: 123
        jump: ACCEPT

    - name: Save iptables rules
      command: iptables-save > /etc/iptables/rules.v4

- name: Configure NTP with Chrony on Client Server
  hosts: client_server
  become: yes
  tasks:
    - name: Install chrony
      apt:
        name: chrony
        state: present
        update_cache: yes

    - name: Configure chrony to use branch server
      lineinfile:
        path: /etc/chrony/chrony.conf
        line: 'server 192.168.5.10 iburst'
        state: present

    - name: Restart chrony
      service:
        name: chrony
        state: restarted
        enabled: yes

    - name: Add iptables rules for chrony
      iptables:
        chain: INPUT
        protocol: udp
        destination_port: 123
        jump: ACCEPT

    - name: Add iptables rules for chrony output
      iptables:
        chain: OUTPUT
        protocol: udp
        source_port: 123
        jump: ACCEPT

    - name: Save iptables rules
      command: iptables-save > /etc/iptables/rules.v4

4. Basic Firewall Rules with additional iptables rules

Playbook:firewall.yml

---
- name: Configure basic firewall rules and iptables
  hosts: branch_server
  become: yes
  tasks:
    - name: Install UFW
      apt:
        name: ufw
        state: present
        update_cache: yes

    - name: Allow SSH
      ufw:
        rule: allow
        name: 'OpenSSH'

    - name: Allow DHCP
      ufw:
        rule: allow
        port: 67
        proto: udp

    - name: Allow DNS
      ufw:
        rule: allow
        port: 53
        proto: udp

    - name: Allow NTP
      ufw:
        rule: allow
        port: 123
        proto: udp

    - name: Enable UFW
      ufw:
        state: enabled

    - name: Add NAT iptables rule
      iptables:
        chain: POSTROUTING
        table: nat
        out_interface: ens38
        jump: MASQUERADE

    - name: Add forward rule for related/established connections
      iptables:
        chain: FORWARD
        in_interface: ens38
        out_interface: ens33
        match: state
        state: RELATED,ESTABLISHED
        jump: ACCEPT

    - name: Add forward rule to allow traffic from ens33 to ens38
      iptables:
        chain: FORWARD
        in_interface: ens33
        out_interface: ens38
        jump: ACCEPT

    - name: Save iptables rules
      command: iptables-save > /etc/iptables/rules.v4

Inventory host File
[main_server]
192.168.1.10[branch_server]
192.168.5.10[client_server]
192.168.5.15[all]
192.168.1.10
192.168.5.10
192.168.5.15

Run the Playbooks

To execute the playbooks, use the following commands:

ansible-playbook -i hosts dhcp.yml
ansible-playbook -i hosts dns.yml
ansible-playbook -i hosts ntp.yml
ansible-playbook -i hosts firewall.yml

These playbooks will automate the configuration of DHCP, DNS, NTP with iptables rules, and basic firewall rules on the specified servers.

Infrastructure as code:

To use Vagrant to provision a VM on VMware Workstation, simulating an additional branch server and client system, follow these steps. This guide will cover installing the necessary tools, setting up Vagrant, and creating Vagrantfiles to provision the VMs.

Prerequisites

1. VMware Workstation: Ensure VMware Workstation is installed.

2. Vagrant: Install Vagrant on your system. https://developer.hashicorp.com/vagrant/install

3. Vagrant VMware Utility: Install the Vagrant VMware Utility. https://developer.hashicorp.com/vagrant/docs/providers/vmware/vagrant-vmware-utility

4. Vagrant VMware Desktop Plugin: Install the Vagrant VMware Desktop plugin.

5. Run the following command in your terminal :

vagrant plugin install vagrant-vmware-desktop

Steps to Provision VMs with Vagrant

1. Create a Directory for Your Vagrant Project

Create a directory for your Vagrant project, and navigate into it:

mkdir vagrant
cd vagrant

2. Initialize Vagrant

Initialize Vagrant in the directory:

vagrant init

This will create a Vagrantfile in the directory. You will modify the Vagrantfiles to contain the resource creation for the 3 VMs. Check for latest OS version at https://app.vagrantup.com/generic

To add additional configurations to each of the VMs, you can use a shell provisioner in the Vagrantfile. Below, I’ve created a script to set up DHCP, iptables, DNS, Chrony, and firewall rules. The script will be called provision.sh, and it will be referenced in the Vagrantfile for each VM.

Vagrant.configure("2") do |config|
  # Define the "main" VM
  config.vm.define "main" do |main|
    main.vm.box = "generic/ubuntu2310"
    main.vm.network "private_network", ip: "192.168.1.10"
    main.vm.hostname = "main"

    main.vm.provider "vmware_desktop" do |v|
      v.vmx["memsize"] = "2048"
      v.vmx["numvcpus"] = "2"
      v.vmx["disk.size"] = "20000"
    end

    main.vm.provision "shell", path: "C:/Users/balog/Desktop/vagrant-branch-client/provision.sh"
  end

  # Define the "branch" VM
  config.vm.define "branch" do |branch|
    branch.vm.box = "generic/ubuntu2310"
    branch.vm.network "private_network", ip: "192.168.5.10"
    branch.vm.hostname = "branch"

    branch.vm.provider "vmware_desktop" do |v|
      v.vmx["memsize"] = "2048"
      v.vmx["numvcpus"] = "2"
      v.vmx["disk.size"] = "20000"
    end

    branch.vm.provision "shell", path: "C:/Users/balog/Desktop/vagrant-branch-client/provision.sh"
  end

  # Define the "client" VM
  config.vm.define "client" do |client|
    client.vm.box = "generic/ubuntu2310"
    client.vm.network "private_network", type: "dhcp"
    client.vm.hostname = "client"

    client.vm.provider "vmware_desktop" do |v|
      v.vmx["memsize"] = "2048"
      v.vmx["numvcpus"] = "2"
      v.vmx["disk.size"] = "20000"
    end

    client.vm.provision "shell", path: "C:/Users/balog/Desktop/vagrant-branch-client/provision.sh"
  end
end

Script provision for vagrant file

#!/bin/bash

# Update and install necessary packages
sudo apt-get update

# Install packages based on hostname
if [[ $(hostname) == "main" || $(hostname) == "branch" ]]; then
    sudo apt-get install -y bind9 bind9utils bind9-doc tinc

    if [[ $(hostname) == "branch" ]]; then
        sudo apt-get install -y isc-dhcp-server chrony
    fi
elif [[ $(hostname) == "client" ]]; then
    sudo apt-get install -y tinc chrony
fi

# Configure BIND9 for Main Server
if [[ $(hostname) == "main" ]]; then
    # BIND9 Configuration
    cat <<EOF | sudo tee /etc/bind/named.conf.local
zone "abc.local" {
    type master;
    file "/etc/bind/db.abc.local";
};

zone "5.168.192.in-addr.arpa" {
    type master;
    file "/etc/bind/db.192.168.5";
};
EOF

    cat <<EOF | sudo tee /etc/bind/db.abc.local
\$TTL    604800
@       IN      SOA     main.abc.local. admin.abc.local. (
                           2         ; Serial
                      604800         ; Refresh
                       86400         ; Retry
                     2419200         ; Expire
                      604800 )       ; Negative Cache TTL
;
@       IN      NS      main.abc.local.
main    IN      A       192.168.1.10
branch  IN      A       192.168.5.10
client  IN      A       192.168.5.15
EOF

    cat <<EOF | sudo tee /etc/bind/db.192.168.5
\$TTL    604800
@       IN      SOA     main.abc.local. admin.abc.local. (
                           2         ; Serial
                      604800         ; Refresh
                       86400         ; Retry
                     2419200         ; Expire
                      604800 )       ; Negative Cache TTL
;
@       IN      NS      main.abc.local.
10      IN      PTR     branch.abc.local.
15      IN      PTR     client.abc.local.
EOF

    cat <<EOF | sudo tee /etc/bind/named.conf.options
options {
    directory "/var/cache/bind";

    forwarders {
        8.8.8.8;
        8.8.4.4;
    };

    dnssec-validation auto;

    listen-on-v6 { any; };
};
EOF

    sudo systemctl start bind9

# Configure BIND9 for Branch Server
elif [[ $(hostname) == "branch" ]]; then
    # BIND9 Configuration
    cat <<EOF | sudo tee /etc/bind/named.conf.local
zone "abc.local" {
    type slave;
    file "/var/cache/bind/db.abc.local";
    masters { 192.168.1.10; };
};

zone "5.168.192.in-addr.arpa" {
    type slave;
    file "/var/cache/bind/db.192.168.5";
    masters { 192.168.1.10; };
};
EOF

    cat <<EOF | sudo tee /etc/bind/named.conf.options
options {
    directory "/var/cache/bind";

    forwarders {
        192.168.1.10;
    };

    dnssec-validation auto;

    listen-on-v6 { any; };
};
EOF

    sudo systemctl start bind9

    # Configure DHCP Server
    cat <<EOF | sudo tee /etc/dhcp/dhcpd.conf
option domain-name "company.local";
default-lease-time 600;
max-lease-time 7200;

subnet 192.168.5.0 netmask 255.255.255.0 {
    range 192.168.5.15 192.168.5.50;
    option routers 192.168.5.10;
    option domain-name-servers 192.168.5.10, 8.8.8.8;
}
EOF

    sudo sed -i 's/INTERFACESv4=""/INTERFACESv4="ens33"/' /etc/default/isc-dhcp-server

    sudo systemctl start isc-dhcp-server

    # Enable IP Forwarding and configure NAT
    sudo sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.conf
    sudo sysctl -p
    sudo iptables -t nat -A POSTROUTING -o ens38 -j MASQUERADE
    sudo iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
    sudo iptables -A FORWARD -j ACCEPT
    sudo sh -c "iptables-save > /etc/iptables/rules.v4"
#    sudo apt-get install iptables-persistent -y

    # Configure Tinc VPN for Branch Server
    sudo mkdir -p /etc/tinc/vpn/hosts

    cat <<EOF | sudo tee /etc/tinc/vpn/tinc.conf
Name = branch
AddressFamily = ipv4
Interface = tun0
ConnectTo = main
EOF

    cat <<EOF | sudo tee /etc/tinc/vpn/tinc-up
#!/bin/sh
ifconfig \$INTERFACE 10.0.0.2 netmask 255.255.255.0
EOF

    cat <<EOF | sudo tee /etc/tinc/vpn/tinc-down
#!/bin/sh
ifconfig \$INTERFACE down
EOF

    sudo chmod +x /etc/tinc/vpn/tinc-up /etc/tinc/vpn/tinc-down

    cat <<EOF | sudo tee /etc/tinc/vpn/hosts/branch
Address = 192.168.5.10
Subnet = 10.0.0.2/32
EOF

    sudo tincd -n vpn -K4096
    sudo systemctl enable tinc@vpn
    sudo systemctl start tinc@vpn

# Client Configuration
elif [[ $(hostname) == "client" ]]; then
    # Configure Netplan
    cat <<EOF | sudo tee /etc/netplan/01-netcfg.yaml
network:
  version: 2
  ethernets:
    ens33:
      dhcp4: yes
      routes:
        - to: default
          via: 192.168.5.10
      nameservers:
        addresses:
          - 192.168.5.10
          - 8.8.8.8
EOF

    sudo netplan apply

    # Configure Chrony
    sudo sed -i '/pool /d' /etc/chrony/chrony.conf
    echo "server branch.abc.local iburst" | sudo tee -a /etc/chrony/chrony.conf
    sudo systemctl start chrony
    sudo systemctl enable chrony
fi

# Configure Chrony on Branch Server
if [[ $(hostname) == "branch" ]]; then
    sudo sed -i '/pool /d' /etc/chrony/chrony.conf
    echo "server main.abc.local prefer iburst" | sudo tee -a /etc/chrony/chrony.conf
    sudo systemctl start chrony
    sudo systemctl enable chrony
fi

# Common Configuration
# Set hostname and update /etc/hosts
sudo hostnamectl set-hostname $(hostname).abc.local
cat <<EOF | sudo tee -a /etc/hosts
192.168.1.10 main.abc.local
192.168.5.10 branch.abc.local
192.168.5.15 client.abc.local
EOF

This provisioner will set up BIND9 for both the main and branch servers, configure Tinc VPN, set up DHCP on the branch server, and provide internet access to the client.

5. Start the VMs

vagrant up

6. Verify the VMs

vagrant status

Use vmware workstation to discover and manage the running VMs

• Open vmware workstation, click on File, then select option scan for virtual machines

• Browse for the installation location of the created VMs, follow the instructions on screen and launch it.

Verify the VMs on the VMware workstation

Other vagrant commands

vagrant halt #to halt the VM
vagrant destroy #to remove the VM setup
vagrant validate #to validate the configuration files in the vagrant directory
vagrant status #show status of vagrant VMs

In Conclusion

Completing this scenario will provide you with practical experience in network administration, enabling you to apply the concepts and commands learned to address real-world challenges faced by Cloud Engineers and System Administrators.