Coding it my way

Resize disk in Proxmox VM

2024-11-01T00:00:00+05:30

I needed to resize a disk in my Proxmox VM. Specifically, I was working with a Debian 12 (Bookworm) VM created through Proxmox VE Helper Scripts. These VMs come with a default size of 2 GB, even when using the advanced installation option. Unfortunately, there’s no way to select the disk size during the initial installation process.

Pre-Requisites

First, install the required tools:

apt install parted fdisk

Steps to Resize the disk

Resize the disk through Proxmox GUI.
Login to the VM and run fdisk -l . You should see warnings similar to:

GPT PMBR size mismatch (167772159 != 314572799) will be corrected by write. The backup GPT table is not on the end of the device. This problem will be corrected by write.

Use parted to fix the partition table:

parted /dev/sda

print

You’ll see a warning:

Warning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 146800640 blocks) or continue with the current setting?

Fix the partition table

Fix/Ignore? F

Resize the partition:

(parted) resizepart 1
Warning: Partition /dev/sda1 is being used. Are you sure you want to continue?
Yes/No? y
End? [85.9GB]? 100%
quit

Apply the changes to the filesystem:

resize2fs /dev/sda1

Note: This method works only for ext4 non-LVM filesystems.

s3cmd Fast way to copy s3 data

2024-10-07T00:00:00+05:30

If you are looking to copy huge amount of data to/from AWS S3

Look no further than s5cmd

Download the entire s3 bucket to local

s5cmd cp  s3://bucket/ .

I was able to download almost 10 GB folder in couple of minutes, which is around 83MBps ~ 10 MB/s

Whereas aws s3 cp --recursive s3://bucket/ . was giving me around 500 KB/s

_class attribute in Spring data MongoDB_

2024-10-05T00:00:00+05:30

Spring data MongoDB adds a _class attribute to every document written to MongoDB

If we want to prvent adding Spring Data MongoDB from adding _class attribute, we can configure this behaviour through:

@Configuration
public class MongoConfig extends AbstractMongoClientConfiguration {
    @Override
    protected boolean autoIndexCreation() {
        return true;
    }
    
    @Bean
    public MappingMongoConverter mappingMongoConverter(MongoDatabaseFactory databaseFactory, MongoMappingContext mappingContext) {
        DbRefResolver dbRefResolver = new DefaultDbRefResolver(databaseFactory);
        MappingMongoConverter converter = new MappingMongoConverter(dbRefResolver, mappingContext);
        converter.setTypeMapper(new DefaultMongoTypeMapper(null));
        return converter;
    }
}

Create local kubernetes cluster using Kind

2024-09-01T00:00:00+05:30

If you want to quickly spin up a local development environment of kubernets cluster, Kubernetes in docker aka Kind is the way to go

Kind runs the kubernetes inside a docker container as opposed to Minikube which used to run kubernetes inside a VM

Running Kind on linux requires to create a cluster configuration file cluster.yaml

Here is the sample configuration which would work for most of the cases:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:

  - |
     kind: InitConfiguration
     nodeRegistration:
     kubeletExtraArgs:
     node-labels: "ingress-ready=true"
     extraPortMappings:
     - containerPort: 80
       hostPort: 80
       protocol: TCP
     - containerPort: 443
       hostPort: 443
       protocol: TCP

kind create cluster --config=cluster.yaml

Upon creating the cluster, kubectl context is automatically set to point to kind cluster.

> kubectl cluster-info --context kind-kind
Kubernetes control plane is running at https://127.0.0.1:46251
CoreDNS is running at https://127.0.0.1:46251/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Load testing Websocket servers using Locust

2024-08-04T00:00:00+05:30

The entire code of this article is present on this github repository

Load testing WebSocket servers is crucial for ensuring their performance and scalability.

This article will guide you through using Locust, a popular open-source load testing tool, to test your Java WebSocket server.

We will use Jetty WebSocket in Java to run our websocket server. Jetty WebSocket is a Java library and part of the larger Jetty project, which is a popular open-source web server and servlet container.

The only message we will be handling is heartbeat requests for simplicity.

 private boolean start(String host, int port) {
    try {
        ServerConnector socketConnector = new ServerConnector(server);
        socketConnector.setPort(port);
        socketConnector.setHost(host);
        server.setConnectors(new Connector[] { socketConnector });
        disableServerVersionHeader();
        WebSocketHandler webSocketHandler = new WebSocketHandler() {
            @Override
            public void configure(WebSocketServletFactory webSocketServletFactory) {
                webSocketServletFactory.register(WebSocketChannelHandler.class);
                webSocketServletFactory.getExtensionFactory().unregister("permessage-deflate");
            }
        };
        server.setHandler(webSocketHandler);
        server.start();
        isRunning = server.isStarted();
        logger.info("Websocket Jetty server started @- {}/{}", host, port);

    } catch (Exception e) {
        logger.error("Unable to start the server on {}/{}", host, port, e);
        isRunning = false;
    }
    return isRunning;
}

Here is a simple handler which handles heartbeat requests

@OnWebSocketMessage
public void onMessage(Session session, String message) {
    try {
        logger.info("WS message received : {}", message);
        processMessage(session, message);
    } catch (Exception e) {
        logger.error("Exception in onMessage for session {} message {} due to {}", session, message,
                e);
    }
}

private void processMessage(Session session, String message) {
    JSONObject json = new JSONObject(message);
    String messageType = json.optString("type").toLowerCase();
    processEventsBasedOnMessageType(session, messageType, json);
}

private void processEventsBasedOnMessageType(Session session, String messageType, JSONObject json) {
    switch (messageType) {
        case "heartbeatreq":
            SystemFactory.getInstance().getWebSocketMessageHandler().handleHeartBeatMessages(session);
            break;
        default:
            logger.info("Unknown message type received : {}", messageType);
    }
}

For the load testing part, we will use the popular Locust load testing framework.

We use websocket-client to manage websockets inside python. It is a very elegant library which allows to handle both short lived and long lived websocket connections.

Whenever we send the heartbeat request to server, we send a timestamp from the client side in the field publishTimestamp. The server then responds with the heartbeatresp and also includes the original timestamp publishTimestamp which was sent in the request in the response. The server also includes the serverDelay which represents the delay of server in processing the heartbeat response.

Our contract looks like this:

{
    "type": "heartbeatreq",
    "userId": 1,
    "publishTimestamp": 1722758706760
}


{
    "type": "heartbeatresp",
    "userId": 1,
    "serverDelay": 1,
    "clientPublishTimestamp": 1722758706760
}

We can subtract the current time (the time we received the message in locust) and the clientPublishTimestamp in the heartbeatresp to calculate the end to end time of processing the heartbeat request message.

When running the load test with Locust, it’s important to ensure that Locust itself is not experiencing high CPU usage. Otherwise, the high latency could be attributed to the lag in receiving the messages in the locust itself, rather than the processing time of the server.

We run the load with 10K connected users with ramp up rate of 500. I choose this number because I am running this on my local desktop computer.

This means we will be connecting 500 websockets / s until we reach total 10K connected sockets on the server.

Locust will take some time to spawn all the 10K users. At this point, we can see that rate of ws_connect is 500 per second.

Once the spawn state is finished, all the websockets are connected. At this point, all the connected websockets will be sending heartbeat requests at every 2 seconds, which comes out to ~ 5000 heartbeat requests per second as shown below.

Conclusion

In this article, we saw how to load test WebSocket servers using locust. The benefit of using locust is to have fine grained control over number of sockets to connect, when to trigger messages as well as calculate end to end latency.

The entire code of this article is present on this github repository

Feel free to use this code or if you have any comments, add comment using the comment box below.

Monitoring Redis Streams lag in Prometheus and Grafana

2024-07-14T00:00:00+05:30

Monitoring Redis Streams lag in Prometheus and Grafana

In this post, we will look at how we can measure redis streams lag from the application perspective.

We would want to measure the lag and plot this in visual format using Prometheus in Grafana

Assumptions

We are going to assume that we have set of streams and set of consumer groups which are consuming from the stream. We are using consumer groups because we want high availability to consume via multiple nodes. We assume that each consumer within consumer groups are dynamic, for example if a consumer is running on a kubernetes node, a node can be restarted, respawned.
We are also going to assume that we are trimming stream length to some constant number of messages at regular intervals to save memory. This is important in terms of monitoring since redis reports some metrics assuming that consumer group has been consuming from beginning of stream, which many a times is not the case.
We are also going to assume that there is some constant minimum rate of message publish on redis stream. Why do we need this assumption? Not enforcing this restriction makes the monitoring even more difficult. We will see why below.
In the examples below, we have streams named ack0, ack1, ack2, ack3, ack4 consumed by consumer group poker-coordinator-ack

How do we define lag?

In terms of Redis streams, we are primarily interested in two metrics:

Undelivered messages: The messages which have been published on redis stream, but not yet have been consumed by the consumer group. This can happen if consumer group is taking a lot of time to process messages. This indicates that either we need to scale horizontally to increase the consumption rate or reduce the publish rate.
Pending messages: The message which have been consumed by the consumer group but yet not acknowledged. This is only applicable for redis streams consumer groups which have acknowledgement enabled. The messages which are not acknowledged, remain in Pending Entries List data structure. It is important to know the length of this data structure to determine if there were some messages in stream which were not acknowledged by the consumer.

We will now examine each of the redis stream related command output below to determine if it can help in monitoring redis streams.

xinfo groups ack0, will give information about ack0 stream consumer groups.

127.0.0.1:6381> xinfo groups ack3
1)  1) "name"
    2) "consumerGroup"
    3) "consumers"
    4) (integer) 2
    5) "pending"
    6) (integer) 0
    7) "last-delivered-id"
    8) "0-0"
    9) "entries-read"
   10) (nil)
   11) "lag"
   12) (integer) 8766
2)  1) "name"
    2) "poker-coordinator-ack"
    3) "consumers"
    4) (integer) 20
    5) "pending"
    6) (integer) 100
    7) "last-delivered-id"
    8) "1720934812760-0"
    9) "entries-read"
   10) (integer) 791553
   11) "lag"
   12) (integer) 11445

This output gives the number of consumer groups for stream ack3. Here, stream ack3 has two consumer groups namely consumerGroup and poker-coordinator-ack.

The number of consumers in both groups are 2 and 20 respectively.
The pending messages in both groups are 0 and 100 respectively.
The last delivered id for both groups is 0-0 and 1720934812760-0 respectively. The first one indicates that its dead and is not running. This metric is useful in monitoring since it can tell us which was the last id which was delivered to this consumer group by redis stream.
The entries read for both groups are nil and 791553 respectively. This number is lifetime aggregate and is usually useless for monitoring purposes. This is because stream length can generally be trimmed at regular intervals for saving memory as I mentioned in the assumptions. Once the stream length is trimmed, this number in comparison to stream length is very large.
The lag for both groups is 8766 and 11445 respectively. The lag is important metric, however its important to understand it.

The lag of a given consumer group is the number of entries in the range between the group’s entries_read and the stream’s entries_added. Put differently, it is the number of entries that are yet to be delivered to the group’s consumers.

It seems that this number is a good metric to monitor for monitoring. However, there is a caveat to it. The way redis calculates this number is by subtracting the total number of messages published on stream minus the number of messages read by consumer group since beginning. In real world, the consumer group might have been created later after the stream is created OR there might have been some messages which were trimmed or deleted from the stream which this consumer group might never receieve. In that case, this number will always report very large value which is not what we want.

xpending ack3 poker-coordinator-ack will give the information about pending messages to the consumer group

127.0.0.1:6379> xpending ack3 poker-coordinator-ack
-> Redirected to slot [15549] located at 127.0.0.1:6381
1) (integer) 100
2) "1720937728800-4"
3) "1720937738803-1"
4) 1) 1) "F8DD1276E0DF496C965EC775CA07397F"
      2) "100"

This outputs that there are 100 pending messages. The smallest message id is 1720937728800-4 and largest message id is 1720937738803-1. The consumer (within the group poker-coordinator-ack) which has pending messages 100 with id F8DD1276E0DF496C965EC775CA07397F

xinfo consumers ack3 poker-coordinator-ack

 1) 1) "name"
    2) "07A1A45C9BCD40238A54F773ECB47B82"
    3) "pending"
    4) (integer) 0
    5) "idle"
    6) (integer) 1463079
    7) "inactive"
    8) (integer) 1463079
 2) 1) "name"
    2) "225EF592894744ADB4F0768797F608CE"
    3) "pending"
    4) (integer) 0
    5) "idle"
    6) (integer) 10170676
    7) "inactive"
    8) (integer) 10170676

The output above gives information about each unique consumer within a consumer group. Since our consumers are dynamic in a containerized infrastructure, the consumer group name is just a UUID. Once the container goes away, the unique consumer name is no longer relevant and we are not interested in the information. It is apparent that we are not interested in each consumer within a group. We want to make sure that a consumer group as a whole is healthy and consuming messages at healthy rate.

Calculating undelivered messages

The best way to calculate undelivered message is get the last delivered ID of the consumer group and the last delivered ID of the stream. Count the number of messages between them and report it as lag. Here is the pseudo code using Redisson client in java.

import org.redisson.api.StreamMessageId;
import org.redisson.api.stream.StreamReadGroupArgs;

public static void updateStreamLagMetrics(RedissonClient redisson, String streamName, String groupName) {
    RStream<String, String> stream = redisson.getStream(streamName);
    
    // Get stream info
    Map<String, Object> streamInfo = stream.getInfo();
    String lastGeneratedId = (String) streamInfo.get("lastGeneratedId");

    // Get group info
    Map<String, Object> groupInfo = stream.listGroups().stream()
            .filter(group -> group.get("name").equals(groupName))
            .findFirst()
            .orElseThrow(() -> new RuntimeException("Group not found"));

    String lastDeliveredId = (String) groupInfo.get("lastDeliveredId");

    // Get pending messages count
    PendingResult pendingInfo = stream.getPendingInfo(groupName);
    long pendingCount = pendingInfo.getTotal();

    // Calculate undelivered messages
    long undeliveredMessages = 0;
    if (!lastGeneratedId.equals(lastDeliveredId)) {
        StreamMessageId lastDeliveredStreamId = parseStreamMessageId(lastDeliveredId);
        
        // Read all messages after the last delivered ID
        List<StreamEntry<String, String>> undeliveredEntries = stream.readGroup(groupName, "temp-consumer", 
            StreamReadGroupArgs.neverDelivered().count(Integer.MAX_VALUE));
        
        undeliveredMessages = undeliveredEntries.size();
        
        // Remove the temporary consumer
        stream.removeConsumer(groupName, "temp-consumer");
    }

    long totalLag = undeliveredMessages + pendingCount;

    // Update Prometheus metrics
    UNDELIVERED_MESSAGES.labels(streamName, groupName).set(undeliveredMessages);
    PENDING_MESSAGES.labels(streamName, groupName).set(pendingCount);
    TOTAL_LAG.labels(streamName, groupName).set(totalLag);
}

private static StreamMessageId parseStreamMessageId(String id) {
    String[] parts = id.split("-");
    if (parts.length != 2) {
        throw new IllegalArgumentException("Invalid stream ID format: " + id);
    }
    return new StreamMessageId(Long.parseLong(parts[0]), Long.parseLong(parts[1]));
}

However, note that counting the number of messages between two IDs is a very expensive operation which I do not recommend doing on a production cluster. Instead, what we can do is get the approximation of time difference between the two and report it as a metric. This is where the assumption that there is some constant minimum rate of message publish on redis stream. comes into picture. If the messages are sparingly published, this approach does not work.

Here is the updated code based on the assumption

    public  void computeStreamLag(RedissonClient redisson, String streamName, String groupName) {
        RStream<String, String> stream = redisson.getStream(streamName);

        // Get stream info
        StreamInfo<String, String> streamInfo = stream.getInfo();

        StreamMessageId lastGeneratedId = streamInfo.getLastGeneratedId();

        // Get group info
        List<StreamGroup> groupsInfo = stream.listGroups();
        StreamGroup groupInfo = groupsInfo.stream()
                .filter(group -> group.getName().equals(groupName))
                .findFirst()
                .orElseThrow(() -> new RuntimeException("Group not found"));

        StreamMessageId lastDeliveredId = groupInfo.getLastDeliveredId();
        // Get pending messages count
        PendingResult pendingInfo = stream.getPendingInfo(groupName);
        long pendingCount = pendingInfo.getTotal();

        // Calculate lag
        // Calculate undelivered messages
        long undeliveredMessages = 0;
        if (!lastGeneratedId.equals(lastDeliveredId)) {
            // If last generated ID is different from last delivered ID, there are undelivered messages
            undeliveredMessages = calculateIdDifference(lastGeneratedId, lastDeliveredId);
        }

        if (undeliveredMessages < 0) {
            undeliveredMessages = 0;
        }

         List<Tag> undeliveredMessagesTag =
                new ArrayList<>(Collections.singleton(Tag.of("type", "undeliveredMessages")));
        List<Tag> pendingMessages =
                new ArrayList<>(Collections.singleton(Tag.of("type", "pendingMessages")));

        this.prometheusMetricWrapper.recordGaugeValue(streamName, undeliveredMessagesTag, undeliveredMessages);
        this.prometheusMetricWrapper.recordGaugeValue(streamName, pendingMessages, pendingCount);
    }

    private static long calculateIdDifference(StreamMessageId id1, StreamMessageId id2) {
        long timeDiff = id1.getId0() - id2.getId0();
        long seqDiff = id1.getId1() - id2.getId1();

        if (timeDiff == 0) {
            return seqDiff;
        } else {
            // This is an approximation. Actual difference might be slightly different
            return (timeDiff) + seqDiff;
        }
    }

    private static StreamMessageId parseStreamMessageId(String id) {
        String[] parts = id.split("-");
        if (parts.length != 2) {
            throw new IllegalArgumentException("Invalid stream ID format: " + id);
        }
        return new StreamMessageId(Long.parseLong(parts[0]), Long.parseLong(parts[1]));
    }

What about measuring reliable processing Lag

We saw above that it is not possible to measure the redis streams lag (measured as count of mesasges) accurately if the rate of publish on the stream varies a lot.

How did we end up solving this?

One of the other ways of solving is to let publisher add the timestamp to the each of stream message. This way, when the message arrives at the consumer, you can calculate the difference and plot it as processing lag.

Conclusion

In this article we saw how to monitor redis streams lag. Here is the final snapshot of the dashboard which came out as a result of this excercise.

Faking time in JVM Process

2024-07-01T00:00:00+05:30

Recently, we were doing automation testing for one of our backend services and encountered an unique scenario wherein we wanted to forward the time by 24 hrs because we had entities which were created on daily basis at certain time.

To automate these scenarios, we had to wait for 24 unless we did either of these things:

Somehow create those entities every few minutes instead of every 24 hours - This would require making the implementation generic for both scenarios.
Fast forward the system time - Since we were running in containers and containers inherit the date time settings of host machine, we had to change the time of host machine. However, that means lot of other containers running in the same machine getting affected by these side effect.

We did not go with first option either, since there were lot of backend validations implemented and removing those validations means making the implementation susceptible to errors.

We looked through many solutions and finally arrived by using a library called faketime

In short,

FakeTime uses a native Java agent to replace System.currentTimeMillis() implementation with the one you can control using system properties.

Getting started with Faketime

Faketime is implemented as a native agent and hence it needs to be built for every platform such as Linux, Mac OS x86, Mac M1 etc.

Fortunately, there exists maven modules for each of the architectures

<!-- Windows 32bit -->
<dependency>
  <groupId>io.github.faketime-java</groupId>
  <artifactId>faketime-agent</artifactId>
  <version>0.8.0</version>
  <classifier>windows32</classifier>
</dependency>

<!-- Windows 64bit -->
<dependency>
  <groupId>io.github.faketime-java</groupId>
  <artifactId>faketime-agent</artifactId>
  <version>0.8.0</version>
  <classifier>windows64</classifier>
</dependency>

<!-- macOS 32bit -->
<dependency>
  <groupId>io.github.faketime-java</groupId>
  <artifactId>faketime-agent</artifactId>
  <version>0.8.0</version>
  <classifier>mac32</classifier>
</dependency>

<!-- macOS 64bit -->
<dependency>
  <groupId>io.github.faketime-java</groupId>
  <artifactId>faketime-agent</artifactId>
  <version>0.8.0</version>
  <classifier>mac64</classifier>
</dependency>

<!-- Linux 32bit -->
<dependency>
  <groupId>io.github.faketime-java</groupId>
  <artifactId>faketime-agent</artifactId>
  <version>0.8.0</version>
  <classifier>linux32</classifier>
</dependency>

<!-- Linux 64bit -->
<dependency>
  <groupId>io.github.faketime-java</groupId>
  <artifactId>faketime-agent</artifactId>
  <version>0.8.0</version>
  <classifier>linux64</classifier>
</dependency>

I have implemented a small github project to demonstrate faking the time using this library.

Here is how you would run it…

Run using Maven

mvn spring-boot:run -Dspring-boot.run.jvmArguments="-agentpath:./src/main/resources/libfaketime -XX:+UnlockDiagnosticVMOptions -XX:DisableIntrinsic=_currentTimeMillis -XX:CompileCommand=quiet -XX:CompileCommand=exclude,java/lang/System.currentTimeMillis -XX:CompileCommand=exclude,jdk/internal/misc/VM.getNanoTimeAdjustment"

Get current time

curl --location 'localhost:8080/time/getTime'

{
    "localDateTime": "2024-06-29T15:42:41.973356",
    "timestamp": 1719655961973,
    "instant": "2024-06-29T10:12:41.973377Z",
    "date": "2024-06-29T10:12:41.973+00:00"
}

Set fake time

curl --location --request POST 'localhost:8080/time/setTime?dateTime=2024-08-01T00%3A00%3A00'

Time set

Get current time

curl --location 'localhost:8080/time/getTime'

{
    "localDateTime": "2024-08-01T00:00:35.085",
    "timestamp": 1722450635085,
    "instant": "2024-07-31T18:30:35.085Z",
    "date": "2024-07-31T18:30:35.085+00:00"
}

Reset back to real time

curl --location --request POST 'localhost:8080/time/resetTime'

Rtkit Daemon flooding journalctl logs

2024-06-22T00:00:00+05:30

Recently I observed, Rtkit daemon has been flooding the Journalctl logs

I frequently found logs such as below in the journalctl logs

Jun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users.
Jun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users.
Jun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users.
Jun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users.
...
Jun 12 23:48:39 madhur-b550mds3h rtkit-daemon[1942]: Failed to look up client: No such file or directory
Jun 12 23:48:42 madhur-b550mds3h rtkit-daemon[1942]: Failed to look up client: No such file or directory
Jun 12 23:48:44 madhur-b550mds3h rtkit-daemon[1942]: Failed to look up client: No such file or directory

Not much aware about this daemon, however, it seems to be used by audio systems such as Pulseaudio and Pipewire

Fixing these logs is simple

systemctl edit rtkit-daemon

Add the following lines

[Service]
LogLevelMax=warning

And restart rtkit daemon using systemctl restart rtkit-daemon

My Homelab Update - Part I

2024-05-01T00:00:00+05:30

Continuing my Homelab journey using Proxmox, I have setup a full fledged home network with multiple services running.

Here is the quick diagram below for reference:

Some of the services I am running on my homenetwork as of now are

Some of the things I am trying to solve:

Easily able to VPN from outside to access my password manager, RSS feeds, media etc
Get rid of PfSense, PfSense is good but I feel its overkill for my needs.
Evaluate Tailscale, Wireguard and OpenVPN

And finally,

Having less power bill while powering my Proxmox server 24x7 :)

Stay tuned for further updates if you are interested.

Quick Websockets client in Python

2024-04-13T00:00:00+05:30

When interacting with lot of websockets server, you need a nifty client to test out the behaviour of websocket server.

Till now, I have been using wscat as the command line utility to interact with websockets server.

However, this command line utility has several disadvantages:

It doesn’t allow you to send custom ping messages
It doesn’t support authentication

Due to these limitations, I wrote a simple python script below which can send a custom ping payload.

This little script helps to quickly test out websocket servers.

import websocket
import json

session_id="12345"

ready_message = {"type":"setup","sid":session_id,"time":1712393684026, "src": "79677227"} 
ping_payload={"type":"heartbeatreq","time":1712393714042,"src":"79677227"}

def on_message(wsapp, message):
    print("Received: ", message)

def on_ping(wsapp, message):
    print("Got a ping! A pong reply has already been automatically sent.")

def on_pong(wsapp, message):
    wsapp.send(message)

def on_open(wsapp):
    print("sending ready")
    ready_msg = json.dumps(ready_message) + "\n"
    wsapp.send(ready_msg)

def on_error(wsapp, message):
    print(message)

def on_close(wsapp, close_status_code, close_reason):
    print("closed", close_reason, close_status_code)

wsapp = websocket.WebSocketApp("ws://localhost:8080",
on_message=on_message, on_ping=on_ping, on_pong=on_pong, on_open=on_open, on_error=on_error, on_close=on_close)
wsapp.run_forever(ping_interval=2, ping_timeout=1, ping_payload=json.dumps(ping_payload)+ "\n")  

Building RediSearch module in Amazon Linux 2

2024-03-08T00:00:00+05:30

RediSearch is a Redis module that provides querying, secondary indexing, and full-text search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch query language to query that data.

There is a very good RediSearch getting started tutorial on github.

There are several steps in bulding RediSearch module. The following steps worked on Amazon Linux 2 machine for us:

yum install git perl perl-DateTime perl-JSON perl-Capture-Tiny
git clone https://github.com/RediSearch/RediSearch.git
wget https://github.com/linux-test-project/lcov/releases/download/v2.0/lcov-2.0-1.noarch.rpm
rpm -ivh /root/lcov-2.0-1.noarch.rpm
cd RediSearch
make setup
make build

At this point, the module should be built at the following location

./RediSearch/bin/linux-x64-release/search/redisearch.so

Copy it to /etc/redis

cp /root/RediSearch/bin/linux-x64-release/search/redisearch.so  /etc/redis/

The module can be activated in redis conf as follows:

loadmodule /etc/redis/redisearch.so

Websocket - Close Status codes

2024-02-11T00:00:00+05:30

A table for reference

Close code (uint16)	Codename	Internal	Customizable	Description
0 - 999		Yes	No	Unused
1000	`CLOSE_NORMAL`	No	No	Successful operation / regular socket shutdown
1001	`CLOSE_GOING_AWAY`	No	No	Client is leaving (browser tab closing)
1002	`CLOSE_PROTOCOL_ERROR`	Yes	No	Endpoint received a malformed frame
1003	`CLOSE_UNSUPPORTED`	Yes	No	Endpoint received an unsupported frame (e.g. binary-only endpoint received text frame)
1004		Yes	No	Reserved
1005	`CLOSED_NO_STATUS`	Yes	No	Expected close status, received none
1006	`CLOSE_ABNORMAL`	Yes	No	No close code frame has been receieved
1007	Unsupported payload	Yes	No	Endpoint received inconsistent message (e.g. malformed UTF-8)
1008	Policy violation	No	No	Generic code used for situations other than 1003 and 1009
1009	`CLOSE_TOO_LARGE`	No	No	Endpoint won’t process large frame
1010	Mandatory extension	No	No	Client wanted an extension which server did not negotiate
1011	Server error	No	No	Internal server error while operating
1012	Service restart	No	No	Server/service is restarting
1013	Try again later	No	No	Temporary server condition forced blocking client’s request
1014	Bad gateway	No	No	Server acting as gateway received an invalid response
1015	TLS handshake fail	Yes	No	Transport Layer Security handshake failure
1016 - 1999		Yes	No	Reserved for later
2000 - 2999		Yes	Yes	Reserved for websocket extensions
3000 - 3999		No	Yes	Registered first come first serve at IANA
4000 - 4999		No	Yes	Available for applications

Proxmox - Setting up user friendly urls

2024-01-07T00:00:00+05:30

I have been recently tinkering up with idea of setting up a homelab and self hosting few services such as DNS Server, Media server, RSS Aggregator etc along with some standard monitoring tools.

I evaluated two products, Proxmox and Unraid

After evaluating both, I choose Proxmox primarily because it seemed to fit my needs more than Unraid. Unraid is primarily useful if you have lot of data which needs to be made available to multiple services.

Whereas, Proxmox seems more suitable to run independent services altogether, either in [Docker](https://en.wikipedia.org/wiki/Docker_(software), VMs or Linux Containers

For me, the size of the data was not going to be very much, atleast initially and I found the idea of running an open source software (Proxmox) more appealing than one I had to purchase (Unraid).

I had a spare 12 CORE / 32 GB machine on which I quickly setup Proxmox and I must say, I really like the way I was quickly able to run the following services in a day:

Jellyfin - Media server
Commafeed - Open source RSS aggregator written in Java
PiHole - DNS Server with Ad Blocking
qbittorrent - Torrent client
Nginx Proxy Manager - A simple reverse proxy
Prometheus - Well known time series database
Grafana - Well known visualization platform

The cool thing about Proxmox is that it uses a Type 1 Hypervisor called Kernel-based Virtual Machine (KVM) to run all the above mentioned services in Linux Containers running on Debian Linux OS

What that essentially means is that all these services get their own unique IP Addresses on my LAN network as opposed to all running on a single IP Addresses and exposing on different ports. Type 1 Hypervisor means that they are able to directly access machine hardware and do not have the any additional layer or overhead of resource hungry VM’s.

Each LXC takes around 200 - 300 MB of disk space and around 512 MB of RAM , making them all fit in a single commodity machine.

One caveat of this approach is that now I have tons of IP addresses and ports for me to memorize to access these services when I require. Though I can bookmark them, I do not find it appealing.

Hence, I decided to setup a reverse proxy in nginx for each of them in order for me to reach them through a very user friendly url. Here is the screenshot of the setup:

The above setup requires some manual setup of setting up a /etc/hosts file with appropriate entries as follows

# Hosts for nginx proxy manager
168.1.133 nginx
168.1.133 plexui
168.1.133 tui
168.1.133 jellyfinui
168.1.133 grafanaui
168.1.133 promui
168.1.133 zkui
168.1.133 piholeui
168.1.133 rssui

Here 192.168.1.133 is the IP Address of Nginx Proxy Manager

With the above setup I can reach my services in friendly urls such as:

http://plexui, http://tui, http://jellyfinui, http://grafanaui etc.

Using Pihole for DNS Records instead of static host file

One might argue that these hosts entries need to be done in each individiaul client that needs to access Proxmox containers and is not a scaleable approach.

That is correct, the reason I went with hosts file approach is that right now, I only have a single client - my desktop PC and I do not foresee running PiHole 24x7.

Incase, I choose PiHole as my permanent DNS resolver and AdBlocker, the scaleable way would be to add these entries in PiHole itself so that it becomes available to any client within my network.

Using Kitty hints to parse maven output

2023-12-28T00:00:00+05:30

Kitty is a very fast and feature rich terminal which has soon become my favourite.

One of the powerful features of Kitty is its customizability.

One such feature is Kitty hints

Kitty hints allows you to select text on terminal and perform an action through a shortcut key. It is completely customizeable.

Common kitty hints action include opening files, folders and opening web urls outputted in terminal.

One example where, I was able to customize it was to parse maven output and makes the file links clickable to land directly on the line numbers.

For example, consider this build output:

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.618 s
[INFO] Finished at: 2023-12-28T10:08:38+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project reactive-programming: Compilation failure: Compilation failure: 
[ERROR] /run/media/madhur/centos/home/madhur/github/personal/reactive-programming/src/main/java/in/co/madhur/mono/FileService.java:[17,9] invalid method declaration; return type required
[ERROR] /run/media/madhur/centos/home/madhur/github/personal/reactive-programming/src/main/java/in/co/madhur/mono/FileService.java:[17,20] illegal start of type
[ERROR] /run/media/madhur/centos/home/madhur/github/personal/reactive-programming/src/main/java/in/co/madhur/mono/Lec08MonoRunnable.java:[15,54] ';' expected
[ERROR] /run/media/madhur/centos/home/madhur/github/personal/reactive-programming/src/main/java/in/co/madhur/mono/Lec06SupplierRefactoring.java:[25,37] ';' expected
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Using kitty hints, we can make the file paths to FileService.java , Lec08MonoRunnable.java and Lec06SupplierRefactoring.java clickable and land it directly on appropriate line numbers.

This required writing custom regex to parse this line and emitting two named capturing groups path and line.

This can be done in kitty.conf as :

map kitty_mod+p kitten hints --type=regex --regex="(?<path>(?:\/[\w-_^ ]+)+\/?(?:[\w.])+[^.]):\[(?<line>\d+),\d+\].?" --program "launch /home/madhur/scripts/editor.py"

The editor.py is a simple program which is invoked with parameters path=...../FileService.java and line=17

In my case, I launch Neovim or IntelliJ depending upon if Intellij is previously running.

This can be simply done through following snippet:

#!/usr/bin/env python3

import sys
import os
import subprocess

myObject = {}
for line in sys.argv:
    if '=' not in line:
        continue
    print(line)
    key, value = line.rstrip("\n").split("=")
    myObject[key] = value

print(myObject)

idea_running = False
pl = subprocess.Popen(['ps', 'aux'], stdout=subprocess.PIPE).communicate()[0]
list_ps = pl.decode('utf-8')

if 'intellij' in list_ps:
    idea_running = True

if idea_running is True and 'java' in myObject['path']:
    print("Intellij found")
    os.system("flatpak run com.jetbrains.IntelliJ-IDEA-Community " + myObject['path'])
else:
    print("Intellij not found")
    os.system("nvim " + " +"+myObject['line'] + " " + myObject['path'])

Android SSL Pinning Bypass

2023-12-05T00:00:00+05:30

I have always been a tinkerer and loved to explore internal of applications.

Android has always fascinated me from the beginning partly because its mostly open source and allows you to customize. Other aspect of Android which has been useful is Rooting

Rooting allows you to take complete control over the application and customize some of the aspects which are not possible in non-rooted Android.

One area where rooting helps is capturing SSL network traffic of the applications. This helps in debugging and learning more about how networks works. In this post, we will see how can we capture the network traffic for inspection in Android. This is mostly for debugging and learning purpose.

Apk-mitm

APK mitm tools modified the original Android application to disable the SSL Pinning. This tool relies on apktool heavily.

This tool always doesn’t work. Recently there has been lot of issues which have no solutions.

However, if it works, this is the most easiest and straightforward way as the modified app can be installed on non-rooted device and just inspected through proxy such as Charles

Android Unpinner

This tool is different from Apk Mitm in a way that it bypasses the SSL pinning code only at runtime. Thus, you no longer have to disassemble and repackage the application.

This tool relies heavily on Frida tools for its job.

Again, this tool is not gauranteed to work 100% of the time.

Rooting

If you can root your device, it is very easy to capture network traffic even if its from SSL pinned app. However, rooting can destroy warranty.

These days, most of the Android apps, will refuse to start if they detect that they are running on a rooted software.

Hence, this option is only viable if you have a spare device.

Rooted AVD

This is my most preferred method and mostly always works. Since the purpose of network inspection is mostly debugging and learning, we can do it on AVD instead of actual device.

rootAVD allows you to seemlessly root the running AVD device, using the simple command

./rootAVD.sh system-images/android-33/google_apis_playstore/x86_64/ramdisk.img

where system-images/android-33/google_apis_playstore/x86_64/ramdisk.img is the path of AVD devices relative to $ANDROID_HOME

Next step, would be to install MagiskTrustUserCerts module of Magisk

This module makes all installed user certificates part of the system certificate store, so that they will automatically be used when building the trust chain. This module makes it unnecessary to add the network_security_config property to an application’s manifest.

Once installed, your Magisk modules section should look like this

Now, the only step remaining is enabling proxy to intercept traffic. I use Charles proxy and this opens the port 8888 for listening to traffic.

Enabling proxy on AVD is as simple as executing:

adb shell settings put global http_proxy 192.168.1.252:8888

where 192.168.1.252 is the local IP address of the host machine.

Disabling proxy can be done using:

adb shell settings put global http_proxy :0

If you guys have been using more preferred way to inspect SSL traffic. Let me know in the comments.