Skip to content

Latest commit

 

History

History
288 lines (229 loc) · 10.3 KB

README.md

File metadata and controls

288 lines (229 loc) · 10.3 KB
Raw

go-read-var-log

REST /var/log reader

Contents

Assumptions

Some assumptions could be considered TODO items for future enhancement

  • The log files are in /var/log
  • The log files are uncompressed plain text files
    • Compressed or archived files (e.g. .gz, bz2, .tar, .tgz) are not supported
  • The log files are sorted by timestamp (oldest first) as it is captured in each log event
  • Log events are separated by a newline character
  • REST response Content-Type is text/plain; charset=utf-8, regardless of the Accept request header
  • All errors are returned to REST callers as HTTP status 500, even if these might be correctable by the caller
  • LRU result caching of compiled regexes and search results are not implemented
    • Search result caching would require invalidation when the file changes
    • Hot searches could have dedicated caches that are eagerly refreshed when the file changes
  • Testing coverage should be added for the controller package
  • Add GOMAXPROCS to Dockerfile or set that in the code
  • There is a get-large-log branch where I hope to experiment with a more efficient way to read very large files, but it is not yet complete and is not formally part of this submission

Endpoints

  • GET /api/v1/logs - Get list of all log files in /var/log that this service can read
  • GET /api/v1/logs/{log} - Get the contents of the log file specified by {log} in /var/log (e.g. /api/v1/logs/system.log)
    Query Parameters:
    • n - Number of lines to return (default: 25, all: 0)
    • q - Case sensitive fixed text filter to apply to each line (default: ""), more performant than r
    • r - Regex filter to apply to each line (default: .*)
    • Note: q and r may both be specified, but at most once each. Multiple q or r parameters are ignored.

Enabling Tech Stack

  • Go - Programming language
    • If not installed, consider: brew install golang or similar as appropriate for how you manage packages for development on your OS
    • Installation is not required if running in Docker
  • HttpRouter - HTTP request router

Design

  • The service package contains the logic of the service, agnostic of presentation and transport
  • The controller package contains the HTTP request handlers

Generation of Sample Input Log Files

Optionally, arbitrarily large fake log files can be generated in json or delimited format to augment any that are naturally in /var/log.

Arguments

Argument Description
-h | --help Print the help message
-l Large File Bytes (default 10,000,000)
-lt Log with timestamps (UTC)
-n Number of lines to return (default 25)
-p HTTP Port to listen on (default 8080)

Running in Docker

  • docker build -t go-read-var-log . - Build the Docker image
  • docker run -v /var/log:/var/log:ro -p 80:8080 go-read-var-log <args> - Run with the given arguments
    Prevents writing to /var/log. When run without <args>, this enables the following from the docker host:
    • curl "localhost/api/v1/logs"
    • curl "localhost/api/v1/logs/system.log?n=10&q=syslog"

Running Locally

  • go run ./main <args> - Run the main.go file with the given arguments
  • IMPORTANT: Improve scalability with GOMAXPROCS: GOMAXPROCS=8 go run ./main <args>

Run Unit Tests Locally

Tests are implemented in *_test.go files in the same package as the code under test. Test data files are stored under testdata directories, which are ignored by go build and go run.

  • go test ./... -count=-1 - Run all unit tests
  • go test ./... -count=-1 -cover - Run all unit tests with coverage
    • Shows 83% coverage of the service package

Sample Output

Get list of all log files in /var/log

❯ curl "localhost/api/v1/logs"
1GB-9million.log
daily.out
displaypolicyd.stdout.log
fct_uninstall.log
fctinstallpost.log
fsck_apfs.log
fsck_apfs_error.log
fsck_hfs.log
install.log
monthly.out
output.log
shutdown_monitor.log
system.log
weekly.out
wifi.log

Query the contents of a 1GB file with 9.2 million logs

/var/log/1GB-9million.log was generated with GitHub mikebd/go-make-log.

❯ go run ./main -p 80
...
logs.go:30: /api/v1/logs/1GB-9million.log?q=error&r=\sfecig$ returned 1 lines in 2.75945789s
logs.go:30: /api/v1/logs/1GB-9million.log?q=error&r=\sfecig$ returned 1 lines in 2.534780081s
logs.go:30: /api/v1/logs/1GB-9million.log?q=error&r=\sfecig$ returned 1 lines in 2.148826909s
logs.go:30: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$ returned 1 lines in 1.427393941s
logs.go:30: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$ returned 1 lines in 1.447307855s
logs.go:30: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$ returned 1 lines in 1.421464319s

☝️ was generated by the following curl commands:

❯ curl "localhost/api/v1/logs/1GB-9million.log?q=error&r=\sfecig$" 
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=error&r=\sfecig$" 
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=error&r=\sfecig$" 
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$" 
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$" 
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$"

The curl output for all of the above is:

2023-10-10T09:27:32.029729Z|error|tesak awehit atep itego anetar utiyav oset h om tulad palopi ac arinem eralo fecig

Benchmark with Apache Bench

Apache Bench is a simple HTTP load testing tool.

Small file

/var/log/daily.out is a 419KB file with 6,892 lines.

❯ ab -c 6 -k -n 10000 'localhost/api/v1/logs/daily.out'

Document Path:          /api/v1/logs/daily.out
Document Length:        1765 bytes

Concurrency Level:      6
Time taken for tests:   3.193 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      19080000 bytes
HTML transferred:       17650000 bytes
Requests per second:    3131.67 [#/sec] (mean)
Time per request:       1.916 [ms] (mean)
Time per request:       0.319 [ms] (mean, across all concurrent requests)
Transfer rate:          5835.18 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     1    2   0.5      2       5
Waiting:        0    2   0.5      2       5
Total:          1    2   0.5      2       5

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      2
  90%      3
  95%      3
  98%      3
  99%      3
 100%      5 (longest request)

Large file with fixed text and regex matching

Non-default GOMAXPROCS=8

When run for 10,000 requests, it failed somewhere between 4,000-5,000 requests with a timeout as response times escalated to over 1 minute.

For > 2,000 requests (started as 10,000 but interrupted after 2,000):

❯ ab -c 6 -k -n 10000 'localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$'

Document Path:          /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$
Document Length:        117 bytes

Concurrency Level:      6
Time taken for tests:   1036.804 seconds
Complete requests:      2105
Failed requests:        0
Keep-Alive requests:    2105
Total transferred:      545195 bytes
HTML transferred:       246285 bytes
Requests per second:    2.03 [#/sec] (mean)
Time per request:       2955.261 [ms] (mean)
Time per request:       492.543 [ms] (mean, across all concurrent requests)
Transfer rate:          0.51 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:  1905 2946 1771.9   2766   36280
Waiting:     1905 2946 1771.9   2766   36280
Total:       1905 2946 1771.9   2766   36280

Percentage of the requests served within a certain time (ms)
  50%   2766
  66%   2811
  75%   2853
  80%   2885
  90%   3007
  95%   3225
  98%   4481
  99%   6722
 100%  36280 (longest request)

Default GOMAXPROCS=1

This fails for -n 10000 with a timeout, but "succeeds" (with poor performance) for -n 50. There is a lot of room for optimization here if this is a use case that must be supported.

❯ ab -c 6 -k -n 50 'localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$'

Document Path:          /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$
Document Length:        117 bytes

Concurrency Level:      6
Time taken for tests:   31.893 seconds
Complete requests:      50
Failed requests:        0
Keep-Alive requests:    50
Total transferred:      12950 bytes
HTML transferred:       5850 bytes
Requests per second:    1.57 [#/sec] (mean)
Time per request:       3827.187 [ms] (mean)
Time per request:       637.865 [ms] (mean, across all concurrent requests)
Transfer rate:          0.40 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:  1832 3469 1508.0   2750    6374
Waiting:     1832 3469 1508.0   2750    6374
Total:       1832 3469 1508.0   2750    6374

Percentage of the requests served within a certain time (ms)
  50%   2750
  66%   3738
  75%   5137
  80%   5368
  90%   5905
  95%   6206
  98%   6374
  99%   6374
 100%   6374 (longest request)