REST /var/log reader
- Assumptions
- Endpoints
- Enabling Tech Stack
- Design
- Generation of Sample Input Log Files
- Arguments
- Running in Docker
- Running Locally
- Run Unit Tests Locally
- Sample Output
- Benchmark with Apache Bench
Some assumptions could be considered TODO items for future enhancement
- The log files are in
/var/log - The log files are uncompressed plain text files
- Compressed or archived files (e.g.
.gz,bz2,.tar,.tgz) are not supported
- Compressed or archived files (e.g.
- The log files are sorted by timestamp (oldest first) as it is captured in each log event
- Log events are separated by a newline character
- REST response Content-Type is text/plain; charset=utf-8, regardless of the Accept request header
- All errors are returned to REST callers as HTTP status 500, even if these might be correctable by the caller
- LRU result caching of compiled regexes and search results are not implemented
- Search result caching would require invalidation when the file changes
- Hot searches could have dedicated caches that are eagerly refreshed when the file changes
- Testing coverage should be added for the
controllerpackage - Add
GOMAXPROCSto Dockerfile or set that in the code
GET /api/v1/logs- Get list of all log files in/var/logthat this service can readGET /api/v1/logs/{log}- Get the contents of the log file specified by{log}in/var/log(e.g./api/v1/logs/system.log)
Query Parameters:n- Number of lines to return (default: 25, all: 0)q- Case sensitive fixed text filter to apply to each line (default:""), more performant thanrr- Regex filter to apply to each line (default:.*)- Note:
qandrmay both be specified, but at most once each. Multipleqorrparameters are ignored.
- Go - Programming language
- If not installed, consider:
brew install golangor similar as appropriate for how you manage packages for development on your OS - Installation is not required if running in Docker
- If not installed, consider:
- HttpRouter - HTTP request router
- The
servicepackage contains the logic of the service, agnostic of presentation and transport - The
controllerpackage contains the HTTP request handlers
Optionally, arbitrarily large fake log files can be generated in json or delimited format to augment any that
are naturally in /var/log.
- See: GitHub mikebd/go-make-log - Generate fake application log files (e.g. as input to log processors)
| Argument | Description |
|---|---|
-h | --help |
Print the help message |
-lt |
Log with timestamps (UTC) |
-n |
Number of lines to return (default 25) |
-p |
HTTP Port to listen on (default 8080) |
docker build -t go-read-var-log .- Build the Docker imagedocker run -v /var/log:/var/log:ro -p 80:8080 go-read-var-log <args>- Run with the given arguments
Prevents writing to/var/log. When run without<args>, this enables the following from the docker host:curl "localhost/api/v1/logs"curl "localhost/api/v1/logs/system.log?n=10&q=syslog"
go run ./main <args>- Run the main.go file with the given arguments- IMPORTANT: Improve scalability with GOMAXPROCS:
GOMAXPROCS=8 go run ./main <args>
Tests are implemented in *_test.go files in the same package as the code under test.
Test data files are stored under testdata directories, which are ignored by go build and go run.
go test ./... -count=-1- Run all unit testsgo test ./... -count=-1 -cover- Run all unit tests with coverage- Shows 83% coverage of the
servicepackage
- Shows 83% coverage of the
❯ curl "localhost/api/v1/logs"
1GB-9million.log
daily.out
displaypolicyd.stdout.log
fct_uninstall.log
fctinstallpost.log
fsck_apfs.log
fsck_apfs_error.log
fsck_hfs.log
install.log
monthly.out
output.log
shutdown_monitor.log
system.log
weekly.out
wifi.log/var/log/1GB-9million.log was generated with GitHub mikebd/go-make-log.
❯ go run ./main -p 80
...
logs.go:30: /api/v1/logs/1GB-9million.log?q=error&r=\sfecig$ returned 1 lines in 2.75945789s
logs.go:30: /api/v1/logs/1GB-9million.log?q=error&r=\sfecig$ returned 1 lines in 2.534780081s
logs.go:30: /api/v1/logs/1GB-9million.log?q=error&r=\sfecig$ returned 1 lines in 2.148826909s
logs.go:30: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$ returned 1 lines in 1.427393941s
logs.go:30: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$ returned 1 lines in 1.447307855s
logs.go:30: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$ returned 1 lines in 1.421464319s☝️ was generated by the following curl commands:
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=error&r=\sfecig$"
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=error&r=\sfecig$"
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=error&r=\sfecig$"
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$"
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$"
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$" The curl output for all of the above is:
2023-10-10T09:27:32.029729Z|error|tesak awehit atep itego anetar utiyav oset h om tulad palopi ac arinem eralo fecigApache Bench is a simple HTTP load testing tool.
/var/log/daily.out is a 419KB file with 6,892 lines.
❯ ab -c 6 -k -n 10000 'localhost/api/v1/logs/daily.out'
Document Path: /api/v1/logs/daily.out
Document Length: 1765 bytes
Concurrency Level: 6
Time taken for tests: 3.193 seconds
Complete requests: 10000
Failed requests: 0
Keep-Alive requests: 10000
Total transferred: 19080000 bytes
HTML transferred: 17650000 bytes
Requests per second: 3131.67 [#/sec] (mean)
Time per request: 1.916 [ms] (mean)
Time per request: 0.319 [ms] (mean, across all concurrent requests)
Transfer rate: 5835.18 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1 2 0.5 2 5
Waiting: 0 2 0.5 2 5
Total: 1 2 0.5 2 5
Percentage of the requests served within a certain time (ms)
50% 2
66% 2
75% 2
80% 2
90% 3
95% 3
98% 3
99% 3
100% 5 (longest request)When run for 10,000 requests, it failed somewhere between 4,000-5,000 requests with a timeout as response times escalated to over 1 minute.
For > 2,000 requests (started as 10,000 but interrupted after 2,000):
❯ ab -c 6 -k -n 10000 'localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$'
Document Path: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$
Document Length: 117 bytes
Concurrency Level: 6
Time taken for tests: 1036.804 seconds
Complete requests: 2105
Failed requests: 0
Keep-Alive requests: 2105
Total transferred: 545195 bytes
HTML transferred: 246285 bytes
Requests per second: 2.03 [#/sec] (mean)
Time per request: 2955.261 [ms] (mean)
Time per request: 492.543 [ms] (mean, across all concurrent requests)
Transfer rate: 0.51 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1905 2946 1771.9 2766 36280
Waiting: 1905 2946 1771.9 2766 36280
Total: 1905 2946 1771.9 2766 36280
Percentage of the requests served within a certain time (ms)
50% 2766
66% 2811
75% 2853
80% 2885
90% 3007
95% 3225
98% 4481
99% 6722
100% 36280 (longest request)This fails for -n 10000 with a timeout, but "succeeds" (with poor performance) for -n 50.
There is a lot of room for optimization here if this is a use case that must be supported.
❯ ab -c 6 -k -n 50 'localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$'
Document Path: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$
Document Length: 117 bytes
Concurrency Level: 6
Time taken for tests: 31.893 seconds
Complete requests: 50
Failed requests: 0
Keep-Alive requests: 50
Total transferred: 12950 bytes
HTML transferred: 5850 bytes
Requests per second: 1.57 [#/sec] (mean)
Time per request: 3827.187 [ms] (mean)
Time per request: 637.865 [ms] (mean, across all concurrent requests)
Transfer rate: 0.40 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 0
Processing: 1832 3469 1508.0 2750 6374
Waiting: 1832 3469 1508.0 2750 6374
Total: 1832 3469 1508.0 2750 6374
Percentage of the requests served within a certain time (ms)
50% 2750
66% 3738
75% 5137
80% 5368
90% 5905
95% 6206
98% 6374
99% 6374
100% 6374 (longest request)