REST /var/log reader
- Assumptions
- Endpoints
- Enabling Tech Stack
- Design
- Generation of Sample Input Log Files
- Arguments
- Running in Docker
- Running Locally
- Run Unit Tests Locally
- Sample Output
- Benchmark with Apache Bench
Some assumptions could be considered TODO items for future enhancement
- The log files are in
/var/log - The log files are uncompressed plain text files
- Compressed or archived files (e.g.
.gz,bz2,.tar,.tgz) are not supported
- Compressed or archived files (e.g.
- The log files are sorted by timestamp (oldest first) as it is captured in each log event
- Log events are separated by a newline character
- REST response Content-Type is text/plain; charset=utf-8, regardless of the Accept request header
- All errors are returned to REST callers as HTTP status 500, even if these might be correctable by the caller
- LRU result caching of compiled regexes and search results are not implemented
- Search result caching would require invalidation when the file changes
- Hot searches could have dedicated caches that are eagerly refreshed when the file changes
- Testing coverage should be added for the
controllerpackage - Add
GOMAXPROCSto Dockerfile or set that in the code - There is a get-large-log branch where I hope to experiment with a more efficient way to read very large files, but it is not yet complete and is not formally part of this submission
GET /api/v1/logs- Get list of all log files in/var/logthat this service can readGET /api/v1/logs/{log}- Get the contents of the log file specified by{log}in/var/log(e.g./api/v1/logs/system.log)
Query Parameters:n- Number of lines to return (default: 25, all: 0)q- Case sensitive fixed text filter to apply to each line (default:""), more performant thanrr- Regex filter to apply to each line (default:.*)- Note:
qandrmay both be specified, but at most once each. Multipleqorrparameters are ignored.
- Go - Programming language
- If not installed, consider:
brew install golangor similar as appropriate for how you manage packages for development on your OS - Installation is not required if running in Docker
- If not installed, consider:
- HttpRouter - HTTP request router
- The
servicepackage contains the logic of the service, agnostic of presentation and transport - The
controllerpackage contains the HTTP request handlers
Optionally, arbitrarily large fake log files can be generated in json or delimited format to augment any that
are naturally in /var/log.
- See: GitHub mikebd/go-make-log - Generate fake application log files (e.g. as input to log processors)
| Argument | Description |
|---|---|
-h | --help |
Print the help message |
-l |
Large File Bytes (default 10,000,000) |
-lt |
Log with timestamps (UTC) |
-n |
Number of lines to return (default 25) |
-p |
HTTP Port to listen on (default 8080) |
docker build -t go-read-var-log .- Build the Docker imagedocker run -v /var/log:/var/log:ro -p 80:8080 go-read-var-log <args>- Run with the given arguments
Prevents writing to/var/log. When run without<args>, this enables the following from the docker host:curl "localhost/api/v1/logs"curl "localhost/api/v1/logs/system.log?n=10&q=syslog"
go run ./main <args>- Run the main.go file with the given arguments- IMPORTANT: Improve scalability with GOMAXPROCS:
GOMAXPROCS=8 go run ./main <args>
Tests are implemented in *_test.go files in the same package as the code under test.
Test data files are stored under testdata directories, which are ignored by go build and go run.
go test ./... -count=-1- Run all unit testsgo test ./... -count=-1 -cover- Run all unit tests with coverage
❯ curl "localhost/api/v1/logs"
1GB-9million.log
daily.out
displaypolicyd.stdout.log
fct_uninstall.log
fctinstallpost.log
fsck_apfs.log
fsck_apfs_error.log
fsck_hfs.log
install.log
monthly.out
output.log
shutdown_monitor.log
system.log
weekly.out
wifi.log/var/log/1GB-9million.log was generated with GitHub mikebd/go-make-log.
❯ go run ./main -p 80
...
logs.go:31: /api/v1/logs/1GB-9million.log?q=error&r=\sfecig$&n=1 returned 1 lines using large strategy in 1.23877778s
logs.go:31: /api/v1/logs/1GB-9million.log?q=error&r=\sfecig$&n=1 returned 1 lines using large strategy in 1.202586854s
logs.go:31: /api/v1/logs/1GB-9million.log?q=error&r=\sfecig$&n=1 returned 1 lines using large strategy in 1.198800304s
logs.go:31: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$&n=1 returned 1 lines using large strategy in 856.594448ms
logs.go:31: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$&n=1 returned 1 lines using large strategy in 866.171423ms
logs.go:31: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$&n=1 returned 1 lines using large strategy in 863.319268ms☝️ was generated by the following curl commands:
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=error&r=\sfecig$&n=1"
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=error&r=\sfecig$&n=1"
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=error&r=\sfecig$&n=1"
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$&n=1"
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$&n=1"
❯ curl "localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$&n=1" The curl output for all of the above is:
2023-10-10T09:27:32.029729Z|error|tesak awehit atep itego anetar utiyav oset h om tulad palopi ac arinem eralo fecigApache Bench is a simple HTTP load testing tool.
/var/log/daily.out is a 498KB file with 7,992 lines.
❯ ab -c 6 -k -n 10000 'localhost/api/v1/logs/daily.out'
Document Path: /api/v1/logs/daily.out
Document Length: 1930 bytes
Concurrency Level: 6
Time taken for tests: 2.062 seconds
Complete requests: 10000
Failed requests: 0
Keep-Alive requests: 10000
Total transferred: 20730000 bytes
HTML transferred: 19300000 bytes
Requests per second: 4849.47 [#/sec] (mean)
Time per request: 1.237 [ms] (mean)
Time per request: 0.206 [ms] (mean, across all concurrent requests)
Transfer rate: 9817.34 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 0 1 0.3 1 3
Waiting: 0 1 0.3 1 3
Total: 0 1 0.3 1 3
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 2
95% 2
98% 2
99% 2
100% 3 (longest request)❯ ab -c 6 -k -n 1000 'localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$&n=1'
Document Path: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$&n=1
Document Length: 117 bytes
Concurrency Level: 6
Time taken for tests: 259.356 seconds
Complete requests: 1000
Failed requests: 0
Keep-Alive requests: 1000
Total transferred: 259000 bytes
HTML transferred: 117000 bytes
Requests per second: 3.86 [#/sec] (mean)
Time per request: 1556.135 [ms] (mean)
Time per request: 259.356 [ms] (mean, across all concurrent requests)
Transfer rate: 0.98 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 909 1550 51.8 1541 1788
Waiting: 908 1550 51.8 1540 1788
Total: 909 1550 51.8 1541 1788
Percentage of the requests served within a certain time (ms)
50% 1541
66% 1559
75% 1568
80% 1578
90% 1605
95% 1632
98% 1692
99% 1757
100% 1788 (longest request)❯ ab -c 6 -k -n 1000 'localhost/api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$&n=1'
Document Path: /api/v1/logs/1GB-9million.log?q=|error|&r=\sfecig$&n=1
Document Length: 117 bytes
Concurrency Level: 6
Time taken for tests: 279.119 seconds
Complete requests: 1000
Failed requests: 0
Keep-Alive requests: 1000
Total transferred: 259000 bytes
HTML transferred: 117000 bytes
Requests per second: 3.58 [#/sec] (mean)
Time per request: 1674.712 [ms] (mean)
Time per request: 279.119 [ms] (mean, across all concurrent requests)
Transfer rate: 0.91 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1276 1663 53.5 1654 2018
Waiting: 1276 1663 53.5 1653 2018
Total: 1276 1663 53.5 1654 2018
Percentage of the requests served within a certain time (ms)
50% 1654
66% 1667
75% 1679
80% 1687
90% 1714
95% 1745
98% 1803
99% 1932
100% 2018 (longest request)