Describe the Issue
Redis is getting out of memory with issues to persist data to disk

To Reproduce
Steps to reproduce the issue:

1. Configure the environment or settings with '...'
2. Run the command '...'
3. Observe the error or unexpected output at '...'
4. Log output/error message

Expected Behavior
A clear and concise description of what you expected to happen when self-hosting.

Screenshots
If applicable, add screenshots or copies of the command line output to help explain the self-hosting issue.

Environment (please complete the following information):

• OS: [e.g. macOS, Linux, Windows]
• Firecrawl Version: [e.g. 1.2.3]
• Node.js Version: [e.g. 14.x]
• Docker Version (if applicable): [e.g. 20.10.14]
• Database Type and Version: [e.g. PostgreSQL 13.4]

Logs

worker-1 | 2024-11-15 16:24:01 info [:]: 🐂 Job done b2c3e207-55ca-4abb-8be1-57a0b1b88cd2
redis-1 | 1:M 15 Nov 2024 16:26:19.086 * 100 changes in 300 seconds. Saving...
redis-1 | 1:M 15 Nov 2024 16:26:19.087 * Background saving started by pid 20
redis-1 | 20:C 15 Nov 2024 16:26:19.089 # Write error while saving DB to the disk(rdbSaveRio): No space left on device
redis-1 | 1:M 15 Nov 2024 16:26:19.188 # Background saving error
worker-1 | /app/node_modules/.pnpm/[email protected]/node_modules/redis-parser/lib/parser.js:179
worker-1 | return new ReplyError(string)
worker-1 | ^
worker-1 |
worker-1 | ReplyError: MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist to disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error. script: accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a, on @user_script:202.
worker-1 | at parseError (/app/node_modules/.pnpm/[email protected]/node_modules/redis-parser/lib/parser.js:179:12)
worker-1 | at parseType (/app/node_modules/.pnpm/[email protected]/node_modules/redis-parser/lib/parser.js:302:14) {
worker-1 | command: {
worker-1 | name: 'evalsha',
worker-1 | args: [
worker-1 | 'accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a',
worker-1 | '11',
worker-1 | 'bull:{scrapeQueue}:wait',
worker-1 | 'bull:{scrapeQueue}:active',
worker-1 | 'bull:{scrapeQueue}:prioritized',
worker-1 | 'bull:{scrapeQueue}:events',
worker-1 | 'bull:{scrapeQueue}:stalled',
worker-1 | 'bull:{scrapeQueue}:limiter',
worker-1 | 'bull:{scrapeQueue}:delayed',
worker-1 | 'bull:{scrapeQueue}:paused',
worker-1 | 'bull:{scrapeQueue}:meta',
worker-1 | 'bull:{scrapeQueue}:pc',
worker-1 | 'bull:{scrapeQueue}:marker',
worker-1 | 'bull:{scrapeQueue}:',
worker-1 | '1731687980406',
worker-1 | Buffer(78) [Uint8Array] [
worker-1 | 222, 0, 4, 165, 116, 111, 107, 101, 110, 217, 36, 51,
worker-1 | 98, 50, 55, 97, 52, 56, 52, 45, 52, 100, 49, 102,
worker-1 | 45, 52, 98, 50, 57, 45, 98, 51, 49, 56, 45, 57,
worker-1 | 55, 51, 101, 48, 54, 51, 56, 54, 51, 49, 55, 172,
worker-1 | 108, 111, 99, 107, 68, 117, 114, 97, 116, 105, 111, 110,
worker-1 | 205, 234, 96, 167, 108, 105, 109, 105, 116, 101, 114, 192,
worker-1 | 164, 110, 97, 109, 101, 192
worker-1 | ]
worker-1 | ]
worker-1 | }
worker-1 | }
worker-1 |
worker-1 | Node.js v20.18.0
worker-1 exited with code 1
redis-1 | 1:M 15 Nov 2024 16:26:25.044 * 100 changes in 300 seconds. Saving...
redis-1 | 1:M 15 Nov 2024 16:26:25.045 * Background saving started by pid 21
redis-1 | 21:C 15 Nov 2024 16:26:25.046 # Write error while saving DB to the disk(rdbSaveRio): No space left on device
redis-1 | 1:M 15 Nov 2024 16:26:25.145 # Background saving error
redis-1 | 1:M 15 Nov 2024 16:26:31.008 * 100 changes in 300 seconds. Saving...
redis-1 | 1:M 15 Nov 2024 16:26:31.009 * Background saving started by pid 22
redis-1 | 22:C 15 Nov 2024 16:26:31.012 # Write error while saving DB to the disk(rdbSaveRio): No space left on device
redis-1 | 1:M 15 Nov 2024 16:26:31.111 # Background saving error
redis-1 | 1:M 15 Nov 2024 16:26:37.066 * 100 changes in 300 seconds. Saving...
redis-1 | 1:M 15 Nov 2024 16:26:37.067 * Background saving started by pid 23
redis-1 | 23:C 15 Nov 2024 16:26:37.068 # Write error while saving DB to the disk(rdbSaveRio): No space left on device
redis-1 | 1:M 15 Nov 2024 16:26:37.167 # Background saving error
redis-1 | 1:M 15 Nov 2024 16:26:43.030 * 100 changes in 300 seconds. Saving...
redis-1 | 1:M 15 Nov 2024 16:26:43.031 * Background saving started by pid 24
redis-1 | 24:C 15 Nov 2024 16:26:43.032 # Write error while saving DB to the disk(rdbSaveRio): No space left on device
redis-1 | 1:M 15 Nov 2024 16:26:43.133 # Background saving error
redis-1 | 1:M 15 Nov 2024 16:26:49.094 * 100 changes in 300 seconds. Saving...
redis-1 | 1:M 15 Nov 2024 16:26:49.096 * Background saving started by pid 25
redis-1 | 25:C 15 Nov 2024 16:26:49.099 # Write error while saving DB to the disk(rdbSaveRio): No space left on device
redis-1 | 1:M 15 Nov 2024 16:26:49.198 # Background saving error
redis-1 | 1:M 15 Nov 2024 16:26:55.074 * 100 changes in 300 seconds. Saving...
redis-1 | 1:M 15 Nov 2024 16:26:55.076 * Background saving started by pid 26
redis-1 | 26:C 15 Nov 2024 16:26:55.078 # Write error while saving DB to the disk(rdbSaveRio): No space left on device
redis-1 | 1:M 15 Nov 2024 16:26:55.177 # Background saving error
redis-1 | 1:M 15 Nov 2024 16:27:01.060 * 100 changes in 300 seconds. Saving...
redis-1 | 1:M 15 Nov 2024 16:27:01.061 * Background saving started by pid 27
redis-1 | 27:C 15 Nov 2024 16:27:01.061 # Write error while saving DB to the disk(rdbSaveRio): No space left on device
redis-1 | 1:M 15 Nov 2024 16:27:01.167 # Background saving error
redis-1 | 1:M 15 Nov 2024 16:27:07.036 * 100 changes in 300 seconds. Saving...
redis-1 | 1:M 15 Nov 2024 16:27:07.037 * Background saving started by pid 28
redis-1 | 28:C 15 Nov 2024 16:27:07.039 # Write error while saving DB to the disk(rdbSaveRio): No space left on device

Configuration
Provide relevant parts of your configuration files (with sensitive information redacted).

Additional Context
Add any other context about the self-hosting issue here, such as specific infrastructure details, network setup, or any modifications made to the original Firecrawl setup.