Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replication error, document too_large #196

Open
PetrusZ opened this issue Apr 2, 2023 · 5 comments
Open

Replication error, document too_large #196

PetrusZ opened this issue Apr 2, 2023 · 5 comments

Comments

@PetrusZ
Copy link

PetrusZ commented Apr 2, 2023

The largest file in my vault is a picture, and it's 12048 kb.

❯ du -a . | sort -n -r | head -n 5
197392  .
193840  ./assets
12048   ./assets/image_1664561617642_0.png
6544    ./assets/image_1664737228970_0.png
6224    ./assets/CleanShot_2022-11-29_at_16.57.58@2x_1669712493583_0.png

When I use the default value of chunk size, which is 100, livesync always returns error document too_large.

But when I set chuck size to 50, it successful replicates the vault.
@vrtmrz
Copy link
Owner

vrtmrz commented Apr 4, 2023

@PetrusZ
Thank you for your reporting! I will try to reproduce it!
May I ask about the configuration of the remote database? I would like to know howmax_document_size and max_http_request_size has been configured.
It can be dumped by the Make report button on the Hatch pane.

@PetrusZ
Copy link
Author

PetrusZ commented Apr 4, 2023 

----remote config----
cluster:
  seedlist: couchdb@couchdb-couchdb-0.couchdb-couchdb.database.svc.cluster.local
cors:
  credentials: "true"
  headers: accept, authorization, content-type, origin, referer
  max_age: "3600"
  methods: GET, PUT, POST, HEAD, DELETE
  origins: app://obsidian.md,capacitor://localhost,http://localhost
chttpd:
  bind_address: any
  max_http_request_size: "4.294967296e+09"
  port: "5984"
  require_valid_user: "true"
admins: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
vendor:
  name: The Apache Software Foundation
feature_flags:
  partitioned||*: "true"
chttpd_auth:
  authentication_redirect: /_utils/session.html
  require_valid_user: "true"
  secret: FBYlTdT8yPsf5Qp6jJ3C
indexers:
  couch_mrview: "true"
prometheus:
  additional_port: "false"
  bind_address: 127.0.0.1
  port: "17986"
httpd:
  WWW-Authenticate: Basic realm="couchdb"
  bind_address: 127.0.0.1
  enable_cors: "true"
  port: "5986"
couch_httpd_auth:
  authentication_db: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
  secret: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
  authentication_redirect: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchdb_engines:
  couch: couch_bt_engine
couchdb:
  database_dir: ./data
  max_document_size: "1.073741824e+10"
  uuid: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
  view_index_dir: ./data

---- Plug-in config ---
couchDB_URI: self-hosted
couchDB_USER: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchDB_PASSWORD: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchDB_DBNAME: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
liveSync: true
syncOnSave: false
syncOnStart: false
savingDelay: 200
lessInformationInLog: false
gcDelay: 0
versionUpFlash: ""
minimumChunkSize: 20
longLineThreshold: 250
showVerboseLog: false
suspendFileWatching: false
trashInsteadDelete: true
periodicReplication: false
periodicReplicationInterval: 60
syncOnFileOpen: false
encrypt: true
passphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
doNotDeleteFolder: false
resolveConflictsByNewerFile: false
batchSave: false
deviceAndVaultName: ""
usePluginSettings: false
showOwnPlugins: false
showStatusOnEditor: false
usePluginSync: true
autoSweepPlugins: true
autoSweepPluginsPeriodic: true
notifyPluginOrSettingUpdated: false
checkIntegrityOnSave: false
batch_size: 100
batches_limit: 40
useHistory: true
disableRequestURI: true
skipOlderFilesOnSync: true
checkConflictOnlyOnOpen: false
syncInternalFiles: false
syncInternalFilesBeforeReplication: false
syncInternalFilesIgnorePatterns: \/node_modules\/, \/\.git\/, \/obsidian-livesync\/,workspace$
syncInternalFilesInterval: 60
additionalSuffixOfDatabaseName: avalon
ignoreVersionCheck: false
lastReadUpdates: 17
deleteMetadataOfDeletedFiles: false
syncIgnoreRegEx: ""
syncOnlyRegEx: ""
customChunkSize: 50
readChunksOnline: true
watchInternalFileChanges: true
automaticallyDeleteMetadataOfDeletedFiles: 0
disableMarkdownAutoMerge: false
writeDocumentsIfConflicted: false
useDynamicIterationCount: false
syncAfterMerge: false
configPassphraseStore: LOCALSTORAGE
encryptedPassphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
encryptedCouchDBConnection: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
permitEmptyPassphrase: false
useIndexedDBAdapter: false
useTimeouts: false
writeLogToTheFile: false
hashCacheMaxCount: 300
hashCacheMaxAmount: 50
concurrencyOfReadChunksOnline: 100
minimumIntervalOfReadChunksOnline: 333

Hi, here is my config.

@vrtmrz
Copy link
Owner

vrtmrz commented Apr 4, 2023

Thank you for the detail! It looks configured by the exponential notation.
I have tried setting it up as well and it does not seem to be parsed as intended. (even 1.0e2).
Could you please check this by setting it up in standard notation?

@PetrusZ
Copy link
Author

PetrusZ commented Apr 4, 2023

OK, max_document_size: 10737418240 and max_http_request_size: 4294967296

@montarion
Copy link

montarion commented May 8, 2023 •
edited
Loading

I had to change my nginx(webserver) config. something in the synchronization was exceeding the body size(since I got an http 413 error), so I added the client_max_body_size directive:

client_max_body_size 200M;

@PetrusZ

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@PetrusZ @montarion @vrtmrz