allow setting s3 concurrency, proxy, timeout, uploadPartSize, putSizeLimit, version, and verify_bucket_exists by jessebot · Pull Request #2271 · nextcloud/docker

jessebot · 2024-07-26T12:39:51Z

Adds missing S3 related environment variables. Fixes #2270

Per the docs here's the missing supported S3 parameters:

concurrency defaults to 5 [Note: This defines the maximum number of concurrent multipart uploads]
proxy defaults to false
timeout defaults to 15
uploadPartSize defaults to 524288000
putSizeLimit defaults to 104857600
version defaults to latest
useMultipartCopy defaults to true
copySizeLimit defaults to 5242880000
verify_bucket_exists defaults to true [Note: Setting this to false after confirming the bucket has been created may provide a performance benefit, but may not be possible in multibucket scenarios.]

This also helps us get ready to merge nextcloud/helm#614 downstream 🙏

Let me know if you need anything else :)

.config/s3.config.php

README.md

jessebot · 2024-10-14T12:14:08Z

@joshtrichards wanted to gently follow up and ask if anything else is needed here?

jessebot · 2024-12-13T09:02:27Z

@J0WI or @joshtrichards can I ask if there's anything else needed here to get this merged into the default branch? Sorry to be a pest. I just want to make sure this gets merged so we can support it in the helm chart properly.

README.md

jessebot · 2025-01-05T09:36:14Z

@joshtrichards or @J0WI could you take another look at this when you have a moment?

jessebot · 2025-01-09T13:43:12Z

README.md

+- `OBJECTSTORE_S3_CONCURRENCY` defines the maximum number of concurrent multipart uploads
+- `OBJECTSTORE_S3_PROXY` (default: `false`)
+- `OBJECTSTORE_S3_TIMEOUT` (not set by default)
+- `OBJECTSTORE_S3_UPLOADPARTSIZE` (not set by default)
+- `OBJECTSTORE_S3_PUTSIZELIMIT` (not set by default)
+- `OBJECTSTORE_S3_USEMULTIPARTCOPY` (default: `false`)
+- `OBJECTSTORE_S3_COPYSIZELIMIT` (not set by default)
+- `OBJECTSTORE_S3_VERSION` (default: `latest`)
+- `OBJECTSTORE_S3_VERIFY_BUCKET_EXISTS` (default: `true`) Setting this to `false` after confirming the bucket has been created may provide a performance benefit, but may not be possible in multibucket scenarios.


@J0WI how's this? I now only set an int value if it is provided, this way, no one has to find out if empty string breaks stuff. I still set default strings and default booleans though, as we do that everywhere else as well and I want to be consistent. I know that if these new int s3 values are not set, it doesn't break anything, so this should be the best of both worlds right? :)

jessebot · 2025-01-09T13:43:44Z

.config/s3.config.php

+  if $concurrency {
+    $CONFIG['objectstore']['arguments']['concurrency'] = $concurrency;
+  }
+
+  if $timeout {
+    $CONFIG['objectstore']['arguments']['timeout'] = $timeout;
+  }
+
+  if $upload_part_size {
+    $CONFIG['objectstore']['arguments']['uploadPartSize'] = $upload_part_size;
+  }
+
+  if $put_size_limit {
+    $CONFIG['objectstore']['arguments']['putSizeLimit'] = $put_size_limit;
+  }
+
+  if $copy_size_limit {
+    $CONFIG['objectstore']['arguments']['copySizeLimit'] = $copy_size_limit;
+  }


these are now all conditionally set by the variables defined at the top of the file that check the env vars.

I like this approach but all tests are failing

hmmm, I don't know why... I am terrible at php. I'll try to take a look this weekend

yaleman · 2025-04-10T23:25:51Z

Any chance of an update on this? It's blocking using the helm chart 😢

…Limit, version, and verify_bucket_exists Signed-off-by: jessebot <jessebot@linux.com>

Signed-off-by: Jesse Hitch <jessebot@linux.com>

Signed-off-by: jessebot <jessebot@linux.com>

J0WI

Something appears to be wrong in the syntax. all tests are failing.

J0WI · 2025-04-18T15:18:16Z

.config/s3.config.php

        'use_path_style' => $use_path == true && strtolower($use_path) !== 'false',
        // required for older protocol versions
-        'legacy_auth' => $use_legacyauth == true && strtolower($use_legacyauth) !== 'false'
+        'useMultipartCopy' => strtolower($useMultipartCopy) !== 'true',


Suggested change

'useMultipartCopy' => strtolower($useMultipartCopy) !== 'true',

'useMultipartCopy' => strtolower($use_multipart_copy) !== 'true',

Or even just strtolower($use_multipart_copy) === 'true',?

J0WI · 2025-04-18T15:20:32Z

.config/s3.config.php

        // required for older protocol versions
-        'legacy_auth' => $use_legacyauth == true && strtolower($use_legacyauth) !== 'false'
+        'useMultipartCopy' => strtolower($useMultipartCopy) !== 'true',
+        'legacy_auth' => $use_legacyauth == true && strtolower($use_legacyauth) !== 'false',


Suggested change

'legacy_auth' => $use_legacyauth == true && strtolower($use_legacyauth) !== 'false',

'legacy_auth' => strtolower($use_path) === 'true',

J0WI · 2025-04-18T15:22:08Z

.config/s3.config.php

    )
  );

+  if $concurrency {


missing brackets around conditions.

SamyDjemai · 2025-11-07T17:56:25Z

Hi @jessebot, any update on this? Not being able to set OBJECTSTORE_S3_VERIFY_BUCKET_EXISTS is currently blocking for me

jessebot commented Jul 29, 2024

View reviewed changes

.config/s3.config.php Outdated Show resolved Hide resolved

This was referenced Jul 29, 2024

Add missing S3 parameters nextcloud/helm#614

Open

New S3 Configuration breaks prior S3 configuration and is missing uploadPartSize nextcloud/helm#612

Open

jessebot force-pushed the add-missing-s3-variables branch from c355acb to 040b741 Compare July 29, 2024 10:48

joshtrichards added enhancement feature: auto config (environment variables) Auto configuring via environment variables labels Sep 10, 2024

joshtrichards reviewed Sep 10, 2024

View reviewed changes

.config/s3.config.php Outdated Show resolved Hide resolved

joshtrichards added the 3. to review label Sep 10, 2024

joshtrichards added this to the Nextcloud 29.0.8 milestone Sep 10, 2024

joshtrichards requested a review from J0WI September 10, 2024 13:48

jessebot commented Sep 20, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

jessebot mentioned this pull request Sep 20, 2024

Kubernetes - Redeploying not working when using S3 as default storage backend nextcloud/helm#590

Open

jessebot requested a review from joshtrichards September 24, 2024 09:14

joshtrichards modified the milestones: Nextcloud 29.0.8, Nextcloud 30.0.2 Oct 9, 2024

jessebot mentioned this pull request Oct 20, 2024

Setting nextcloud.objectStore.s3.legacyAuth is not being set as string env var nextcloud/helm#637

Closed

joshtrichards self-assigned this Nov 15, 2024

joshtrichards modified the milestones: Nextcloud 30.0.2, Nextcloud 30.0.3 Nov 24, 2024

jessebot force-pushed the add-missing-s3-variables branch from 10d6edd to 2b545ea Compare December 17, 2024 18:13

jessebot commented Jan 2, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

jessebot commented Jan 2, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

jessebot force-pushed the add-missing-s3-variables branch from d7baf2f to 1581991 Compare January 2, 2025 16:19

jessebot commented Jan 9, 2025

View reviewed changes

jessebot force-pushed the add-missing-s3-variables branch from c29fea3 to e35ea1e Compare January 9, 2025 13:44

allow setting s3 concurrency, proxy, timeout, uploadPartSize, putSize…

bb0354f

…Limit, version, and verify_bucket_exists Signed-off-by: jessebot <jessebot@linux.com>

jessebot added 4 commits April 18, 2025 14:25

Update .config/s3.config.php - don't set defaults for new s3 values

3aed7a1

Signed-off-by: Jesse Hitch <jessebot@linux.com>

Update README.md

547a09d

Signed-off-by: Jesse Hitch <jessebot@linux.com>

Update README.md

49e139c

Signed-off-by: Jesse Hitch <jessebot@linux.com>

only set variables if they're actively in use

1bfb735

Signed-off-by: jessebot <jessebot@linux.com>

J0WI force-pushed the add-missing-s3-variables branch from e35ea1e to 1bfb735 Compare April 18, 2025 14:25

J0WI requested changes Apr 18, 2025

View reviewed changes

J0WI reviewed Apr 18, 2025

View reviewed changes

.config/s3.config.php

)

);

if $concurrency {

Copy link

Contributor

J0WI Apr 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing brackets around conditions.

	'useMultipartCopy' => strtolower($useMultipartCopy) !== 'true',
	'useMultipartCopy' => strtolower($use_multipart_copy) !== 'true',

	'legacy_auth' => $use_legacyauth == true && strtolower($use_legacyauth) !== 'false',
	'legacy_auth' => strtolower($use_path) === 'true',

Conversation

jessebot commented Jul 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jessebot commented Oct 14, 2024

Uh oh!

jessebot commented Dec 13, 2024

Uh oh!

Uh oh!

Uh oh!

jessebot commented Jan 5, 2025

Uh oh!

jessebot Jan 9, 2025

Choose a reason for hiding this comment

Uh oh!

jessebot Jan 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

J0WI Jan 9, 2025

Choose a reason for hiding this comment

Uh oh!

jessebot Jan 10, 2025

Choose a reason for hiding this comment

Uh oh!

yaleman commented Apr 10, 2025

Uh oh!

J0WI left a comment

Choose a reason for hiding this comment

Uh oh!

J0WI Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

J0WI Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

J0WI Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

J0WI Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

SamyDjemai commented Nov 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jessebot commented Jul 26, 2024 •

edited

Loading

jessebot Jan 9, 2025 •

edited

Loading