Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: partial enable s3 and tmp file system #429

Merged
merged 26 commits into from
Nov 5, 2024

Conversation

nyannyacha
Copy link
Collaborator

@nyannyacha nyannyacha commented Oct 28, 2024

What kind of change does this PR introduce?

Feature, Enhancement

Description

This PR implements s3 file system and tmp file system.
Access to them is supported via some of the Deno APIs below.

// Note that it will not be available for APIs with the sync suffix.
open
stat
realPath
create
remove
writeFile
writeTextFile
readFile
readTextFile

To access each filesystem, the path must be preceded by a prefix.
Below are the prefixes required to access the file system.

File System Prefix
S3 /s3
Tmp /tmp

Note that for the s3 file system, the path segment that appears right after /s3 is treated as a bucket name.
Below is a snippet with a detailed description of the s3 path.

await Deno.readFile("/s3/meowmeow/example.json");                 
//                     │     │       │                             
//                     │     │       │                             
//                     │     │       └──object key
//                     │     └──────────bucket name                
//                     └────────────────prefix to access s3 storage

The following is a sample code snippet that attempts to access the file system in the import statement.
Note that accessing the file system in this way is not currently allowed (but may be reconsidered in the future).

await Deno.writeTextFile("/tmp/meowmeow.ts", `
    import { meow } from "/tmp/meowmeow2.ts";
    export { meow }
`);

await Deno.writeTextFile("/tmp/meowmeow2.ts", `
    export function meow() {
        return "meowmeow"
    }
`);

const module = await import("/tmp/meowmeow.ts");

console.log(module);
console.log(module.meow());

You can also pass configurations for these file systems in the main worker script.

...
const s3FsConfig = {
  // optional
  appName: "meowmeow",
  // optional
  endpointUrl: "https://***REDACTED***.supabase.red/storage/v1/s3/",
  // optional
  region: "ap-southeast-1",
  // optional
  forcePathStyle: true,
  // required
  credentials: {
    // required
    accessKeyId: "***REDACTED***",
    // required
    secretAccessKey: "***REDACTED***",
    // optional
    // expiresAfter: <UNIX_TIMESTAMP_SEC>
  },
  // optional
  retry_config: {
    // optional, "standard" | "adaptive"
    mode: "standard",
    // optional
    maxAttempts: 3,
    // optional
    initialBackoffSec: 1,
    // optional
    maxBackoffSec: 20,
    // optional, "reconnect_on_transient_error" | "reuse_all_connections"
    reconnect_mode: "reconnect_on_transient_error",
    // optional
    use_static_exponential_base: false
  }
};

// tmp file system is enabled by default in the user worker.
// if no extra configuration is required, you do not need to pass it to the function.
const tmpFsConfig = {
  // optional, the base path where the temp directory will be created
  base: <PATH>,
  // optional, prefix to be used for temp directory name
  prefix: "meow",
  // optional, suffix to be used for temp directory name
  suffix: "meow",
  // optional, number of random bytes to be used for temp directory name
  random_len: 6,
  // optional, the quota limit for the file system.
  // if this is not specified, files can be written to the file system unlimited.
  quota: <BYTES>
};

return await EdgeRuntime.userWorkers.create({
  ...
  s3FsConfig,
  tmpFsConfig
});

@nyannyacha nyannyacha force-pushed the feat-partial-enable-s3-tmp-fs branch 2 times, most recently from 7c37f32 to 187af32 Compare October 29, 2024 02:18
@nyannyacha nyannyacha marked this pull request as ready for review October 29, 2024 09:57
@nyannyacha nyannyacha force-pushed the feat-partial-enable-s3-tmp-fs branch from 357b461 to 52af82d Compare November 1, 2024 03:42
@@ -87,7 +87,12 @@ where

// use a dummy npm registry url
let npm_registry_url = ModuleSpecifier::parse("https://localhost/").unwrap();
let root_path = std::env::temp_dir().join(format!("sb-compile-{}", current_exe_name));
let root_path = if cfg!(target_family = "unix") {
PathBuf::from("/var/tmp")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does it need to be /var/tmp instead of /tmp?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was changed because if the contents of eszip binary is mounted under /tmp, it will conflict with tmpfs and become inaccessible.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah gotcha!

@nyannyacha nyannyacha merged commit b10fb3d into supabase:main Nov 5, 2024
3 checks passed
@nyannyacha nyannyacha deleted the feat-partial-enable-s3-tmp-fs branch November 5, 2024 00:22
Copy link

github-actions bot commented Nov 5, 2024

🎉 This PR is included in version 1.61.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants