S3 Configuration?

ramses · January 13, 2021, 5:50am

Love how Twake is going to be the game change in co-working spaces and project sharing platforms.
Please keep up the good work guys.

I’m currently facing some issues regarding S3 integration.

Twake in deployed in a VM based docker in ESXi and is working great.
We have a MinIO server instance deployed in a Synology NAS server (in a docker as well)

How to setup up things in Parameters.php under the S3 section?
Is this below correct?

“S3” => [
“base_url” => “http//192.168.0.200:9000”,
“use” => true,
“version” => “latest”,
“buckets_prefix” => “mybucketname”,
“buckets” => [
“fr” => “eu-west-3”
],
“credentials” => [
“key” => “mykey”,
“secret” => “mysecretkey”
]
],

Do we need to deactivate Drive > False when willing to use S3?
Is this below correct?

“local” => [
“use” => false,
“location” => “…/drive/”,
“preview_location” => “…/web/medias/”,
“preview_public_path” => “/medias/”
],

Do we need to apply/use any connector for this to work?
If yes, which one?

Many thanks to anyone who could shed some lights and assist me with this;)

Long live Twake!

romaric · January 14, 2021, 11:09am

Hello @ramses and sorry for this late reply, I already saw a question about Minio from you somewhere else and I was overbooked.

Setting “local.use” to false is the good way to go. Your S3 configuration is also good.

As far as I know your configuration is correct, can you tell me what is the current behaviour of Twake when uploading documents ?

Can you follow this document https://docs.min.io/docs/how-to-use-aws-sdk-for-php-with-minio-server.html and ensure you use the good parameters ? (I only see the region that can maybe changed to us-east-1 if there is no region in Minio)

I also see that they say to set the use_path_style_endpoint to true, as we do not support this feature yet, could you try to edit this file and add this option in the $options variable?
twake/backend/core/src/Twake/Drive/Services/Storage/Adapter_AWS.php

I also see that we build the bucket name like this: “[bucket-prefix]twake.[bucket-region]”

It would be very great to have feedback on this if you have time, if not I will add this option but it will not be available before 3 weeks.

Thanks a lot for your warm words about Twake,
Romaric

romaric · January 14, 2021, 11:11am

I changed this lines in the Adapter_AWS.php file if you want to test:

$this->aws_bucket_name = isset($s3_config["bucket_name"]) ? $s3_config["bucket_name"] : ($this->aws_buckets_prefix . 'twake.' . $region);
$this->aws_bucket_region = $region;

$options = [
    'version' => $this->aws_version,
    'region' => $this->aws_bucket_region,
    'use_path_style_endpoint' => isset($s3_config["use_path_style_endpoint"]) ? $s3_config["use_path_style_endpoint"] : false,
    'credentials' => [
        'key' => $this->aws_credentials_key,
        'secret' => $this->aws_credentials_secret
    ]
];

It allows you to have this new configuration to your Parameters.php:

“S3” => [
“base_url” => “http//192.168.0.200:9000”,
“use” => true,
“version” => “latest”,
"use_path_style_endpoint" => true,
“bucket_name” => “mybucketname”,
“buckets_prefix” => “”,
“buckets” => [
“fr” => “eu-west-3”
],
“credentials” => [
“key” => “mykey”,
“secret” => “mysecretkey”
]
],

ramses · January 14, 2021, 12:30pm

Hi @romaric,

Thank you very much for getting back to me. I fully understand you might be overwhelmed by all the requests and I’m grateful you’re looking into my issue.

I’ve made the modifications you’ve suggested
And reading the modified Adapter_AWS.php code makes perfect sense.

However, when trying to upload a file to drive, 0% gets uploaded and the windows progress bar is red

Quick question regarding implementation of updated code:

I ran these commands : sudo docker-compose stop
Then integrated your code update in Adapter_AWS.php and also saved Parameters.php, please see both versions down below.
then executed these commands :
sudo docker-compose up -d
sudo docker-compose exec nginx yarn build

Are these commands enough?

Thank you again for your assistance Romaric! Really appreciated.

This is my current updated Parameters.php setup

“S3” => [
“base_url” => “http//192.168.0.200:9000”,
“use” => true,
“version” => “latest”,
“use_path_style_endpoint” => true,
“bucket_name” => “twake-data”,
“buckets_prefix” => “”,
“buckets” => [
“fr” => “eu-west-3”
],
“credentials” => [
“key” => “key”,
“secret” => “secretkey”
]
],
“local” => [
“use” => false,
“location” => “…/drive/”,
“preview_location” => “…/web/medias/”,
“preview_public_path” => “/medias/”
],

I’ve also updated Adapter_AWS.php following your directions with

$this->aws_bucket_name = isset($s3_config[“bucket_name”]) ? $s3_config[“bucket_name”] : ($this->aws_buckets_prefix . ‘twake.’ . $region);
$this->aws_bucket_region = $region;
    $options = [
        'version' => $this->aws_version,
        'region' => $this->aws_bucket_region,
        'use_path_style_endpoint' => isset($s3_config["use_path_style_endpoint"]) ? $s3_config["use_path_style_endpoint"] : false,
        'credentials' => [
            'key' => $this->aws_credentials_key,
            'secret' => $this->aws_credentials_secret
        ]
    ];

ramses · January 14, 2021, 1:04pm

Quick update @romaric,

Seeing I couldn’t upload files and S3 integration wasn’t working yet,
I updated Parameters.php again with these entries as to deactivate S3 for now, and continue the use of legacy Drive functions.

Interestingly, I can download previously uploaded files, however uploading new files doesn’t work anymore.
- Any suggestion?

Many thanks again for your time!

UPDATE:
It all works perfectly fine setting up True/False - I was impatient

Still need to work out S3 tho

romaric · January 14, 2021, 1:17pm

For the local bucket mode, try to set the acces rights to your drive volume (with chmod 777) and tell me if it works again.

For the S3 mode, you need to enter “development mode” for docker (tell docker to use the hosts php files instead of the image ones). Here is the procedure:

Add this volume to php service in docker-compose.yml:
- ./backend/core/:/twake-core:cached

And then follow this procedure:

docker-compose stop php
docker-compose up -d php
docker-compose exec php php composer.phar install
docker-compose restart php

At this time your modifications to Adapter_S3 should be took into account.
Please changes both
twake/backend/core/src/Twake/Drive/Services/OldFileSystem/Adapter_AWS_DriveFileSystem.php
and
twake/backend/core/src/Twake/Drive/Services/Storage/Adapter_AWS.php
Like in this commit:

Thanks a lot for your time,
Romaric

ramses · January 14, 2021, 6:37pm

HI @romaric,

Thank you very much for your assistance again.

So far, having implemented all the code update you’ve detailed, It seems that files aren’t going through to minIO S3 storage.

Checking logs\nginx\twakeapp_dev_error.log, attempting to upload a file generates the error below:

2021/01/14 18:31:38 [error] 10#10: *484 FastCGI sent in stderr: "PHP message: InvalidArgumentException: Endpoints must be full URIs and include a scheme and host in /twake-core/vendor/aws/aws-sdk-php/src/ClientResolver.php:669

Being after this for half day already, I’m giving it a break.
However if there is any option for you were you could take over my machine over SSH, I’m at your service to debug Twake S3 integration.

Hopefully you’ll have any useful suggestion to assist me into debugging this.

Thank you again

romaric · January 15, 2021, 9:41am

Hi, I’m working on this part of Twake so I’m ready to help you on this. A ssh access would be very great of course, but you can already get more information on the error with this command:

sudo docker-compose logs -f --tail 100 php

It will give you the php logs, according to the error you see in nginx, there is some url the AWS SDK cannot build, I still don’t know why.

You can also try to put the code from Minio in the __construct part of Adapter_AWS.php and see if you can create a single file on minio:

$s3 = new Aws\S3\S3Client([
        'version' => 'latest',
        'region'  => 'us-east-1',
        'endpoint' => 'http://localhost:9000',
        'use_path_style_endpoint' => true,
        'credentials' => [
                'key'    => 'YOUR-ACCESSKEYID',
                'secret' => 'YOUR-SECRETACCESSKEY',
            ],
]);


// Send a PutObject request and get the result object.
$insert = $s3->putObject([
     'Bucket' => 'testbucket',
     'Key'    => 'testkey',
     'Body'   => 'Hello from MinIO!!'
]);

If this works, I suggest to add error_log(json_encode($option)); variable just before the new S3Client($options) line and compare with the configuration you should have.

If this is too difficult or you don’t have time, you can give me access to your machine by PM, I will try to test and fix this today.

Romaric

ramses · January 19, 2021, 9:24am

Just to give feedback to anyone following this thread:

Romeric and I have been working on S3 integration over the last couples days.
A few bugs are currently being ironed out by Romaric and Twake awesome team about S3 buckets and read/write operations.

Twake is definitively worth the wait for those willing to run on premises services!

I’ll update in few hours/days.

Thanks