Twake not running after update

Dear All

I just updated twake and it stop running. I looks that it is an nginx problem. Are there know issues?

What have I done?

docker-compose stop
docker-compose rm #Remove images (not volumes so your data is safe)
docker-compose pull #Get new images
docker-compose up -d

The inicial installation was done a by the end of 2020 and I did never made an update.

This is a regular setup with apache as proxy. Apache is working fine! Before I could also see it in http://my.ip.add.ress:8000 but now nothing!

when I run
/var/Twake/twake# docker-compose up -d

I get
Starting twake_elasticsearch_1 … done
Starting twake_rabbitmq_1 … done
Starting twake_websockets_1 … done
Starting twake_scylladb_1 … done
Starting twake_php_1 … done
Starting twake_nginx_1 … done

but the logs
host not found in upstream “node” in /etc/nginx/sites-enabled/site:34

any help is welcome and congrats for the fantastic work!
Nuno

1 Like

same problem, but node container also crash with

node_1      | Error: Cannot find module '/usr/src/app/dist/server.js'
node_1      |     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:831:15)
node_1      |     at Function.Module._load (internal/modules/cjs/loader.js:687:27)
node_1      |     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
node_1      |     at internal/main/run_main_module.js:17:47 {
node_1      |   code: 'MODULE_NOT_FOUND',
node_1      |   requireStack: []

Hello,
in order to use the latest Twake version, you need to follow this migration guide:

We started a big migration work to go from PHP to NodeJS and also be compatible in the future with MongoDB.

There is three steps to follow to do the migration:

  1. Update your configuration and prepare node configuration
  2. Update your docker-compose.yml (basically add node in the configuration)
  3. Run the migration script on your database to get back your channels.

After the two first steps you will be able to start Twake but you will not see any of your channels. To get back your channels you need to run the migration script from step 3.

Don’t worry, even if the migration guide indicate to do backup (as a good practice) the migration script will not alter your data.

This is a first big update for us, next updates will not be that hard once the new node backend is setup with proper configuration. We are sorry for the difficulties.

Hi

when I did

docker run --env NODE_ENV=production migration

I got this

yarn run v1.22.5
$ tsc && node dist/start.js
WARNING: NODE_ENV value of ‘production’ did not match any deployment config file names.
WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
(node:35) UnhandledPromiseRejectionWarning: NoHostAvailableError: All host(s) tried for query failed. First host tried, 127.0.0.1:9042: Error: connect ECONNREFUSED 127.0.0.1:9042
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16) {
errno: ‘ECONNREFUSED’,
code: ‘ECONNREFUSED’,
syscall: ‘connect’,
address: ‘127.0.0.1’,
port: 9042

Any tip?

Thanks!
\N

Hi, I will update documentation if necessary, thanks for your questions !

So you need to set up your configuration for the migration, start by copying the default configuration (being in the migration/Q3toQ4 folder):

cp config/default.json config/production.json

Then edit the configuration like this:

{
  "encryption": {
    "key": "c9a17eab88ab63bb3e90c027196a89776651a7c06651a7c0", //Find it in your Parameters.php db.encryption_key
    "defaultIv": "twake_constantiv" //Should be this in your case (not in parameters by default)
  },
  "db": {
    "contactPoints": ["localhost"], //If you run Twake in the initial docker compose, try to open the port 9042 in the docker-compose.yml (open by default in the docker-compose.yml.dist) and then use the IP 127.17.0.1 (host IP), if your scylladb is running from an external machine, then use the IP of this machine.
    "localDataCenter": "datacenter1",
    "keyspace": "twake"
  }
}

The script will connect to your scylladb database and run the migration, if your database has a username / password, the configuration can change (don’t hesitate to ask me)

Hi, It does have a user name and password.

in backend/Parameters.php

“driver” => “pdo_cassandra”,
“host” => “scylladb”,
“port” => 9042,
“dbname” => “twake”,
“user” => “xxxx”,
“password” => “xxxxxx”,
“encryption_key” => “c9a17eab88ab63bb3e90c027196a89776651a7c06651a7c0”

I’m still getting the message of not finding production after
cp config/default.json config/production.json
NODE_ENV value of ‘production’ did not match any deployment config file names.

After updating the production.json config, you must rebuild the docker container:

docker build . -t migration; docker run --env NODE_ENV=production migration

Here is the configuration for a database with username and password:

{
  ...
  "db": {
   ...
    "username": "admin",
    "password": "some_password",
  }
}

thank you. the build worked

but… the scylla ip seems to be wrong

I did
docker exec -it twake_scylladb_1 nodetool status

Using /etc/scylla/scylla.yaml as the config file
Datacenter: datacenter1

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
– Address Load Tokens Owns Host ID Rack
UN 172.18.0.3 31.08 MB 256 ? d74e1161-f68a-4035-9674-f507d6832591 rack1

and used this IP but didn’t work. I tried before, of course, the 127.17.0.1, 127.0.0.1, localhost, etc

Sorry for being such a pain in the neck (and newbie)
Best

Can you tell me what happen when you do this from host machine?
curl localhost:9042

If the response is random binary data or the message “Warning: Binary output can mess up your terminal.” then your port is open and you should be using 127.17.0.1 in migration configuration.

If there is no response or the message “Connection refused”, then you probably did not open the port 9042 on docker.Check your twake/docker-compose.yml file contains this scylladb service (note the ‘ports’ line that open your 9042 and make it accessible to the host machine):

scylladb:
    image: scylladb/scylla:4.1.0
    command: --smp 1 --memory 400M
    ports:
      - 9042:9042
    volumes:
      - ./docker-data/scylladb:/var/lib/scylla

Once the mirgation is done, you can remove this port (in production it should not be open)

Hi

curl localhost:9042
curl: (1) Received HTTP/0.9 when not allowed

scylladb:
image: scylladb/scylla
ports:
- 9180:9180
- 9100:9100
- 9160:9160
- 9042:9042
volumes:
- ./docker-data/scylladb:/var/lib/scylla

Did you start scylladb before starting the migration script ?
docker-compose up -d scylladb (from Twake/twake/ folder)
And then
docker build . -t migration; docker run --env NODE_ENV=production migration

Yes, scylla was running.

this is why when I do

docker exec -it twake_scylladb_1 nodetool status

I get this

– Address Load Tokens Owns Host ID Rack
UN 172.18.0.3 31.08 MB 256 ? d74e1161-f68a-4035-9674-f507d6832591 rack1

this ip is a bit strange but I don’t know enough about containers to know if this is a problem.

Hello @romaric we are running into a similar issue.
We changed the settings as required (I think). For us, ScyllaDB got corrupted somehow. We had to remove a bunch of the scylladb system tables to even get ScyllaDB to start up:

e[36mscylladb_1  |e[0m ERROR 2021-03-02 14:30:02,459 [shard 0] database - Exception while populating keyspace 'system_schema' with column family 'dropped_columns' from file '/var/lib/scylla/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f': sstables::malformed_sstable_exception (invalid version for file md-1591-big-TOC.txt. Name doesn't match any known version.)
e[36mscylladb_1  |e[0m ERROR 2021-03-02 14:30:02,459 [shard 0] database - Exception while populating keyspace 'system_schema' with column family 'types' from file '/var/lib/scylla/data/system_schema/types-5a8b1ca866023f77a0459273d308917a': sstables::malformed_sstable_exception (invalid version for file md-19530-big-TOC.txt. Name doesn't match any known version.)
e[36mscylladb_1  |e[0m ERROR 2021-03-02 14:30:02,459 [shard 0] database - Exception while populating keyspace 'system_schema' with column family 'indexes' from file '/var/lib/scylla/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f': sstables::malformed_sstable_exception (invalid version for file md-1543-big-Data.db. Name doesn't match any known version.)
e[36mscylladb_1  |e[0m ERROR 2021-03-02 14:30:02,459 [shard 0] database - Exception while populating keyspace 'system_schema' with column family 'triggers' from file '/var/lib/scylla/data/system_schema/triggers-4df70b666b05325195a132b54005fd48': sstables::malformed_sstable_exception (invalid version for file md-19530-big-TOC.txt. Name doesn't match any known version.)
e[36mscylladb_1  |e[0m ERROR 2021-03-02 14:30:02,459 [shard 0] database - Exception while populating keyspace 'system_schema' with column family 'views' from file '/var/lib/scylla/data/system_schema/views-9786ac1cdd583201a7cdad556410c985': sstables::malformed_sstable_exception (invalid version for file md-1158-big-Index.db. Name doesn't match any known version.)
e[36mscylladb_1  |e[0m ERROR 2021-03-02 14:30:02,459 [shard 0] database - Exception while populating keyspace 'system_schema' with column family 'keyspaces' from file '/var/lib/scylla/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6': sstables::malformed_sstable_exception (invalid version for file md-1718-big-Statistics.db. Name doesn't match any known version.)

After removing all offending files as per ScyllaDB docs ScyllaDB at least was able to start again.
However, we are now at the same point that @moreno is. Our ScyllaDB does show up at the IP you suggested. Some relevant parts from docker inspect:

"PortBindings": {
                "9042/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "9042"
                    }
                ]
            },
...
                    "IPAddress": "172.18.0.2",

Using the command that @moreno posted, ScyllaDB seems to report it is working:

docker exec -it twake_scylladb_1 nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns    Host ID                               Rack
UN  172.18.0.2  199.33 KB  256          ?       023db100-54f4-45f9-8f1e-93bff347865e  rack1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

However, using your curl suggestion appears to provide the same problem for us:

$ curl 127.0.0.1:9042
curl: (1) Received HTTP/0.9 when not allowed

The migrate script then also does not work as intended of course:

yarn run v1.22.5
$ tsc && node dist/start.js
NoHostAvailableError: All host(s) tried for query failed. First host tried, 172.18.0.2:9042: DriverError: Connection timeout
    at Socket.connectTimedOut (/usr/src/app/node_modules/cassandra-driver/lib/connection.js:205:19)
    at Object.onceWrapper (events.js:420:28)
    at Socket.emit (events.js:314:20)
    at Socket._onTimeout (net.js:483:8)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7) {
  info: 'Cassandra Driver Error'
}. See innerErrors.
    at ControlConnection._borrowFirstConnection (/usr/src/app/node_modules/cassandra-driver/lib/control-connection.js:301:15)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  info: 'Represents an error when a query cannot be performed because no host is available or could be reached by the driver.',
  innerErrors: {
    '172.18.0.2:9042': DriverError: Connection timeout
        at Socket.connectTimedOut (/usr/src/app/node_modules/cassandra-driver/lib/connection.js:205:19)
        at Object.onceWrapper (events.js:420:28)
        at Socket.emit (events.js:314:20)
        at Socket._onTimeout (net.js:483:8)
        at listOnTimeout (internal/timers.js:554:17)
        at processTimers (internal/timers.js:497:7) {
      info: 'Cassandra Driver Error'
    }
  }
}
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

It would seem that there is something else broken or misconfigured on our end. :frowning_face:

Hi, here is an other way to do this:

  1. Go to Twake/twake/docker-compose.yml and add this service:

  migration:
    build:
      context: ../migration/2020Q3_to_2020Q4
      dockerfile: ./Dockerfile
    environment:
      - NODE_ENV=production
    depends_on:
      - scylladb
  1. Go to Twake/migration/2020Q3_to_2020Q4/config/production.json and use this:
{
  "encryption": {
    "key": "---",
    "defaultIv": "twake_constantiv"
  },
  "db": {
    "contactPoints": ["scylladb"],
    "localDataCenter": "datacenter1",
    "keyspace": "twake",
    ...
  }
}
  1. From Twake/twake/ do:
docker-compose up -d scylladb
  1. Wait for it to be ready (max 5 minutes in multiple nodes mode), you should see this:
$ docker-compose logs scylladb | grep "NORMAL:"
# scylladb_1   | INFO  2021-03-03 16:31:14,103 [shard 0] storage_service - NORMAL: node is now in normal status
  1. Run the migration
docker-compose up migration

Hello @romaric thanks, I did it in reverse yesterday; I made a super simple docker-compose.yml in Twake/migration/2020Q3_to_2020Q4/ that contained the scylladb config from Twake/twake/docker-compose.yml. Alas, that did not help:
ResponseError: Keyspace 'twake' does not exist
I also tried your method:

Starting twake_migration_1 ... done
Attaching to twake_migration_1
migration_1  | yarn run v1.22.5
migration_1  | $ tsc && node dist/start.js
migration_1  | ResponseError: Keyspace 'twake' does not exist
migration_1  |     at FrameReader.readError (/usr/src/app/node_modules/cassandra-driver/lib/readers.js:389:17)
migration_1  |     at Parser.parseBody (/usr/src/app/node_modules/cassandra-driver/lib/streams.js:209:66)
migration_1  |     at Parser._transform (/usr/src/app/node_modules/cassandra-driver/lib/streams.js:152:10)
migration_1  |     at Parser.Transform._read (_stream_transform.js:191:10)
migration_1  |     at Parser.Transform._write (_stream_transform.js:179:12)
migration_1  |     at doWrite (_stream_writable.js:403:12)
migration_1  |     at writeOrBuffer (_stream_writable.js:387:5)
migration_1  |     at Parser.Writable.write (_stream_writable.js:318:11)
migration_1  |     at Protocol.ondata (_stream_readable.js:718:22)
migration_1  |     at Protocol.emit (events.js:314:20) {
migration_1  |   info: 'Represents an error message from the server',
migration_1  |   code: 8704
migration_1  | }
migration_1  |
migration_1  | /usr/src/app/dist/start.js:30
migration_1  |                 throw "Something went wrong in getting Channels";
migration_1  |                 ^
migration_1  | Something went wrong in getting Channels
migration_1  | (Use `node --trace-uncaught ...` to show where the exception was thrown)
migration_1  | error Command failed with exit code 1.
migration_1  | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
twake_migration_1 exited with code 1

In both cases, it complains that it cannot find the requested keyspace.
ScyllaDB does run:
scylladb_1 | INFO 2021-03-04 13:13:55,426 [shard 0] storage_service - NORMAL: node is now in normal status

So, I went into ScyllaDB to see which keyspaces are available:

docker exec -it twake_scylladb_1 /bin/bash

   _____            _ _       _____  ____
  / ____|          | | |     |  __ \|  _ \
 | (___   ___ _   _| | | __ _| |  | | |_) |
  \___ \ / __| | | | | |/ _` | |  | |  _ <
  ____) | (__| |_| | | | (_| | |__| | |_) |
 |_____/ \___|\__, |_|_|\__,_|_____/|____/
               __/ |
              |___/


Nodetool:
        nodetool help
CQL Shell:
        cqlsh
More documentation available at:
        http://www.scylladb.com/doc/

[root@3955b7e0c943 /]# Cqlsh
bash: Cqlsh: command not found
[root@3955b7e0c943 /]# cqlsh
Connected to  at 172.19.0.2:9042.
[cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> Describe keyspaces;

system_traces  system_schema  system_auth  system  system_distributed

cqlsh>

So it would seem, sadly, that the earlier corruption basically wiped out the Twake keyspace. It is still present on-disk, but not in the ScyllaDB keyspace schema.

In that case, the script will never work as intended. We intend to start from scratch.