mysql update warnings

[DaanHoogland]

This adds a mysql specific upgrade notes page

[DaanHoogland]

@rhtyd @andrijapanicsb I'm not sure if this is working and enough but please chime in
@blueorangutan docbuild

[yadvr]

cc @DaanHoogland @andrijapanicsb I've added some suggestions

[DaanHoogland]

tnx @rhtyd will look tonight.

[DaanHoogland]

@blueorangutan docbuild

[DaanHoogland]

@andrijapanicsb can you add a comment on what we should say more about versions? tnx

[andrijapanicsb]

I can't Daan - we've seen this ONCE reported by we-know-who - and I think this PR is not making any use to anyone but the one being affected by the specific MySQL 5.6 issue (we didn't see an issue on 5.5, nor 5.7, so I think this must be some bug on that specific 5.6 or similar (did you do any tests with other versions? if not I can do, various 5.5 and up version, running 4.14, then upgrading to 4.15, if the effort is justified (and I think it's not, but I'm happy to invest time))

Furthermore, this kind of warning is SO WIDE in nature, we are saying it's possible to miss any SQL update in MANY different cases (we are giving one example) but this is not true, as we would have seen issues left, right and centre.

I know this is counter-productive, but I would propose dropping this PR completely, IMO, unless we are aware of a VERY specific case which needs to be documented.

[DaanHoogland]

I've checked that the default gets applied with "MySQL [5.5.5-10.3.25-MariaDB-0ubuntu0.20.04.1]" and "5.5.68-MariaDB". In my local "MySQL [5.7.28-log]" it gets applied as well. I am not aware of the precise matrix and wouldn't mind if i had the time to explore it; i don't want to make the time right now. I'd say we enter the data we have and let the community evolve the knowledge here over time.

[DaanHoogland]

@blueorangutan docbuild

[PaulAngus]

Can someone add a link to where this potential issue was reported please.

I agree with Andrija, this is a scary message that basically says: upgrading may or may not trash your database, and that the Admin needs to check their schema, looking for a non-specific problem, and then apply a non-specific fix.

If there is an issue, it's our job to ensure that the MySQL upgrade statements work, not the end-users.

Worst case, we should check that the schema changes are applied correctly as we go (not a bad idea to do this as a standard procedure for all schema updates). We can then retry with the alternate syntax, or just fail fast.

[andrijapanicsb]

This was a one-time issue that our British friends have occurred when upgrading their test env from 4.13 to 4.15 while using a specific version of MySQL 5.6.xx - the SQL statement silently failed and the upgrade succeeded (SQL that failed was "ALTER TABLE nics MODIFY COLUMN update_time timestamp DEFAULT CURRENT_TIMESTAMP;" - this is the SQL statement required to support MySQL 8, but I've also seen this being required to be manually executed on MySQL 5.7.latest even with ACS 4.14 (Oracle backported something from 8.x to 5.7.latest)

The British friends issue is what triggered Daan writing this statement, but that alone I think is not enough.
The issue I saw is more worrying, that even MySQL 5.7.latest needs this manual fix for older version of ACS.

[DaanHoogland]

@PaulAngus I like your suggestion but I think that is out of scope. Also you and @andrijapanicsb have a point about not giving to vague instructions. I have no exact version matrix and can not make the description more precise. I'll keep this open but convert to draft so we can extend it if we have more info.

[DaanHoogland]

@blueorangutan docbuild

[blueorangutan]

@DaanHoogland a Jenkins job has been kicked to build the document. I'll keep you posted as I make progress.

[blueorangutan]

Doc build preview: http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/212. (SL-JID 51)

[yadvr]

The issue has been fixed in https://github.com/apache/cloudstack/pull/4068/files for users using ACS 4.15 with a fresh installed env with MySQL8, or users who upgrade to MySQL8 prior to upgrading to ACS 4.15.

The specific issue of nics update_time column issue would still affect anyone who may upgrade CloudStack 4.15+ first and then later decide to upgrade their MySQL to 8+ (or as Andrija found newer minor version of MySQL 5.7.xxx). The root cause is a limitation/issue with older MySQL version which don't run the query (the query effectively doesn't change the column type). Since we can never know when a user may decide to upgrade to MySQL8/5.7.xxx and it's a MySQL issue, we can document it to say the users upgrading to certain MySQL versions can fix the nics exception issue by running the query (it doesn't require restarting of mgmt server or MySQL server, just fixing the column type before doing any nic/network/vm operations).

Commit reference: apache/cloudstack@d949302 (query seen in 4.14->4.15 upgrade path).

[DaanHoogland]

@blueorangutan docbuild

[DaanHoogland]

@blueorangutan docbuild

[yadvr]

@DaanHoogland can you rebase against 4.15 branch?

Changes are more specific now, LGTM. Let's also wait for other reviewers to give them lgtm.

[andrijapanicsb]

@rhtyd this is still very confusing - can you please review my proposal above ^^^

[yadvr]

@blueorangutan docbuild

[yadvr]

@andrijapanicsb it's probably because the PR needs to be rebased against 4.15 branch cc @DaanHoogland
In the meanwhile can you explain what is confusing (if other than the branch/rebase issue)?

[andrijapanicsb]

Users who may upgrade their MySQL server after upgrading to Apache

        CloudStack 4.15 or later, may need to run the following SQL query to
      
      
        fix an issue with "cloud.nics" table's column type which may lead to
      
      
        exception seen in the management server logs<!--EndFragment-->

This message is not complete/correct, because:

as I stated, I had this message reported by 2 users even when upgrading from older ACS versions to ACS 4.14 and MySQL 5.7.xx (parallel updates we do with users, so also MySQL is of a newer, 5.7.xx version) - we historically had this column type set in a certain way - and as of MySQL 5.7.xx and MySQL 8 - all ACS versions are "broken" in that regards - i.e. no VM's can be started - and

1. in pre-4.15 used with MySQL 5..7.xx you still need to manually execute this statement to fix your 4.13, or 4.14 env, etc.
2. in 4.15, this statement can fail (weird enough) during the upgrade, and again needs to be executed manually.

What I'm saying for both < 4.15 and >4.15 there are issues seen with MySQL 5.7.xx/8.x

Hope that makes sense, or we can discuss offline.

[DaanHoogland]

I do not have a clear understanding of the circumstances under which this may occur, but I believe you @andrijapanicsb , can you enter your correction as a comment/suggestion to the code, please?

[DaanHoogland]

@rhtyd thos conflicts have to do with changes that should go in 4.16 and not in the 4.15 branch. Is something else merged that shouldn't have?

[andrijapanicsb]

I've shared my 2 cents - even 4.13 or older, with MySQL 5.7.latest will have the same cloud.nics issues - so I prefer to keep it in the way I wrote it - but this is democracy, so... feel free to decide.

I do not have a clear understanding of the circumstances under which this may occur, but I believe you @andrijapanicsb , can you enter your correction as a comment/suggestion to the code, please?

I already did @DaanHoogland (for some reason, I can not apply the suggestion, but also can't add you to the list of the Reviewers... your name doesn't pop up... ) - up to you to accept the suggestion in code/merge

[DaanHoogland]

@rhtyd , I don't think this commit should be on the 4.15 branch, should it?

[yadvr]

@DaanHoogland you need to rebase your branch against latest 4.15, this branch is out of sync

[DaanHoogland]

@blueorangutan docbuild

[blueorangutan]

@DaanHoogland a Jenkins job has been kicked to build the document. I'll keep you posted as I make progress.

[blueorangutan]

Doc build preview: http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/212. (SL-JID 85)

[andrijapanicsb]

@DaanHoogland can you advise WHERE can I see this mysql.rst page being included - I can't find it when browsing http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/212

thx

[DaanHoogland]

http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/212/upgrading/upgrade/mysql.html

[yadvr]

@andrijapanicsb are you lgtm with latest changes?

[andrijapanicsb]

Yep, assuming the SQL is all correct (the fix) - I'm LGTM.

thx