install issues

[spencer411]

Just trying to install and run this on our cluster. When I run the command: snakemake -rp -s MAGpy --cores 1 --use-conda test

I get the following error, but everything else seems to run fine:

scripts/add_tax.py ran with some errors: File "/network/rit/lab/andamlab/bin/MAGpy/scripts/add_tax.py", line 95
k = names[l]
TabError: inconsistent use of tabs and spaces in indentation

Further more if I move to the next step and run: snakemake -rp -s MAGpy --cores 16 --use-conda setup

It finishes very quickly, and reviewing the log file it says:

The flag 'directory' used in rule all is only valid for outputs, not inputs.
Building DAG of jobs...
Nothing to be done.

Am I missing something here with the install?

[mw55309]

Let me look into this, it's clearly not working Can you tell me which version of python and Snakemake you're using? Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: spencer411 ***@***.***> Sent: Tuesday, September 21, 2021 7:41:07 PM To: WatsonLab/MAGpy ***@***.***> Cc: Subscribed ***@***.***> Subject: [WatsonLab/MAGpy] install issues (#39) This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe. Just trying to install and run this on our cluster. When I run the command: snakemake -rp -s MAGpy --cores 1 --use-conda test I get the following error, but everything else seems to run fine: scripts/add_tax.py ran with some errors: File "/network/rit/lab/andamlab/bin/MAGpy/scripts/add_tax.py", line 95 k = names[l] TabError: inconsistent use of tabs and spaces in indentation Further more if I move to the next step and run: snakemake -rp -s MAGpy --cores 16 --use-conda setup It finishes very quickly, and reviewing the log file it says: The flag 'directory' used in rule all is only valid for outputs, not inputs. Building DAG of jobs... Nothing to be done. Am I missing something here with the install? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#39>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AANR6UF7VG7647KTZ3FON6TUDDGUHANCNFSM5EPIX2ZQ>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

[spencer411]

If I activate the magpy_install env and type "python --version" its says Python 3.9.7. Nevertheless if I type conda info, it says python version : 3.8.8.final.0.

Looking back at when I ran conda env create -f envs/install.yaml it says I installed snakemake-6.8.0

The series of code I ran based on the github site looked like this, just ensure I am doing things correctly:

git clone https://github.com/WatsonLab/MAGpy.git
cd MAGpy
conda env create -f envs/install.yaml
conda activate magpy_install
snakemake -rp -s MAGpy --cores 1 --use-conda test
snakemake -rp -s MAGpy --cores 16 --use-conda setup

Note that I am using miniconda3

Thanks!

[mw55309]

OK, latest commit should have solved most things

You need to run

rm -rf magpy_dbs

after running the tests, sorry about that

[spencer411]

Having a new issue here when I run "snakemake -rp -s MAGpy --cores 16 --use-conda setup".

It run fine for a while and then I get:

Database sequences 219174961
Database letters 75271144009
Database hash c95ef6a1a2e26795e19cb50f839b44e7
Total time 2711s
[Wed Oct 6 09:58:53 2021]
Error in rule setup:
jobid: 0
output: magpy_dbs
conda-env: /network/rit/lab/andamlab/bin/MAGpy/.snakemake/conda/12db78cf454bed3ea850367e3b9143d7
shell:

	mkdir -p magpy_dbs && cd magpy_dbs

	wget -q ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_trembl.fasta.gz
	diamond makedb --threads 4 --in uniprot_trembl.fasta.gz -d uniprot_trembl
	rm uniprot_trembl.fasta.gz

	# Sourmash
	wget -O genbank-k21.lca.json.gz -q https://osf.io/d7rv8/download
	wget -O genbank-k31.lca.json.gz -q https://osf.io/4f8n3/download
	wget -O genbank-k51.lca.json.gz -q https://osf.io/nemkw/download

	# Pfam
	wget -q ftp://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/Pfam-A.hmm.gz
	wget -q ftp://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/Pfam-A.hmm.dat.gz
	#wget -q ftp://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/active_site.dat.gz
	gunzip Pfam-A.hmm.gz Pfam-A.hmm.dat.gz #active_site.dat.gz
	hmmpress Pfam-A.hmm


	# get checkM data
	mkdir checkm_data
	cd checkm_data
	wget -q https://data.ace.uq.edu.au/public/CheckM_databases/checkm_data_2015_01_16.tar.gz
	gunzip < checkm_data_2015_01_16.tar.gz | tar xvf -
	cd ..

	cd ..

	echo '{"uniprot_sprot": "magpy_dbs/uniprot_trembl.dmnd","sourmash_gbk_k21": "magpy_dbs/genbank-k21.lca.json.gz","sourmash_gbk_k31": "magpy_dbs/genbank-k31.lca.json.gz","sourmash_gbk_k51": "magpy_dbs/genbank-k51.lca.json.gz","pfam_dir": "magpy_dbs","checkm_dataroot": "magpy_dbs/checkm_data"}' > config.json

	
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job setup since they might be corrupted:
magpy_dbs
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /network/rit/lab/andamlab/bin/MAGpy/.snakemake/log/2021-10-06T090641.681886.snakemake.log

[mw55309]

Ach that's annoying because it's going to delete everything and only one step went wrong.

I will try and split the steps and see what's happening

I also got an error, and tracked it down to the sourmash wget steps:

Resolving osf.io (osf.io)... 35.190.84.173
Connecting to osf.io (osf.io)|35.190.84.173|:443... connected.
ERROR: cannot verify osf.io's certificate, issued by ‘/C=US/O=Let's Encrypt/CN=R3’:
  Issued certificate has expired.
To connect to osf.io insecurely, use `--no-check-certificate'.

You could try that, or wait until I test this thing...

[mw55309]

I just pushed a new commit which splits up the steps, so shouldn't delete progress if just one of them fails

It all just ran fine for me :)