Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

updating #119

Merged
merged 10 commits into from
Nov 14, 2022
Merged

updating #119

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
7719a26
fix broken file names
HadleyKing Oct 14, 2022
6af0647
Added ngs_id_script.py and template docs
Jgergely11 Oct 14, 2022
ae25547
Added ngs_id_script.py and template dataset
Jgergely11 Oct 14, 2022
0d1b402
Bump numpy from 1.21.4 to 1.22.0
dependabot[bot] Oct 16, 2022
260b325
ngs from API (#115)
Jgergely11 Nov 2, 2022
a41d9dd
Update galaxy_converter.py
HadleyKing Nov 8, 2022
54328dc
ARGOSDB API argosdb_api_query
HadleyKing Oct 27, 2022
8c45d52
saving changes
HadleyKing Nov 11, 2022
0c7becb
make argosdb_api_query.py script
HadleyKing Nov 14, 2022
cd36d68
Merge branch 'multi'
HadleyKing Nov 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added ngs_id_script.py and template docs
  • Loading branch information
@Jgergely11 @HadleyKing
Jgergely11 authored and HadleyKing committed Oct 16, 2022
commit 6af0647e8f709ece9af53f59952302cb95b7e1b2
107 changes: 107 additions & 0 deletions lib/ngs_id_script.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
import csv
import pandas as pd
pd.options.mode.chained_assignment = None # removes warning message from overwriting --- I think

#Read through HIVE Lab ngs_QC and pull out relevant data
with open("c:/users/jgerg/data.argosdb/home/template_ngsQC_HL.tsv", 'r') as inFile1:
tsvreader = csv.reader(inFile1, delimiter="\t")
with open('trimmed_HL.tsv', 'w', newline='') as outFile:
tsvwriter = csv.writer(outFile, delimiter="\t")
#remove headers from ngs_QC file
headings=next(tsvreader)
#append headers for ngs_id
tsvwriter.writerow(['organism_name', 'leaf_node', 'genome_assembly_id', 'taxonomy_id', 'bioproject','sra_run_id','ref_org','selection_notes','lab_name','files_processed'])
for row in tsvreader:
tsvwriter.writerow([row[0],row[1],row[2],row[3],row[9],row[11]])
##
##
### Removes duplicate genome assembly id's but keeps any row that does not have an assembly id
### Need to add this code - it doesnt correctly remove duplicates of samples with no genome assembly id
### For example the different SARS variants - none of them have genome assembly ids - maybe also removing duplicate SRRs will suffice
###
###
data_hl = pd.read_table('trimmed_HL.tsv', sep='\t')
df_hl = data_hl
df_hl=df_hl[df_hl['genome_assembly_id'].isnull() | ~df_hl[df_hl['genome_assembly_id'].notnull()].duplicated(subset='genome_assembly_id',keep='first')]
df_hl.lab_name = 'Hive Lab'
df_hl.files_processed = 'ngsQC_HL'

###Will print tsv for unique genome assemly ids for HIVE Lab
#df_hl.to_csv('trimmed_HL.tsv', sep = '\t', index = False)

#Repeat for Pond Lab
with open("c:/users/jgerg/data.argosdb/home/template_ngsQC_Pond.tsv", 'r') as inFile2:
tsvreader = csv.reader(inFile2, delimiter="\t")
with open('trimmed_Pond.tsv', 'w', newline='') as outFile:
tsvwriter = csv.writer(outFile, delimiter="\t")
#remove headers from ngs_QC file
headings=next(tsvreader)
#append headers for ngs_id
tsvwriter.writerow(['organism_name', 'leaf_node', 'genome_assembly_id', 'taxonomy_id', 'bioproject','sra_run_id','ref_org','selection_notes','lab_name','files_processed'])
for row in tsvreader:
tsvwriter.writerow([row[0],row[1],row[2],row[3],row[9],row[11]])
data_p = pd.read_table('trimmed_Pond.tsv', sep='\t')
df_p = data_p
df_p=df_p[df_p['genome_assembly_id'].isnull() | ~df_p[df_p['genome_assembly_id'].notnull()].duplicated(subset='genome_assembly_id',keep='first')]
df_p.lab_name = 'Pond Lab'
df_p.files_processed = 'ngsQC_Pond'

###Will print tsv for unique genome assemly ids for Pond Lab
#df_p.to_csv('trimmed_Pond.tsv', sep = '\t', index = False)


#Repeat for Crandall Lab
with open("c:/users/jgerg/data.argosdb/home/template_ngsQC_Crandall.tsv", 'r') as inFile3:
tsvreader = csv.reader(inFile3, delimiter="\t")
with open('trimmed_Crandall.tsv', 'w', newline='') as outFile:
tsvwriter = csv.writer(outFile, delimiter="\t")
#remove headers from ngs_QC file
headings=next(tsvreader)
#append headers for ngs_id
tsvwriter.writerow(['organism_name', 'leaf_node', 'genome_assembly_id', 'taxonomy_id', 'bioproject','sra_run_id','ref_org','selection_notes','lab_name','files_processed'])
for row in tsvreader:
tsvwriter.writerow([row[0],row[1],row[2],row[3],row[9],row[11]])
data_c = pd.read_table('trimmed_Crandall.tsv', sep='\t')
df_c = data_c
df_c=df_c[df_c['genome_assembly_id'].isnull() | ~df_c[df_c['genome_assembly_id'].notnull()].duplicated(subset='genome_assembly_id',keep='first')]
df_c.lab_name = 'Crandall Lab'
df_c.files_processed = 'ngsQC_Crandall'

###Will print tsv for unique genome assemly ids for HIVE Lab
#df_c.to_csv('trimmed_Crandall.tsv', sep = '\t', index = False)



dfs = [df_hl, df_p, df_c]
combined_file = pd.concat(dfs)


dfinal=combined_file
#Populate Reference Orgs
def ref_org(row):
if row['genome_assembly_id'] == 'GCA_000865725.1': # (A/Puerto Rico/8/1934(H1N1))
return "Yes"
elif row['genome_assembly_id'] == 'GCA_009858895.3': #isolate Wuhan-Hu-1
return "Yes"
elif row['genome_assembly_id'] == 'GCA_001558355.2': #LT2
return "Yes"
elif row['genome_assembly_id'] == 'GCA_000857325.2': #Marburg
return "Yes"
elif row['genome_assembly_id'] == 'GCA_003102975.1': #HXB2
return "Yes"
else:
return "No"
dfinal=dfinal.assign(ref_org=dfinal.apply(ref_org, axis =1))


#Populate selection_notes for all organisms in ARGOS Bioproject
def selection_notes(row):
if row['bioproject'] == 'PRJNA231221': #ARGOS Bioproject
return "Belongs to FDA-ARGOS PRJNA231221."
## add ifelse to populate selection_notes for SARS

#Drop bioproject column - was only used to populate selection_notes - isnt part of ngs_id_list
dfinal=dfinal.assign(selection_notes=dfinal.apply(selection_notes, axis=1))
dfinal=dfinal.drop('bioproject', axis=1)
dfinal=dfinal[dfinal['genome_assembly_id'].isnull() | ~dfinal[dfinal['genome_assembly_id'].notnull()].duplicated(subset='genome_assembly_id',keep='first')]
dfinal.to_csv('c:/users/jgerg/data.argosdb/home/test_ngs_id_list.tsv', sep = '\t', index = False)
126 changes: 126 additions & 0 deletions lib/trimmed_Crandall.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
organism_name leaf_node genome_assembly_id taxonomy_id bioproject sra_run_id ref_org selection_notes lab_name files_processed
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409217
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409217
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409217
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409217
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409217
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409217
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409217
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409217
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409217
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409217
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409217
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409216
Salmonella enterica Heidelberg GCA_002984035.1 611 PRJNA231221 SRR5409216
Salmonella enterica Hadar GCA_002984035.1 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002984035.1 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002984035.1 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417084
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417083
Salmonella enterica Hadar GCA_002209205.2 149385 PRJNA231221 SRR5417083
Salmonella enterica Heidelberg GCA_002209205.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_002209205.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_002209205.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950244
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950245
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950247
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950247
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950246
Salmonella enterica Heidelberg GCA_001188125.2 611 PRJNA231221 SRR1950246
Salmonella enterica Houtenae GCA_001188125.2 1173947 PRJNA231221 SRR1950250
Salmonella enterica Houtenae GCA_001188125.2 1173947 PRJNA231221 SRR1950250
Salmonella enterica Houtenae GCA_001188125.2 1173947 PRJNA231221 SRR1950251
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950251
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950253
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Salmonella enterica Houtenae GCA_001188155.2 1173947 PRJNA231221 SRR1950252
Loading