diff --git a/CHANGELOG.md b/CHANGELOG.md index 3359abb..b9eb88d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,9 +1,14 @@ -**202303.04** +## 202303.05 +- Added in ability to scrape and include PCR results for wastewater. + +## 202303.04 + +- Added in scraping of logs from the PCR table to add to wastewater submissions. - Completed partial imports that will add in missing reagents found in the kit indicated by the user. - Added web documentation to the help menu. -**202303.03** +## 202303.03 - Increased robustness by utilizing PyQT6 widget names to pull data from forms instead of previously used label/input zip. - Above allowed for creation of more helpful prompts. diff --git a/FAQ.md b/FAQ.md new file mode 100644 index 0000000..e69de29 diff --git a/README.md b/README.md index 1eea30d..73a796a 100644 --- a/README.md +++ b/README.md @@ -1,42 +1,62 @@ +## Startup: +1. Open the app using the shortcut in the Submissions folder. For example: 'L:\Robotics Laboratory Support\Submissions\submissions_v122b.exe - Shortcut.lnk' (Version may have changed). + a. Ignore the large black window of fast scrolling text, it is there for debugging purposes. + b. The 'Submissions' tab should be open by default. + ## Logging in New Run: *should fit 90% of usage cases* - 1. Ensure a properly formatted Submission Excel form has been filled out. (It will save you a few headaches) - a. All fields should be filled in to ensure proper lookups of reagents. - 2. Open the app using the shortcut in the Submissions folder. For example: "L:\Robotics Laboratory Support\Submissions\submissions_v122b.exe - Shortcut.lnk" (Version may have changed). - a. Ignore the large black window of fast scrolling text, it is there for debugging purposes. - b. The 'Submissions' tab should be open by default. - 3. Click on 'File' in the menu bar, followed by 'Import' and use the locate the form you definitely made sure was properly filled out in step 1. - 4. Click "Ok". - 5. Most of the fields in the form should be automatically filled in from the form area to the left of the screen. - 6. You may need to maximize the app to ensure you can see all the info. - 7. Any fields that are not automatically filled in can be filled in manually from the drop down menus. - 8. Once you are certain all the information is correct, click 'Submit' at the bottom of the form. - 9. Add in any reagents the app doesn't recognize. - 10. Once the new run shows up at the bottom of the Submissions, everything is fine. - 11. In case of any mistakes, the run can be overwritten by a reimport. +1. Ensure a properly formatted Submission Excel form has been filled out. (It will save you a few headaches) + a. All fields should be filled in to ensure proper lookups of reagents. +2. Click on 'File' in the menu bar, followed by 'Import Submission' and use the file dialog to locate the form you definitely made sure was properly filled out in step 1. +3. Click 'Ok'. +4. Most of the fields in the form should be automatically filled in from the form area to the left of the screen. +5. You may need to maximize the app to ensure you can see all the info. +6. Any fields that are not automatically filled in can be filled in manually from the drop down menus. +7. Once you are certain all the information is correct, click 'Submit' at the bottom of the form. +8. Add in any reagents the app doesn't recognize. +9. Once the new run shows up at the bottom of the Submissions, everything is fine. +10. In case of any mistakes, the run can be overwritten by a reimport. ## Check existing Run: - 1. Details of existing runs can be checked by double clicking on the row of interest in the summary sheet on the right of the 'Submissions' tab. - 2. All information available on the run should be available in the resulting text window. This information can be exported by clicking 'Export PDF' at the top. +1. Details of existing runs can be checked by double clicking on the row of interest in the summary sheet on the right of the 'Submissions' tab. +2. All information available on the run should be available in the resulting text window. This information can be exported by clicking 'Export PDF' at the top. ## Generating a report: - 1. Click on 'Reports' -> 'Make Report' in the menu bar. - 2. Select the start date and the end date you want for the report. Click 'ok'. - 3. Use the file dialog to select a location to save the report. - a. Both an excel sheet and a pdf should be generated containing summary information for submissions made by each client lab. +1. Click on 'Reports' -> 'Make Report' in the menu bar. +2. Select the start date and the end date you want for the report. Click 'ok'. +3. Use the file dialog to select a location to save the report. + a. Both an excel sheet and a pdf should be generated containing summary information for submissions made by each client lab. + +## Importing PCR results: +1. Click on 'File' -> 'Import PCR Results'. +2. Use the file dialog to locate the .xlsx file you want to import. +3. Click 'Okay'. ## Checking Controls: - 1. Controls for bacterial runs are now incorporated directly into the submissions database using webview. (Admittedly this performance is not as good as with a browser, so you will have to triage your data) - 2. Click on the "Controls" tab. - 3. Range of dates for controls can be selected from the date pickers at the top. - a. If start date is set after end date, the start date will default back to 3 months before end date. - b. Recommendation is to use less than 6 month date range keeping in mind that higher data density will affect performance (with kraken being the worst so far) - 4. Analysis type and subtype can be set using the drop down menus. (Only kraken has a subtype so far). +1. Controls for bacterial runs are now incorporated directly into the submissions database using webview. (Admittedly this performance is not as good as with a browser, so you will have to triage your data) +2. Click on the "Controls" tab. +3. Range of dates for controls can be selected from the date pickers at the top. + a. If start date is set after end date, the start date will default back to 3 months before end date. + b. Recommendation is to use less than 6 month date range keeping in mind that higher data density will affect performance (with kraken being the worst so far) +4. Analysis type and subtype can be set using the drop down menus. (Only kraken has a subtype so far). ## Adding new Kit: - 1. Instructions to come. +1. Instructions to come. + +## Linking Controls: + +1. Click "Monthly" -> "Link Controls". Entire process should be handled automatically. + +## Linking Extraction Logs: + +1. Click "Monthly" -> "Link Extraction Logs". +2. Chose the .csv file taken from the extraction table runlogs folder. + +## Linking PCR Logs: +1. Click "Monthly" -> "Link PCR Logs". +2. Chose the .csv file taken from the PCR table runlogs folder. diff --git a/alembic.ini b/alembic.ini index c3757ec..f147f5e 100644 --- a/alembic.ini +++ b/alembic.ini @@ -55,8 +55,9 @@ version_path_separator = os # Use os.pathsep. Default configuration used for ne # are written from script.py.mako # output_encoding = utf-8 -sqlalchemy.url = sqlite:///L:\Robotics Laboratory Support\Submissions\submissions.db -; sqlalchemy.url = sqlite:///C:\Users\lwark\Documents\Archives\DB_backups\submissions-20230213.db +; sqlalchemy.url = sqlite:///L:\Robotics Laboratory Support\Submissions\submissions.db +sqlalchemy.url = sqlite:///C:\Users\lwark\Documents\Archives\DB_backups\submissions-20230302.db +; sqlalchemy.url = sqlite:///C:\Users\lwark\Documents\python\submissions\tests\test_assets\submissions_test.db [post_write_hooks] diff --git a/alembic/versions/0ee7ffa026b2_added_pcr_info_to_wastewater_subs.py b/alembic/versions/0ee7ffa026b2_added_pcr_info_to_wastewater_subs.py new file mode 100644 index 0000000..5eba2df --- /dev/null +++ b/alembic/versions/0ee7ffa026b2_added_pcr_info_to_wastewater_subs.py @@ -0,0 +1,32 @@ +"""added pcr info to wastewater subs + +Revision ID: 0ee7ffa026b2 +Revises: 3d80e4a17a26 +Create Date: 2023-03-22 14:51:37.871062 + +""" +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision = '0ee7ffa026b2' +down_revision = '3d80e4a17a26' +branch_labels = None +depends_on = None + + +def upgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + with op.batch_alter_table('_submissions', schema=None) as batch_op: + batch_op.add_column(sa.Column('pcr_info', sa.JSON(), nullable=True)) + + # ### end Alembic commands ### + + +def downgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + with op.batch_alter_table('_submissions', schema=None) as batch_op: + batch_op.drop_column('pcr_info') + + # ### end Alembic commands ### diff --git a/alembic/versions/8adc85dd9b92_added_pcr_info_to_wastewater_samples.py b/alembic/versions/8adc85dd9b92_added_pcr_info_to_wastewater_samples.py new file mode 100644 index 0000000..2b004b1 --- /dev/null +++ b/alembic/versions/8adc85dd9b92_added_pcr_info_to_wastewater_samples.py @@ -0,0 +1,31 @@ +"""added pcr info to wastewater samples + +Revision ID: 8adc85dd9b92 +Revises: 0ee7ffa026b2 +Create Date: 2023-03-27 13:46:06.173379 + +""" +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision = '8adc85dd9b92' +down_revision = '0ee7ffa026b2' +branch_labels = None +depends_on = None + + +def upgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + with op.batch_alter_table('_ww_samples', schema=None) as batch_op: + batch_op.add_column(sa.Column('pcr_results', sa.JSON(), nullable=True)) + + # ### end Alembic commands ### + + +def downgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + with op.batch_alter_table('_ww_samples', schema=None) as batch_op: + batch_op.drop_column('pcr_results') + # ### end Alembic commands ### diff --git a/src/submissions/__init__.py b/src/submissions/__init__.py index 7ca50e6..26d63dc 100644 --- a/src/submissions/__init__.py +++ b/src/submissions/__init__.py @@ -1,6 +1,22 @@ # __init__.py +from pathlib import Path + # Version of the realpython-reader package -__version__ = "202303.3b" +__project__ = "submissions" +__version__ = "202303.4b" __author__ = {"name":"Landon Wark", "email":"Landon.Wark@phac-aspc.gc.ca"} __copyright__ = "2022-2023, Government of Canada" + +project_path = Path(__file__).parents[2].absolute() + +class bcolors: + HEADER = '\033[95m' + OKBLUE = '\033[94m' + OKCYAN = '\033[96m' + OKGREEN = '\033[92m' + WARNING = '\033[93m' + FAIL = '\033[91m' + ENDC = '\033[0m' + BOLD = '\033[1m' + UNDERLINE = '\033[4m' diff --git a/src/submissions/backend/db/functions.py b/src/submissions/backend/db/functions.py index f934883..a8f2cb6 100644 --- a/src/submissions/backend/db/functions.py +++ b/src/submissions/backend/db/functions.py @@ -5,6 +5,7 @@ Convenience functions for interacting with the database. from . import models from .models.kits import reagenttypes_kittypes from .models.submissions import reagents_submissions +from .models.samples import WWSample import pandas as pd import sqlalchemy.exc import sqlite3 @@ -392,7 +393,11 @@ def submissions_to_df(ctx:dict, sub_type:str|None=None) -> pd.DataFrame: try: df = df.drop("ext_info", axis=1) except: - logger.warning(f"Couldn't drop 'controls' column from submissionsheet df.") + logger.warning(f"Couldn't drop 'ext_info' column from submissionsheet df.") + try: + df = df.drop("pcr_info", axis=1) + except: + logger.warning(f"Couldn't drop 'pcr_info' column from submissionsheet df.") return df @@ -676,4 +681,39 @@ def delete_submission_by_id(ctx:dict, id:int) -> None: for sample in sub.samples: ctx['database_session'].delete(sample) ctx["database_session"].delete(sub) + ctx["database_session"].commit() + + +def lookup_ww_sample_by_rsl_sample_number(ctx:dict, rsl_number:str) -> models.WWSample: + """ + Retrieves wastewater sampel from database by rsl sample number + + Args: + ctx (dict): settings passed dwon from gui + rsl_number (str): sample number assigned by robotics lab + + Returns: + models.WWSample: instance of wastewater sample + """ + return ctx['database_session'].query(models.WWSample).filter(models.WWSample.rsl_number==rsl_number).first() + + +def update_ww_sample(ctx:dict, sample_obj:dict): + """ + Retrieves wastewater sample by rsl number (sample_obj['sample']) and updates values from constructed dictionary + + Args: + ctx (dict): settings passed down from gui + sample_obj (dict): dictionary representing new values for database object + """ + ww_samp = lookup_ww_sample_by_rsl_sample_number(ctx=ctx, rsl_number=sample_obj['sample']) + if ww_samp != None: + for key, value in sample_obj.items(): + logger.debug(f"Setting {key} to {value}") + # set attribute 'key' to 'value' + setattr(ww_samp, key, value) + else: + logger.error(f"Unable to find sample {sample_obj['sample']}") + return + ctx['database_session'].add(ww_samp) ctx["database_session"].commit() \ No newline at end of file diff --git a/src/submissions/backend/db/models/samples.py b/src/submissions/backend/db/models/samples.py index e00cd8e..c458ff3 100644 --- a/src/submissions/backend/db/models/samples.py +++ b/src/submissions/backend/db/models/samples.py @@ -2,7 +2,7 @@ All models for individual samples. ''' from . import Base -from sqlalchemy import Column, String, TIMESTAMP, INTEGER, ForeignKey, FLOAT, BOOLEAN +from sqlalchemy import Column, String, TIMESTAMP, INTEGER, ForeignKey, FLOAT, BOOLEAN, JSON from sqlalchemy.orm import relationship @@ -25,11 +25,12 @@ class WWSample(Base): testing_type = Column(String(64)) site_status = Column(String(64)) notes = Column(String(2000)) - ct_n1 = Column(FLOAT(2)) - ct_n2 = Column(FLOAT(2)) + ct_n1 = Column(FLOAT(2)) #: AKA ct for N1 + ct_n2 = Column(FLOAT(2)) #: AKA ct for N2 seq_submitted = Column(BOOLEAN()) ww_seq_run_id = Column(String(64)) sample_type = Column(String(8)) + pcr_results = Column(JSON) def to_string(self) -> str: @@ -48,9 +49,13 @@ class WWSample(Base): Returns: dict: well location and id NOTE: keys must sync with BCSample to_sub_dict below """ + if self.ct_n1 != None and self.ct_n2 != None: + name = f"{self.ww_sample_full_id}\n\t- ct N1: {'{:.2f}'.format(self.ct_n1)}, ct N2: {'{:.2f}'.format(self.ct_n1)}" + else: + name = self.ww_sample_full_id return { "well": self.well_number, - "name": self.ww_sample_full_id, + "name": name, } diff --git a/src/submissions/backend/db/models/submissions.py b/src/submissions/backend/db/models/submissions.py index e480b58..15b2ad0 100644 --- a/src/submissions/backend/db/models/submissions.py +++ b/src/submissions/backend/db/models/submissions.py @@ -179,5 +179,20 @@ class Wastewater(BasicSubmission): derivative submission type from BasicSubmission """ samples = relationship("WWSample", back_populates="rsl_plate", uselist=True) + pcr_info = Column(JSON) # ww_sample_id = Column(String, ForeignKey("_ww_samples.id", ondelete="SET NULL", name="fk_WW_sample_id")) - __mapper_args__ = {"polymorphic_identity": "wastewater", "polymorphic_load": "inline"} \ No newline at end of file + __mapper_args__ = {"polymorphic_identity": "wastewater", "polymorphic_load": "inline"} + + def to_dict(self) -> dict: + """ + Extends parent class method to add controls to dict + + Returns: + dict: dictionary used in submissions summary + """ + output = super().to_dict() + try: + output['pcr_info'] = json.loads(self.pcr_info) + except TypeError as e: + pass + return output \ No newline at end of file diff --git a/src/submissions/backend/excel/parser.py b/src/submissions/backend/excel/parser.py index 5b82520..1269467 100644 --- a/src/submissions/backend/excel/parser.py +++ b/src/submissions/backend/excel/parser.py @@ -1,16 +1,18 @@ ''' contains parser object for pulling values from client generated submission sheets. ''' +from getpass import getuser import pandas as pd from pathlib import Path from backend.db.models import WWSample, BCSample +from backend.db import lookup_ww_sample_by_rsl_sample_number import logging from collections import OrderedDict import re import numpy as np from datetime import date import uuid -from tools import check_not_nan +from tools import check_not_nan, retrieve_rsl_number logger = logging.getLogger(f"submissions.{__name__}") @@ -203,6 +205,7 @@ class SheetParser(object): sample_parser = SampleParser(submission_info.iloc[16:40]) sample_parse = getattr(sample_parser, f"parse_{self.sub['submission_type'].lower()}_samples") self.sub['samples'] = sample_parse() + self.sub['csv'] = self.xl.parse("Copy to import file", dtype=object) class SampleParser(object): @@ -284,3 +287,112 @@ class SampleParser(object): new.well_number = sample['Unnamed: 1'] new_list.append(new) return new_list + + +class PCRParser(object): + """ + Object to pull data from Design and Analysis PCR export file. + """ + def __init__(self, ctx:dict, filepath:Path|None = None) -> None: + """ + Initializes object. + + Args: + ctx (dict): settings passed down from gui. + filepath (Path | None, optional): file to parse. Defaults to None. + """ + self.ctx = ctx + logger.debug(f"Parsing {filepath.__str__()}") + if filepath == None: + logger.error(f"No filepath given.") + self.xl = None + else: + try: + self.xl = pd.ExcelFile(filepath.__str__()) + except ValueError as e: + logger.error(f"Incorrect value: {e}") + self.xl = None + except PermissionError: + logger.error(f"Couldn't get permissions for {filepath.__str__()}. Operation might have been cancelled.") + return + # self.pcr = OrderedDict() + self.pcr = {} + self.plate_num, self.submission_type = retrieve_rsl_number(filepath.__str__()) + logger.debug(f"Set plate number to {self.plate_num} and type to {self.submission_type}") + self.samples = [] + parser = getattr(self, f"parse_{self.submission_type}") + parser() + + + def parse_general(self, sheet_name:str): + """ + Parse general info rows for all types of PCR results + + Args: + sheet_name (str): Name of sheet in excel workbook that holds info. + """ + df = self.xl.parse(sheet_name=sheet_name, dtype=object).fillna("") + # self.pcr['file'] = df.iloc[1][1] + self.pcr['comment'] = df.iloc[0][1] + self.pcr['operator'] = df.iloc[1][1] + self.pcr['barcode'] = df.iloc[2][1] + self.pcr['instrument'] = df.iloc[3][1] + self.pcr['block_type'] = df.iloc[4][1] + self.pcr['instrument_name'] = df.iloc[5][1] + self.pcr['instrument_serial'] = df.iloc[6][1] + self.pcr['heated_cover_serial'] = df.iloc[7][1] + self.pcr['block_serial'] = df.iloc[8][1] + self.pcr['run-start'] = df.iloc[9][1] + self.pcr['run_end'] = df.iloc[10][1] + self.pcr['run_duration'] = df.iloc[11][1] + self.pcr['sample_volume'] = df.iloc[12][1] + self.pcr['cover_temp'] = df.iloc[13][1] + self.pcr['passive_ref'] = df.iloc[14][1] + self.pcr['pcr_step'] = df.iloc[15][1] + self.pcr['quant_cycle_method'] = df.iloc[16][1] + self.pcr['analysis_time'] = df.iloc[17][1] + self.pcr['software'] = df.iloc[18][1] + self.pcr['plugin'] = df.iloc[19][1] + self.pcr['exported_on'] = df.iloc[20][1] + self.pcr['imported_by'] = getuser() + return df + + def parse_wastewater(self): + """ + Parse specific to wastewater samples. + """ + df = self.parse_general(sheet_name="Results") + self.samples_df = df.iloc[23:][0:] + # iloc is [row][column] + for ii, row in self.samples_df.iterrows(): + try: + sample_obj = [sample for sample in self.samples if sample['sample'] == row[3]][0] + except IndexError: + sample_obj = dict( + sample = row[3], + ) + logger.debug(f"Got sample obj: {sample_obj}") + # logger.debug(f"row: {row}") + # rsl_num = row[3] + # # logger.debug(f"Looking up: {rsl_num}") + # ww_samp = lookup_ww_sample_by_rsl_sample_number(ctx=self.ctx, rsl_number=rsl_num) + # logger.debug(f"Got: {ww_samp}") + match row[4]: + case "N1": + if isinstance(row[12], float): + sample_obj['ct_n1'] = row[12] + else: + sample_obj['ct_n1'] = 0.0 + case "N2": + if isinstance(row[12], float): + sample_obj['ct_n2'] = row[12] + else: + sample_obj['ct_n2'] = 0.0 + case _: + logger.warning(f"Unexpected input for row[4]: {row[4]}") + self.samples.append(sample_obj) + + + + + diff --git a/src/submissions/frontend/__init__.py b/src/submissions/frontend/__init__.py index 15fec0e..f58d2aa 100644 --- a/src/submissions/frontend/__init__.py +++ b/src/submissions/frontend/__init__.py @@ -1,6 +1,7 @@ ''' Operations for all user interactions. ''' +import inspect import json import re import sys @@ -23,15 +24,15 @@ from xhtml2pdf import pisa # import plotly.express as px import yaml import pprint -from backend.excel import convert_data_list_to_df, make_report_xlsx, make_report_html, SheetParser +from backend.excel import convert_data_list_to_df, make_report_xlsx, make_report_html, SheetParser, PCRParser from backend.db import (construct_submission_info, lookup_reagent, construct_reagent, store_submission, lookup_kittype_by_use, lookup_regent_by_type_name, lookup_all_orgs, lookup_submissions_by_date_range, get_all_Control_Types_names, create_kit_from_yaml, get_all_available_modes, get_all_controls_by_type, get_control_subtypes, lookup_all_submissions_by_type, get_all_controls, lookup_submission_by_rsl_num, - create_org_from_yaml, store_reagent + create_org_from_yaml, store_reagent, lookup_ww_sample_by_rsl_sample_number, lookup_kittype_by_name, + update_ww_sample ) -from backend.db import lookup_kittype_by_name from .functions import extract_form_info from tools import check_not_nan, check_kit_integrity, check_if_app # from backend.excel.reports import @@ -89,9 +90,11 @@ class App(QMainWindow): helpMenu.addAction(self.helpAction) helpMenu.addAction(self.docsAction) fileMenu.addAction(self.importAction) + fileMenu.addAction(self.importPCRAction) reportMenu.addAction(self.generateReportAction) maintenanceMenu.addAction(self.joinControlsAction) maintenanceMenu.addAction(self.joinExtractionAction) + maintenanceMenu.addAction(self.joinPCRAction) def _createToolBar(self): """ @@ -107,13 +110,15 @@ class App(QMainWindow): """ creates actions """ - self.importAction = QAction("&Import", self) + self.importAction = QAction("&Import Submission", self) + self.importPCRAction = QAction("&Import PCR Results", self) self.addReagentAction = QAction("Add Reagent", self) self.generateReportAction = QAction("Make Report", self) self.addKitAction = QAction("Import Kit", self) self.addOrgAction = QAction("Import Org", self) self.joinControlsAction = QAction("Link Controls") - self.joinExtractionAction = QAction("Link Ext Logs") + self.joinExtractionAction = QAction("Link Extraction Logs") + self.joinPCRAction = QAction("Link PCR Logs") self.helpAction = QAction("&About", self) self.docsAction = QAction("&Docs", self) @@ -123,6 +128,7 @@ class App(QMainWindow): connect menu and tool bar item to functions """ self.importAction.triggered.connect(self.importSubmission) + self.importPCRAction.triggered.connect(self.importPCRResults) self.addReagentAction.triggered.connect(self.add_reagent) self.generateReportAction.triggered.connect(self.generateReport) self.addKitAction.triggered.connect(self.add_kit) @@ -133,6 +139,7 @@ class App(QMainWindow): self.table_widget.datepicker.end_date.dateChanged.connect(self._controls_getter) self.joinControlsAction.triggered.connect(self.linkControls) self.joinExtractionAction.triggered.connect(self.linkExtractions) + self.joinPCRAction.triggered.connect(self.linkPCR) self.helpAction.triggered.connect(self.showAbout) self.docsAction.triggered.connect(self.openDocs) @@ -180,7 +187,8 @@ class App(QMainWindow): (?P^submitted_date$) | (?P)^submitting_lab$ | (?P)^samples$ | - (?P^lot_.*$) + (?P^lot_.*$) | + (?P^csv$) """, re.VERBOSE) for item in prsr.sub: logger.debug(f"Item: {item}") @@ -213,14 +221,19 @@ class App(QMainWindow): msg = AlertPop(message="Make sure to check your extraction kit in the excel sheet!", status="warning") msg.exec() # create combobox to hold looked up kits + # add_widget = KitSelector(ctx=self.ctx, submission_type=prsr.sub['submission_type'], parent=self) add_widget = QComboBox() + # add_widget.currentTextChanged.connect(self.kit_reload) # lookup existing kits by 'submission_type' decided on by sheetparser uses = [item.__str__() for item in lookup_kittype_by_use(ctx=self.ctx, used_by=prsr.sub['submission_type'])] # if len(uses) > 0: add_widget.addItems(uses) # else: # add_widget.addItems(['bacterial_culture']) - self.ext_kit = prsr.sub[item] + if check_not_nan(prsr.sub[item]): + self.ext_kit = prsr.sub[item] + else: + self.ext_kit = add_widget.currentText() case 'submitted_date': # create label self.table_widget.formlayout.addWidget(QLabel(item.replace("_", " ").title())) @@ -234,7 +247,9 @@ class App(QMainWindow): add_widget.setDate(date.today()) case 'reagent': # create label - self.table_widget.formlayout.addWidget(QLabel(item.replace("_", " ").title())) + reg_label = QLabel(item.replace("_", " ").title()) + reg_label.setObjectName(f"lot_{item}_label") + self.table_widget.formlayout.addWidget(reg_label) # create reagent choice widget add_widget = ImportReagent(ctx=self.ctx, item=item, prsr=prsr) self.reagents[item] = prsr.sub[item] @@ -243,10 +258,13 @@ class App(QMainWindow): logger.debug(f"{item}: {prsr.sub[item]}") self.samples = prsr.sub[item] add_widget = None + case 'csv': + self.csv = prsr.sub[item] case _: # anything else gets added in as a line edit self.table_widget.formlayout.addWidget(QLabel(item.replace("_", " ").title())) add_widget = QLineEdit() + logger.debug(f"Setting widget text to {str(prsr.sub[item]).replace('_', ' ')}") add_widget.setText(str(prsr.sub[item]).replace("_", " ")) try: add_widget.setObjectName(item) @@ -256,21 +274,59 @@ class App(QMainWindow): logger.error(e) # compare self.reagents with expected reagents in kit if hasattr(self, 'ext_kit'): - kit = lookup_kittype_by_name(ctx=self.ctx, name=self.ext_kit) - kit_integrity = check_kit_integrity(kit, [item.replace("lot_", "") for item in self.reagents]) - if kit_integrity != None: - msg = AlertPop(message=kit_integrity['message'], status="critical") - msg.exec() - for item in kit_integrity['missing']: - self.table_widget.formlayout.addWidget(QLabel(f"Lot {item.replace('_', ' ').title()}")) - add_widget = ImportReagent(ctx=self.ctx, item=item) - self.table_widget.formlayout.addWidget(add_widget) + self.kit_integrity_completion() # create submission button + # submit_btn = QPushButton("Submit") + # submit_btn.setObjectName("submit_btn") + # self.table_widget.formlayout.addWidget(submit_btn) + # submit_btn.clicked.connect(self.submit_new_sample) + logger.debug(f"Imported reagents: {self.reagents}") + + + def kit_reload(self): + """ + Removes all reagents from form before running kit integrity completion. + """ + for item in self.table_widget.formlayout.parentWidget().findChildren(QWidget): + # item.setParent(None) + if isinstance(item, QLabel): + if item.text().startswith("Lot"): + item.setParent(None) + else: + logger.debug(f"Type of {item.objectName()} is {type(item)}") + if item.objectName().startswith("lot_"): + item.setParent(None) + self.kit_integrity_completion() + + + def kit_integrity_completion(self): + """ + Performs check of imported reagents + NOTE: this will not change self.reagents which should be fine + since it's only used when looking up + """ + logger.debug(inspect.currentframe().f_back.f_code.co_name) + kit_widget = self.table_widget.formlayout.parentWidget().findChild(QComboBox, 'extraction_kit') + logger.debug(f"Kit selector: {kit_widget}") + self.ext_kit = kit_widget.currentText() + logger.debug(f"Checking integrity of {self.ext_kit}") + kit = lookup_kittype_by_name(ctx=self.ctx, name=self.ext_kit) + reagents_to_lookup = [item.replace("lot_", "") for item in self.reagents] + logger.debug(f"Reagents for lookup for {kit.name}: {reagents_to_lookup}") + kit_integrity = check_kit_integrity(kit, reagents_to_lookup) + if kit_integrity != None: + msg = AlertPop(message=kit_integrity['message'], status="critical") + msg.exec() + for item in kit_integrity['missing']: + self.table_widget.formlayout.addWidget(QLabel(f"Lot {item.replace('_', ' ').title()}")) + add_widget = ImportReagent(ctx=self.ctx, item=item) + self.table_widget.formlayout.addWidget(add_widget) submit_btn = QPushButton("Submit") + submit_btn.setObjectName("lot_submit_btn") self.table_widget.formlayout.addWidget(submit_btn) submit_btn.clicked.connect(self.submit_new_sample) - logger.debug(f"Imported reagents: {self.reagents}") + def submit_new_sample(self): """ Attempt to add sample to database when 'submit' button clicked @@ -341,6 +397,15 @@ class App(QMainWindow): # reset form for item in self.table_widget.formlayout.parentWidget().findChildren(QWidget): item.setParent(None) + if hasattr(self, 'csv'): + dlg = QuestionAsker("Export CSV?", "Would you like to export the csv file?") + if dlg.exec(): + home_dir = Path(self.ctx["directory_path"]).joinpath(f"{base_submission.rsl_plate_num}.csv").resolve().__str__() + fname = Path(QFileDialog.getSaveFileName(self, "Save File", home_dir, filter=".csv")[0]) + try: + self.csv.to_csv(fname.__str__(), index=False) + except PermissionError: + logger.debug(f"Could not get permissions to {fname}. Possibly the request was cancelled.") def add_reagent(self, reagent_lot:str|None=None, reagent_type:str|None=None, expiry:date|None=None): @@ -468,8 +533,6 @@ class App(QMainWindow): msg.exec() - - def _controls_getter(self): """ Lookup controls from database and send to chartmaker @@ -550,6 +613,9 @@ class App(QMainWindow): def linkControls(self): + """ + Adds controls pulled from irida to relevant submissions + """ all_bcs = lookup_all_submissions_by_type(self.ctx, "Bacterial Culture") logger.debug(all_bcs) all_controls = get_all_controls(self.ctx) @@ -598,6 +664,9 @@ class App(QMainWindow): def linkExtractions(self): + """ + Links extraction logs from .csv files to relevant submissions. + """ home_dir = str(Path(self.ctx["directory_path"])) fname = Path(QFileDialog.getOpenFileName(self, 'Open file', home_dir, filter = "csv(*.csv)")[0]) with open(fname.__str__(), 'r') as f: @@ -648,6 +717,107 @@ class App(QMainWindow): dlg.exec() + def linkPCR(self): + """ + Links PCR logs from .csv files to relevant submissions. + """ + home_dir = str(Path(self.ctx["directory_path"])) + fname = Path(QFileDialog.getOpenFileName(self, 'Open file', home_dir, filter = "csv(*.csv)")[0]) + with open(fname.__str__(), 'r') as f: + runs = [col.strip().split(",") for col in f.readlines()] + count = 0 + for run in runs: + obj = dict( + start_time=run[0].strip(), + rsl_plate_num=run[1].strip(), + biomek_status=run[2].strip(), + quant_status=run[3].strip(), + experiment_name=run[4].strip(), + end_time=run[5].strip() + ) + # for ii in range(6, len(run)): + # obj[f"column{str(ii-5)}_vol"] = run[ii] + sub = lookup_submission_by_rsl_num(ctx=self.ctx, rsl_num=obj['rsl_plate_num']) + try: + logger.debug(f"Found submission: {sub.rsl_plate_num}") + except AttributeError: + continue + if hasattr(sub, 'pcr_info') and sub.pcr_info != None: + existing = json.loads(sub.pcr_info) + else: + existing = None + try: + if json.dumps(obj) in sub.pcr_info: + logger.debug(f"Looks like we already have that info.") + continue + else: + count += 1 + except TypeError: + logger.error(f"No json to dump") + if existing != None: + try: + logger.debug(f"Updating {type(existing)}: {existing} with {type(obj)}: {obj}") + existing.append(obj) + logger.debug(f"Setting: {existing}") + sub.pcr_info = json.dumps(existing) + except TypeError: + logger.error(f"Error updating!") + sub.pcr_info = json.dumps([obj]) + logger.debug(f"Final ext info for {sub.rsl_plate_num}: {sub.pcr_info}") + else: + sub.pcr_info = json.dumps([obj]) + self.ctx['database_session'].add(sub) + self.ctx["database_session"].commit() + dlg = AlertPop(message=f"We added {count} logs to the database.", status='information') + dlg.exec() + + + def importPCRResults(self): + """ + Imports results exported from Design and Analysis .eds files + """ + home_dir = str(Path(self.ctx["directory_path"])) + fname = Path(QFileDialog.getOpenFileName(self, 'Open file', home_dir, filter = "xlsx(*.xlsx)")[0]) + parser = PCRParser(ctx=self.ctx, filepath=fname) + logger.debug(f"Attempting lookup for {parser.plate_num}") + sub = lookup_submission_by_rsl_num(ctx=self.ctx, rsl_num=parser.plate_num) + try: + logger.debug(f"Found submission: {sub.rsl_plate_num}") + except AttributeError: + logger.error(f"Submission of number {parser.plate_num} not found.") + return + # jout = json.dumps(parser.pcr) + count = 0 + if hasattr(sub, 'pcr_info') and sub.pcr_info != None: + existing = json.loads(sub.pcr_info) + else: + # jout = None + existing = None + if existing != None: + try: + logger.debug(f"Updating {type(existing)}: {existing} with {type(parser.pcr)}: {parser.pcr}") + if json.dumps(parser.pcr) not in sub.pcr_info: + existing.append(parser.pcr) + logger.debug(f"Setting: {existing}") + sub.pcr_info = json.dumps(existing) + except TypeError: + logger.error(f"Error updating!") + sub.pcr_info = json.dumps([parser.pcr]) + logger.debug(f"Final pcr info for {sub.rsl_plate_num}: {sub.pcr_info}") + else: + sub.pcr_info = json.dumps([parser.pcr]) + self.ctx['database_session'].add(sub) + logger.debug(f"Existing {type(sub.pcr_info)}: {sub.pcr_info}") + logger.debug(f"Inserting {type(json.dumps(parser.pcr))}: {json.dumps(parser.pcr)}") + self.ctx["database_session"].commit() + logger.debug(f"Got {len(parser.samples)} to update!") + for sample in parser.samples: + logger.debug(f"Running update on: {sample['sample']}") + update_ww_sample(ctx=self.ctx, sample_obj=sample) + dlg = AlertPop(message=f"We added PCR info to {sub.rsl_plate_num}.", status='information') + dlg.exec() + + class AddSubForm(QWidget): def __init__(self, parent): diff --git a/src/submissions/frontend/custom_widgets/misc.py b/src/submissions/frontend/custom_widgets/misc.py index b15d970..b49d4cc 100644 --- a/src/submissions/frontend/custom_widgets/misc.py +++ b/src/submissions/frontend/custom_widgets/misc.py @@ -2,6 +2,7 @@ Contains miscellaneous widgets for frontend functions ''' from datetime import date +import typing from PyQt6.QtWidgets import ( QLabel, QVBoxLayout, QLineEdit, QComboBox, QDialog, @@ -10,10 +11,11 @@ from PyQt6.QtWidgets import ( QHBoxLayout, ) from PyQt6.QtCore import Qt, QDate, QSize +# from submissions.backend.db.functions import lookup_kittype_by_use # from submissions.backend.db import lookup_regent_by_type_name_and_kit_name from tools import check_not_nan from ..functions import extract_form_info -from backend.db import get_all_reagenttype_names, lookup_all_sample_types, create_kit_from_yaml, lookup_regent_by_type_name#, lookup_regent_by_type_name_and_kit_name +from backend.db import get_all_reagenttype_names, lookup_all_sample_types, create_kit_from_yaml, lookup_regent_by_type_name, lookup_kittype_by_use#, lookup_regent_by_type_name_and_kit_name from backend.excel.parser import SheetParser from jinja2 import Environment, FileSystemLoader import sys @@ -329,9 +331,13 @@ class ImportReagent(QComboBox): else: if len(relevant_reagents) > 1: logger.debug(f"Found {prsr.sub[item]['lot']} in relevant reagents: {relevant_reagents}. Moving to front of list.") - relevant_reagents.insert(0, relevant_reagents.pop(relevant_reagents.index(prsr.sub[item]['lot']))) + idx = relevant_reagents.index(str(prsr.sub[item]['lot'])) + logger.debug(f"The index we got for {prsr.sub[item]['lot']} in {relevant_reagents} was {idx}") + moved_reag = relevant_reagents.pop(idx) + relevant_reagents.insert(0, moved_reag) else: logger.debug(f"Found {prsr.sub[item]['lot']} in relevant reagents: {relevant_reagents}. But no need to move due to short list.") logger.debug(f"New relevant reagents: {relevant_reagents}") + self.setObjectName(f"lot_{item}") self.addItems(relevant_reagents) - \ No newline at end of file + diff --git a/src/submissions/templates/submission_details.html b/src/submissions/templates/submission_details.html index 3d39d6e..3e84286 100644 --- a/src/submissions/templates/submission_details.html +++ b/src/submissions/templates/submission_details.html @@ -3,9 +3,10 @@ Submission Details for {{ sub['Plate Number'] }} + {% set excluded = ['reagents', 'samples', 'controls', 'ext_info', 'pcr_info'] %}

Submission Details for {{ sub['Plate Number'] }}

-

{% for key, value in sub.items() if key != 'reagents' and key != 'samples' and key != 'controls' and key != 'ext_info' %} +

{% for key, value in sub.items() if key not in excluded %} {% if loop.index == 1 %}    {% if key=='Cost' %}{{ key }}: {{ "${:,.2f}".format(value) }}{% else %}{{ key }}: {{ value }}{% endif %}
{% else %} @@ -62,5 +63,25 @@ {% endfor %}

{% endfor %} {% endif %} + {% if sub['pcr_info'] %} + {% for entry in sub['pcr_info'] %} + {% if 'comment' not in entry.keys() %} +

qPCR Momentum Status:

+ {% else %} +

qPCR Status:

+ {% endif %} +

{% for key, value in entry.items() if key != 'imported_by'%} + {% if loop.index == 1%} +    {{ key|replace('_', ' ')|title() }}: {{ value }}
+ {% else %} + {% if "column" in key %} +     {{ key|replace('_', ' ')|title() }}: {{ value }}uL
+ {% else %} +     {{ key|replace('_', ' ')|title() }}: {{ value }}
+ {% endif %} + {% endif %} + {% endfor %}

+ {% endfor %} + {% endif %} \ No newline at end of file diff --git a/src/submissions/templates/submission_details.txt b/src/submissions/templates/submission_details.txt index 1e8dd9b..d35da8d 100644 --- a/src/submissions/templates/submission_details.txt +++ b/src/submissions/templates/submission_details.txt @@ -1,6 +1,7 @@ {# template for constructing submission details #} - -{% for key, value in sub.items() if key != 'reagents' and key != 'samples' and key != 'controls' and key != 'ext_info' %} +{% set excluded = ['reagents', 'samples', 'controls', 'ext_info', 'pcr_info'] %} +{# for key, value in sub.items() if key != 'reagents' and key != 'samples' and key != 'controls' and key != 'ext_info' #} +{% for key, value in sub.items() if key not in excluded %} {% if key=='Cost' %} {{ key }}: {{ "${:,.2f}".format(value) }} {% else %} {{ key }}: {{ value }} {% endif %} {% endfor %} Reagents: @@ -9,7 +10,7 @@ Reagents: {% if sub['samples']%} Samples: {% for item in sub['samples'] %} - {{ item['well'] }}: {{ item['name'] }}{% endfor %}{% endif %} + {{ item['well'] }}: {{ item['name'] }}{% endfor %}{% endif %} {% if sub['controls'] %} Attached Controls: {% for item in sub['controls'] %} @@ -22,6 +23,10 @@ Attached Controls: {% if sub['ext_info'] %}{% for entry in sub['ext_info'] %} Extraction Status: {% for key, value in entry.items() %} - {{ key|replace('_', ' ')|title() }}: {{ value }}{% endfor %} -{% endfor %} -{% endif %} \ No newline at end of file + {{ key|replace('_', ' ')|title() }}: {{ value }}{% endfor %}{% endfor %}{% endif %} +{% if sub['pcr_info'] %}{% for entry in sub['pcr_info'] %} +{% if 'comment' not in entry.keys() %}qPCR Momentum Status:{% else %} +qPCR Status{% endif %} +{% for key, value in entry.items() if key != 'imported_by' %} + {{ key|replace('_', ' ')|title() }}: {{ value }}{% endfor %}{% endfor %} +{% endif %} diff --git a/src/submissions/tools/__init__.py b/src/submissions/tools/__init__.py index a1c83e8..2e2613e 100644 --- a/src/submissions/tools/__init__.py +++ b/src/submissions/tools/__init__.py @@ -1,11 +1,13 @@ ''' Contains miscellaenous functions used by both frontend and backend. ''' +import re import sys import numpy as np import logging import getpass from backend.db.models import BasicSubmission, KitType +from typing import Tuple logger = logging.getLogger(f"submissions.{__name__}") @@ -115,4 +117,15 @@ def check_if_app(ctx:dict=None) -> bool: if getattr(sys, 'frozen', False): return True else: - return False \ No newline at end of file + return False + + +def retrieve_rsl_number(in_str:str) -> Tuple[str, str]: + in_str = in_str.split("\\")[-1] + logger.debug(f"Attempting match of {in_str}") + regex = re.compile(r""" + (?PRSL-WW-20\d{6})|(?PRSL-\d{2}-\d{4}) + """, re.VERBOSE) + m = regex.search(in_str) + return (m.group(), m.lastgroup) + \ No newline at end of file