Added importing of PCR results.

CHANGELOG.md

@@ -1,9 +1,14 @@
-**202303.04**
+## 202303.05
 
+- Added in ability to scrape and include PCR results for wastewater.
+
+## 202303.04
+
+- Added in scraping of logs from the PCR table to add to wastewater submissions.
 - Completed partial imports that will add in missing reagents found in the kit indicated by the user.
 - Added web documentation to the help menu.
 
-**202303.03**
+## 202303.03
 
 - Increased robustness by utilizing PyQT6 widget names to pull data from forms instead of previously used label/input zip.
 - Above allowed for creation of more helpful prompts.

README.md

@@ -1,42 +1,62 @@
+## Startup:
+1. Open the app using the shortcut in the Submissions folder. For example: 'L:\Robotics Laboratory Support\Submissions\submissions_v122b.exe - Shortcut.lnk' (Version may have changed).
+	a. Ignore the large black window of fast scrolling text, it is there for debugging purposes.
+	b. The 'Submissions' tab should be open by default.
+
 ## Logging in New Run:
 *should fit 90% of usage cases*
 
-	1. Ensure a properly formatted Submission Excel form has been filled out. (It will save you a few headaches)
+1. Ensure a properly formatted Submission Excel form has been filled out. (It will save you a few headaches)
-	    a. All fields should be filled in to ensure proper lookups of reagents.
+    a. All fields should be filled in to ensure proper lookups of reagents.
-	2. Open the app using the shortcut in the Submissions folder. For example: "L:\Robotics Laboratory Support\Submissions\submissions_v122b.exe - Shortcut.lnk" (Version may have changed).
+2. Click on 'File' in the menu bar, followed by 'Import Submission' and use the file dialog to locate the form you definitely made sure was properly filled out in step 1.
-		a. Ignore the large black window of fast scrolling text, it is there for debugging purposes.
+3. Click 'Ok'.
-		b. The 'Submissions' tab should be open by default.
+4. Most of the fields in the form should be automatically filled in from the form area to the left of the screen.
-	3. Click on 'File' in the menu bar, followed by 'Import' and use the locate the form you definitely made sure was properly filled out in step 1.
+5. You may need to maximize the app to ensure you can see all the info.
-	4. Click "Ok".
+6. Any fields that are not automatically filled in can be filled in manually from the drop down menus.
-	5. Most of the fields in the form should be automatically filled in from the form area to the left of the screen.
+7. Once you are certain all the information is correct, click 'Submit' at the bottom of the form.
-	6. You may need to maximize the app to ensure you can see all the info.
+8. Add in any reagents the app doesn't recognize.
-	7. Any fields that are not automatically filled in can be filled in manually from the drop down menus.
+9. Once the new run shows up at the bottom of the Submissions, everything is fine.
-	8. Once you are certain all the information is correct, click 'Submit' at the bottom of the form.
+10. In case of any mistakes, the run can be overwritten by a reimport.
-	9. Add in any reagents the app doesn't recognize.
-	10. Once the new run shows up at the bottom of the Submissions, everything is fine.
-	11. In case of any mistakes, the run can be overwritten by a reimport.
 
 ## Check existing Run:
 
-	1. Details of existing runs can be checked by double clicking on the row of interest in the summary sheet on the right of the 'Submissions' tab.
+1. Details of existing runs can be checked by double clicking on the row of interest in the summary sheet on the right of the 'Submissions' tab.
-	2. All information available on the run should be available in the resulting text window. This information can be exported by clicking 'Export PDF' at the top.
+2. All information available on the run should be available in the resulting text window. This information can be exported by clicking 'Export PDF' at the top.
 
 ## Generating a report:
 
-	1. Click on 'Reports' -> 'Make Report' in the menu bar.
+1. Click on 'Reports' -> 'Make Report' in the menu bar.
-	2. Select the start date and the end date you want for the report. Click 'ok'.
+2. Select the start date and the end date you want for the report. Click 'ok'.
-	3. Use the file dialog to select a location to save the report.
+3. Use the file dialog to select a location to save the report.
-		a. Both an excel sheet and a pdf should be generated containing summary information for submissions made by each client lab.
+	a. Both an excel sheet and a pdf should be generated containing summary information for submissions made by each client lab.
+
+## Importing PCR results:
+1. Click on 'File' -> 'Import PCR Results'.
+2. Use the file dialog to locate the .xlsx file you want to import.
+3. Click 'Okay'.
 	
 ## Checking Controls:
 
-	1. Controls for bacterial runs are now incorporated directly into the submissions database using webview. (Admittedly this performance is not as good as with a browser, so you will have to triage your data)
+1. Controls for bacterial runs are now incorporated directly into the submissions database using webview. (Admittedly this performance is not as good as with a browser, so you will have to triage your data)
-	2. Click on the "Controls" tab.
+2. Click on the "Controls" tab.
-	3. Range of dates for controls can be selected from the date pickers at the top.
+3. Range of dates for controls can be selected from the date pickers at the top.
-		a. If start date is set after end date, the start date will default back to 3 months before end date.
+	a. If start date is set after end date, the start date will default back to 3 months before end date.
-		b. Recommendation is to use less than 6 month date range keeping in mind that higher data density will affect performance (with kraken being the worst so far)
+	b. Recommendation is to use less than 6 month date range keeping in mind that higher data density will affect performance (with kraken being the worst so far)
-	4. Analysis type and subtype can be set using the drop down menus. (Only kraken has a subtype so far).
+4. Analysis type and subtype can be set using the drop down menus. (Only kraken has a subtype so far).
 
 ## Adding new Kit:
 
-	1. Instructions to come.
+1. Instructions to come.
+
+## Linking Controls:
+
+1. Click "Monthly" -> "Link Controls". Entire process should be handled automatically.
+
+## Linking Extraction Logs:
+
+1. Click "Monthly" -> "Link Extraction Logs".
+2. Chose the .csv file taken from the extraction table runlogs folder.
+
+## Linking PCR Logs:
+1. Click "Monthly" -> "Link PCR Logs".
+2. Chose the .csv file taken from the PCR table runlogs folder.

alembic.ini

@@ -55,8 +55,9 @@ version_path_separator = os  # Use os.pathsep. Default configuration used for ne
 # are written from script.py.mako
 # output_encoding = utf-8
 
-sqlalchemy.url = sqlite:///L:\Robotics Laboratory Support\Submissions\submissions.db
+; sqlalchemy.url = sqlite:///L:\Robotics Laboratory Support\Submissions\submissions.db
-; sqlalchemy.url = sqlite:///C:\Users\lwark\Documents\Archives\DB_backups\submissions-20230213.db
+sqlalchemy.url = sqlite:///C:\Users\lwark\Documents\Archives\DB_backups\submissions-20230302.db
+; sqlalchemy.url = sqlite:///C:\Users\lwark\Documents\python\submissions\tests\test_assets\submissions_test.db
 
 
 [post_write_hooks]

alembic/versions/0ee7ffa026b2_added_pcr_info_to_wastewater_subs.py

@@ -0,0 +1,32 @@
+"""added pcr info to wastewater subs
+
+Revision ID: 0ee7ffa026b2
+Revises: 3d80e4a17a26
+Create Date: 2023-03-22 14:51:37.871062
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = '0ee7ffa026b2'
+down_revision = '3d80e4a17a26'
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('_submissions', schema=None) as batch_op:
+        batch_op.add_column(sa.Column('pcr_info', sa.JSON(), nullable=True))
+        
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('_submissions', schema=None) as batch_op:
+        batch_op.drop_column('pcr_info')
+
+    # ### end Alembic commands ###

alembic/versions/8adc85dd9b92_added_pcr_info_to_wastewater_samples.py

@@ -0,0 +1,31 @@
+"""added pcr info to wastewater samples
+
+Revision ID: 8adc85dd9b92
+Revises: 0ee7ffa026b2
+Create Date: 2023-03-27 13:46:06.173379
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = '8adc85dd9b92'
+down_revision = '0ee7ffa026b2'
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('_ww_samples', schema=None) as batch_op:
+        batch_op.add_column(sa.Column('pcr_results', sa.JSON(), nullable=True))
+
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('_ww_samples', schema=None) as batch_op:
+        batch_op.drop_column('pcr_results')
+    # ### end Alembic commands ###

src/submissions/__init__.py

@@ -1,6 +1,22 @@
 # __init__.py
 
+from pathlib import Path
+
 # Version of the realpython-reader package
-__version__ = "202303.3b"
+__project__ = "submissions"
+__version__ = "202303.4b"
 __author__ = {"name":"Landon Wark", "email":"Landon.Wark@phac-aspc.gc.ca"}
 __copyright__ = "2022-2023, Government of Canada"
+
+project_path = Path(__file__).parents[2].absolute()
+
+class bcolors:
+    HEADER = '\033[95m'
+    OKBLUE = '\033[94m'
+    OKCYAN = '\033[96m'
+    OKGREEN = '\033[92m'
+    WARNING = '\033[93m'
+    FAIL = '\033[91m'
+    ENDC = '\033[0m'
+    BOLD = '\033[1m'
+    UNDERLINE = '\033[4m'

src/submissions/backend/db/functions.py

@@ -5,6 +5,7 @@ Convenience functions for interacting with the database.
 from . import models
 from .models.kits import reagenttypes_kittypes
 from .models.submissions import reagents_submissions
+from .models.samples import WWSample
 import pandas as pd
 import sqlalchemy.exc
 import sqlite3
@@ -392,7 +393,11 @@ def submissions_to_df(ctx:dict, sub_type:str|None=None) -> pd.DataFrame:
     try:
         df = df.drop("ext_info", axis=1)
     except:
-        logger.warning(f"Couldn't drop 'controls' column from submissionsheet df.")
+        logger.warning(f"Couldn't drop 'ext_info' column from submissionsheet df.")
+    try:
+        df = df.drop("pcr_info", axis=1)
+    except:
+        logger.warning(f"Couldn't drop 'pcr_info' column from submissionsheet df.")
     return df
      
     
@@ -677,3 +682,38 @@ def delete_submission_by_id(ctx:dict, id:int) -> None:
         ctx['database_session'].delete(sample)
     ctx["database_session"].delete(sub)
     ctx["database_session"].commit()
+
+
+def lookup_ww_sample_by_rsl_sample_number(ctx:dict, rsl_number:str) -> models.WWSample:
+    """
+    Retrieves wastewater sampel from database by rsl sample number
+
+    Args:
+        ctx (dict): settings passed dwon from gui
+        rsl_number (str): sample number assigned by robotics lab
+
+    Returns:
+        models.WWSample: instance of wastewater sample
+    """    
+    return ctx['database_session'].query(models.WWSample).filter(models.WWSample.rsl_number==rsl_number).first()
+
+
+def update_ww_sample(ctx:dict, sample_obj:dict):
+    """
+    Retrieves wastewater sample by rsl number (sample_obj['sample']) and updates values from constructed dictionary
+
+    Args:
+        ctx (dict): settings passed down from gui
+        sample_obj (dict): dictionary representing new values for database object
+    """    
+    ww_samp = lookup_ww_sample_by_rsl_sample_number(ctx=ctx, rsl_number=sample_obj['sample'])
+    if ww_samp != None:
+        for key, value in sample_obj.items():
+            logger.debug(f"Setting {key} to {value}")
+            # set attribute 'key' to 'value'
+            setattr(ww_samp, key, value)
+    else:
+        logger.error(f"Unable to find sample {sample_obj['sample']}")
+        return
+    ctx['database_session'].add(ww_samp)
+    ctx["database_session"].commit()

src/submissions/backend/db/models/samples.py

@@ -2,7 +2,7 @@
 All models for individual samples.
 '''
 from . import Base
-from sqlalchemy import Column, String, TIMESTAMP, INTEGER, ForeignKey, FLOAT, BOOLEAN
+from sqlalchemy import Column, String, TIMESTAMP, INTEGER, ForeignKey, FLOAT, BOOLEAN, JSON
 from sqlalchemy.orm import relationship
 
 
@@ -25,11 +25,12 @@ class WWSample(Base):
     testing_type = Column(String(64)) 
     site_status = Column(String(64))
     notes = Column(String(2000))
-    ct_n1 = Column(FLOAT(2))
+    ct_n1 = Column(FLOAT(2)) #: AKA ct for N1
-    ct_n2 = Column(FLOAT(2))
+    ct_n2 = Column(FLOAT(2)) #: AKA ct for N2
     seq_submitted = Column(BOOLEAN())
     ww_seq_run_id = Column(String(64))
     sample_type = Column(String(8))
+    pcr_results = Column(JSON)
     
 
     def to_string(self) -> str:
@@ -48,9 +49,13 @@ class WWSample(Base):
         Returns:
             dict: well location and id NOTE: keys must sync with BCSample to_sub_dict below
         """
+        if self.ct_n1 != None and self.ct_n2 != None:
+            name = f"{self.ww_sample_full_id}\n\t- ct N1: {'{:.2f}'.format(self.ct_n1)}, ct N2: {'{:.2f}'.format(self.ct_n1)}"
+        else:
+            name = self.ww_sample_full_id
         return {
             "well": self.well_number,
-            "name": self.ww_sample_full_id,
+            "name": name,
         }
 
 

src/submissions/backend/db/models/submissions.py

@@ -179,5 +179,20 @@ class Wastewater(BasicSubmission):
     derivative submission type from BasicSubmission
     """    
     samples = relationship("WWSample", back_populates="rsl_plate", uselist=True)
+    pcr_info = Column(JSON)
     # ww_sample_id = Column(String, ForeignKey("_ww_samples.id", ondelete="SET NULL", name="fk_WW_sample_id"))
     __mapper_args__ = {"polymorphic_identity": "wastewater", "polymorphic_load": "inline"}
+
+    def to_dict(self) -> dict:
+        """
+        Extends parent class method to add controls to dict
+
+        Returns:
+            dict: dictionary used in submissions summary
+        """        
+        output = super().to_dict()
+        try:
+            output['pcr_info'] = json.loads(self.pcr_info)
+        except TypeError as e:
+            pass
+        return output

src/submissions/backend/excel/parser.py

@@ -1,16 +1,18 @@
 '''
 contains parser object for pulling values from client generated submission sheets.
 '''
+from getpass import getuser
 import pandas as pd
 from pathlib import Path
 from backend.db.models import WWSample, BCSample
+from backend.db import lookup_ww_sample_by_rsl_sample_number
 import logging
 from collections import OrderedDict
 import re
 import numpy as np
 from datetime import date
 import uuid
-from tools import check_not_nan
+from tools import check_not_nan, retrieve_rsl_number
 
 logger = logging.getLogger(f"submissions.{__name__}")
 
@@ -203,6 +205,7 @@ class SheetParser(object):
         sample_parser = SampleParser(submission_info.iloc[16:40])
         sample_parse = getattr(sample_parser, f"parse_{self.sub['submission_type'].lower()}_samples")
         self.sub['samples'] = sample_parse()
+        self.sub['csv'] = self.xl.parse("Copy to import file", dtype=object)
 
 
 class SampleParser(object):
@@ -284,3 +287,112 @@ class SampleParser(object):
             new.well_number = sample['Unnamed: 1']
             new_list.append(new)
         return new_list
+
+
+class PCRParser(object):
+    """
+    Object to pull data from Design and Analysis PCR export file.
+    """    
+    def __init__(self, ctx:dict, filepath:Path|None = None) -> None:
+        """
+        Initializes object.
+
+        Args:
+            ctx (dict): settings passed down from gui.
+            filepath (Path | None, optional): file to parse. Defaults to None.
+        """        
+        self.ctx = ctx
+        logger.debug(f"Parsing {filepath.__str__()}")        
+        if filepath == None:
+            logger.error(f"No filepath given.")
+            self.xl = None
+        else:
+            try:
+                self.xl = pd.ExcelFile(filepath.__str__())
+            except ValueError as e:
+                logger.error(f"Incorrect value: {e}")
+                self.xl = None
+            except PermissionError:
+                logger.error(f"Couldn't get permissions for {filepath.__str__()}. Operation might have been cancelled.")
+                return
+        # self.pcr = OrderedDict()
+        self.pcr = {}
+        self.plate_num, self.submission_type = retrieve_rsl_number(filepath.__str__())
+        logger.debug(f"Set plate number to {self.plate_num} and type to {self.submission_type}")
+        self.samples = []
+        parser = getattr(self, f"parse_{self.submission_type}")
+        parser()
+        
+
+    def parse_general(self, sheet_name:str):
+        """
+        Parse general info rows for all types of PCR results
+
+        Args:
+            sheet_name (str): Name of sheet in excel workbook that holds info.
+        """        
+        df = self.xl.parse(sheet_name=sheet_name, dtype=object).fillna("")
+        # self.pcr['file'] = df.iloc[1][1]
+        self.pcr['comment'] = df.iloc[0][1]
+        self.pcr['operator'] = df.iloc[1][1]
+        self.pcr['barcode'] = df.iloc[2][1]
+        self.pcr['instrument'] = df.iloc[3][1]
+        self.pcr['block_type'] = df.iloc[4][1]
+        self.pcr['instrument_name'] = df.iloc[5][1]
+        self.pcr['instrument_serial'] = df.iloc[6][1]
+        self.pcr['heated_cover_serial'] = df.iloc[7][1]
+        self.pcr['block_serial'] = df.iloc[8][1]
+        self.pcr['run-start'] = df.iloc[9][1]
+        self.pcr['run_end'] = df.iloc[10][1]
+        self.pcr['run_duration'] = df.iloc[11][1]
+        self.pcr['sample_volume'] = df.iloc[12][1]
+        self.pcr['cover_temp'] = df.iloc[13][1]
+        self.pcr['passive_ref'] = df.iloc[14][1]
+        self.pcr['pcr_step'] = df.iloc[15][1]
+        self.pcr['quant_cycle_method'] = df.iloc[16][1]
+        self.pcr['analysis_time'] = df.iloc[17][1]
+        self.pcr['software'] = df.iloc[18][1]
+        self.pcr['plugin'] = df.iloc[19][1]
+        self.pcr['exported_on'] = df.iloc[20][1]
+        self.pcr['imported_by'] = getuser()
+        return df
+
+    def parse_wastewater(self):
+        """
+        Parse specific to wastewater samples.
+        """        
+        df = self.parse_general(sheet_name="Results")
+        self.samples_df = df.iloc[23:][0:]
+        # iloc is [row][column]
+        for ii, row in self.samples_df.iterrows():
+            try:
+                sample_obj = [sample for sample in self.samples if sample['sample'] == row[3]][0]    
+            except IndexError:
+                sample_obj = dict(
+                    sample = row[3],
+                )
+            logger.debug(f"Got sample obj: {sample_obj}") 
+            # logger.debug(f"row: {row}")
+            # rsl_num = row[3]
+            # # logger.debug(f"Looking up: {rsl_num}")
+            # ww_samp = lookup_ww_sample_by_rsl_sample_number(ctx=self.ctx, rsl_number=rsl_num)
+            # logger.debug(f"Got: {ww_samp}")
+            match row[4]:
+                case "N1":
+                    if isinstance(row[12], float):
+                        sample_obj['ct_n1'] = row[12]
+                    else:
+                        sample_obj['ct_n1'] = 0.0
+                case "N2":
+                    if isinstance(row[12], float):
+                        sample_obj['ct_n2'] = row[12]
+                    else:
+                        sample_obj['ct_n2'] = 0.0
+                case _:
+                    logger.warning(f"Unexpected input for row[4]: {row[4]}")
+            self.samples.append(sample_obj)
+                
+
+
+            
+            

src/submissions/frontend/__init__.py

@@ -1,6 +1,7 @@
 '''
 Operations for all user interactions.
 '''
+import inspect
 import json
 import re
 import sys
@@ -23,15 +24,15 @@ from xhtml2pdf import pisa
 # import plotly.express as px
 import yaml
 import pprint
-from backend.excel import convert_data_list_to_df, make_report_xlsx, make_report_html, SheetParser
+from backend.excel import convert_data_list_to_df, make_report_xlsx, make_report_html, SheetParser, PCRParser
 from backend.db import (construct_submission_info, lookup_reagent, 
     construct_reagent, store_submission, lookup_kittype_by_use,
     lookup_regent_by_type_name, lookup_all_orgs, lookup_submissions_by_date_range,
     get_all_Control_Types_names, create_kit_from_yaml, get_all_available_modes, get_all_controls_by_type,
     get_control_subtypes, lookup_all_submissions_by_type, get_all_controls, lookup_submission_by_rsl_num,
-    create_org_from_yaml, store_reagent
+    create_org_from_yaml, store_reagent, lookup_ww_sample_by_rsl_sample_number, lookup_kittype_by_name,
+    update_ww_sample
 )
-from backend.db import lookup_kittype_by_name
 from .functions import extract_form_info
 from tools import check_not_nan, check_kit_integrity, check_if_app
 # from backend.excel.reports import 
@@ -89,9 +90,11 @@ class App(QMainWindow):
         helpMenu.addAction(self.helpAction)
         helpMenu.addAction(self.docsAction)
         fileMenu.addAction(self.importAction)
+        fileMenu.addAction(self.importPCRAction)
         reportMenu.addAction(self.generateReportAction)
         maintenanceMenu.addAction(self.joinControlsAction)
         maintenanceMenu.addAction(self.joinExtractionAction)
+        maintenanceMenu.addAction(self.joinPCRAction)
         
     def _createToolBar(self):
         """
@@ -107,13 +110,15 @@ class App(QMainWindow):
         """
         creates actions
         """        
-        self.importAction = QAction("&Import", self)
+        self.importAction = QAction("&Import Submission", self)
+        self.importPCRAction = QAction("&Import PCR Results", self)
         self.addReagentAction = QAction("Add Reagent", self)
         self.generateReportAction = QAction("Make Report", self)
         self.addKitAction = QAction("Import Kit", self)
         self.addOrgAction = QAction("Import Org", self)
         self.joinControlsAction = QAction("Link Controls")
-        self.joinExtractionAction = QAction("Link Ext Logs")
+        self.joinExtractionAction = QAction("Link Extraction Logs")
+        self.joinPCRAction = QAction("Link PCR Logs")
         self.helpAction = QAction("&About", self)
         self.docsAction = QAction("&Docs", self)
 
@@ -123,6 +128,7 @@ class App(QMainWindow):
         connect menu and tool bar item to functions
         """
         self.importAction.triggered.connect(self.importSubmission)
+        self.importPCRAction.triggered.connect(self.importPCRResults)
         self.addReagentAction.triggered.connect(self.add_reagent)
         self.generateReportAction.triggered.connect(self.generateReport)
         self.addKitAction.triggered.connect(self.add_kit)
@@ -133,6 +139,7 @@ class App(QMainWindow):
         self.table_widget.datepicker.end_date.dateChanged.connect(self._controls_getter)
         self.joinControlsAction.triggered.connect(self.linkControls)
         self.joinExtractionAction.triggered.connect(self.linkExtractions)
+        self.joinPCRAction.triggered.connect(self.linkPCR)
         self.helpAction.triggered.connect(self.showAbout)
         self.docsAction.triggered.connect(self.openDocs)
 
@@ -180,7 +187,8 @@ class App(QMainWindow):
             (?P<submitted_date>^submitted_date$) |
             (?P<submitting_lab>)^submitting_lab$ |
             (?P<samples>)^samples$ |
-            (?P<reagent>^lot_.*$)
+            (?P<reagent>^lot_.*$) |
+            (?P<csv>^csv$)
         """, re.VERBOSE)
         for item in prsr.sub:
             logger.debug(f"Item: {item}")
@@ -213,14 +221,19 @@ class App(QMainWindow):
                         msg = AlertPop(message="Make sure to check your extraction kit in the excel sheet!", status="warning")
                         msg.exec()
                     # create combobox to hold looked up kits
+                    # add_widget = KitSelector(ctx=self.ctx, submission_type=prsr.sub['submission_type'], parent=self)
                     add_widget = QComboBox()
+                    # add_widget.currentTextChanged.connect(self.kit_reload)
                     # lookup existing kits by 'submission_type' decided on by sheetparser
                     uses = [item.__str__() for item in lookup_kittype_by_use(ctx=self.ctx, used_by=prsr.sub['submission_type'])]
                     # if len(uses) > 0:
                     add_widget.addItems(uses)
                     # else:
                         # add_widget.addItems(['bacterial_culture'])
-                    self.ext_kit = prsr.sub[item]
+                    if check_not_nan(prsr.sub[item]):
+                        self.ext_kit = prsr.sub[item]
+                    else:
+                        self.ext_kit = add_widget.currentText()
                 case 'submitted_date':
                     # create label
                     self.table_widget.formlayout.addWidget(QLabel(item.replace("_", " ").title()))
@@ -234,7 +247,9 @@ class App(QMainWindow):
                         add_widget.setDate(date.today())
                 case 'reagent':
                     # create label
-                    self.table_widget.formlayout.addWidget(QLabel(item.replace("_", " ").title()))
+                    reg_label = QLabel(item.replace("_", " ").title())
+                    reg_label.setObjectName(f"lot_{item}_label")
+                    self.table_widget.formlayout.addWidget(reg_label)
                     # create reagent choice widget
                     add_widget = ImportReagent(ctx=self.ctx, item=item, prsr=prsr)
                     self.reagents[item] = prsr.sub[item]
@@ -243,10 +258,13 @@ class App(QMainWindow):
                     logger.debug(f"{item}: {prsr.sub[item]}")
                     self.samples = prsr.sub[item]
                     add_widget = None
+                case 'csv':
+                    self.csv = prsr.sub[item]
                 case _:
                     # anything else gets added in as a line edit
                     self.table_widget.formlayout.addWidget(QLabel(item.replace("_", " ").title()))
                     add_widget = QLineEdit()
+                    logger.debug(f"Setting widget text to {str(prsr.sub[item]).replace('_', ' ')}")
                     add_widget.setText(str(prsr.sub[item]).replace("_", " "))
             try:
                 add_widget.setObjectName(item)
@@ -256,20 +274,58 @@ class App(QMainWindow):
                 logger.error(e)
         # compare self.reagents with expected reagents in kit
         if hasattr(self, 'ext_kit'):
-            kit = lookup_kittype_by_name(ctx=self.ctx, name=self.ext_kit)
+            self.kit_integrity_completion()
-            kit_integrity = check_kit_integrity(kit, [item.replace("lot_", "") for item in self.reagents])
-            if kit_integrity != None:
-                msg = AlertPop(message=kit_integrity['message'], status="critical")
-                msg.exec()
-                for item in kit_integrity['missing']:
-                    self.table_widget.formlayout.addWidget(QLabel(f"Lot {item.replace('_', ' ').title()}"))
-                    add_widget = ImportReagent(ctx=self.ctx, item=item)
-                    self.table_widget.formlayout.addWidget(add_widget)
         # create submission button
+        # submit_btn = QPushButton("Submit")
+        # submit_btn.setObjectName("submit_btn")
+        # self.table_widget.formlayout.addWidget(submit_btn)
+        # submit_btn.clicked.connect(self.submit_new_sample)
+        logger.debug(f"Imported reagents: {self.reagents}")
+
+
+    def kit_reload(self):
+        """
+        Removes all reagents from form before running kit integrity completion.
+        """        
+        for item in self.table_widget.formlayout.parentWidget().findChildren(QWidget):
+            # item.setParent(None)
+            if isinstance(item, QLabel):
+                if item.text().startswith("Lot"):
+                    item.setParent(None)
+            else:
+                logger.debug(f"Type of {item.objectName()} is {type(item)}")
+                if item.objectName().startswith("lot_"):
+                    item.setParent(None)
+        self.kit_integrity_completion()
+    
+
+    def kit_integrity_completion(self):
+        """
+        Performs check of imported reagents
+        NOTE: this will not change self.reagents which should be fine
+        since it's only used when looking up 
+        """        
+        logger.debug(inspect.currentframe().f_back.f_code.co_name)
+        kit_widget = self.table_widget.formlayout.parentWidget().findChild(QComboBox, 'extraction_kit')
+        logger.debug(f"Kit selector: {kit_widget}")
+        self.ext_kit = kit_widget.currentText()
+        logger.debug(f"Checking integrity of {self.ext_kit}")
+        kit = lookup_kittype_by_name(ctx=self.ctx, name=self.ext_kit)
+        reagents_to_lookup = [item.replace("lot_", "") for item in self.reagents]
+        logger.debug(f"Reagents for lookup for {kit.name}: {reagents_to_lookup}")
+        kit_integrity = check_kit_integrity(kit, reagents_to_lookup)
+        if kit_integrity != None:
+            msg = AlertPop(message=kit_integrity['message'], status="critical")
+            msg.exec()
+            for item in kit_integrity['missing']:
+                self.table_widget.formlayout.addWidget(QLabel(f"Lot {item.replace('_', ' ').title()}"))
+                add_widget = ImportReagent(ctx=self.ctx, item=item)
+                self.table_widget.formlayout.addWidget(add_widget)
         submit_btn = QPushButton("Submit")
+        submit_btn.setObjectName("lot_submit_btn")
         self.table_widget.formlayout.addWidget(submit_btn)
         submit_btn.clicked.connect(self.submit_new_sample)
-        logger.debug(f"Imported reagents: {self.reagents}")
+        
 
     def submit_new_sample(self):
         """
@@ -341,6 +397,15 @@ class App(QMainWindow):
         # reset form
         for item in self.table_widget.formlayout.parentWidget().findChildren(QWidget):
             item.setParent(None)
+        if hasattr(self, 'csv'):
+            dlg = QuestionAsker("Export CSV?", "Would you like to export the csv file?")
+            if dlg.exec():
+                home_dir = Path(self.ctx["directory_path"]).joinpath(f"{base_submission.rsl_plate_num}.csv").resolve().__str__()
+                fname = Path(QFileDialog.getSaveFileName(self, "Save File", home_dir, filter=".csv")[0])
+                try:
+                    self.csv.to_csv(fname.__str__(), index=False)
+                except PermissionError:
+                    logger.debug(f"Could not get permissions to {fname}. Possibly the request was cancelled.")
 
 
     def add_reagent(self, reagent_lot:str|None=None, reagent_type:str|None=None, expiry:date|None=None):
@@ -468,8 +533,6 @@ class App(QMainWindow):
         msg.exec()
 
 
-
-
     def _controls_getter(self):
         """
         Lookup controls from database and send to chartmaker
@@ -550,6 +613,9 @@ class App(QMainWindow):
 
 
     def linkControls(self):
+        """
+        Adds controls pulled from irida to relevant submissions
+        """        
         all_bcs = lookup_all_submissions_by_type(self.ctx, "Bacterial Culture")
         logger.debug(all_bcs)
         all_controls = get_all_controls(self.ctx)
@@ -598,6 +664,9 @@ class App(QMainWindow):
 
 
     def linkExtractions(self):
+        """
+        Links extraction logs from .csv files to relevant submissions.
+        """        
         home_dir = str(Path(self.ctx["directory_path"]))
         fname = Path(QFileDialog.getOpenFileName(self, 'Open file', home_dir, filter = "csv(*.csv)")[0])
         with open(fname.__str__(), 'r') as f:
@@ -648,6 +717,107 @@ class App(QMainWindow):
         dlg.exec()
 
 
+    def linkPCR(self):
+        """
+        Links PCR logs from .csv files to relevant submissions.
+        """        
+        home_dir = str(Path(self.ctx["directory_path"]))
+        fname = Path(QFileDialog.getOpenFileName(self, 'Open file', home_dir, filter = "csv(*.csv)")[0])
+        with open(fname.__str__(), 'r') as f:
+            runs = [col.strip().split(",") for col in f.readlines()]
+        count = 0
+        for run in runs:
+            obj = dict(
+                    start_time=run[0].strip(), 
+                    rsl_plate_num=run[1].strip(), 
+                    biomek_status=run[2].strip(), 
+                    quant_status=run[3].strip(),
+                    experiment_name=run[4].strip(),
+                    end_time=run[5].strip()
+                )
+            # for ii in range(6, len(run)):
+            #     obj[f"column{str(ii-5)}_vol"] = run[ii]
+            sub = lookup_submission_by_rsl_num(ctx=self.ctx, rsl_num=obj['rsl_plate_num'])
+            try:
+                logger.debug(f"Found submission: {sub.rsl_plate_num}")
+            except AttributeError:
+                continue
+            if hasattr(sub, 'pcr_info') and sub.pcr_info != None:
+                existing = json.loads(sub.pcr_info)
+            else:
+                existing = None
+            try:
+                if json.dumps(obj) in sub.pcr_info:
+                    logger.debug(f"Looks like we already have that info.")
+                    continue
+                else:
+                    count += 1
+            except TypeError:
+                logger.error(f"No json to dump")
+            if existing != None:
+                try:
+                    logger.debug(f"Updating {type(existing)}: {existing} with {type(obj)}: {obj}")
+                    existing.append(obj)
+                    logger.debug(f"Setting: {existing}")
+                    sub.pcr_info = json.dumps(existing)
+                except TypeError:
+                    logger.error(f"Error updating!")
+                    sub.pcr_info = json.dumps([obj])
+                logger.debug(f"Final ext info for {sub.rsl_plate_num}: {sub.pcr_info}")
+            else:
+                sub.pcr_info = json.dumps([obj])        
+            self.ctx['database_session'].add(sub)
+            self.ctx["database_session"].commit()
+        dlg = AlertPop(message=f"We added {count} logs to the database.", status='information')
+        dlg.exec()
+
+
+    def importPCRResults(self):
+        """
+        Imports results exported from Design and Analysis .eds files
+        """        
+        home_dir = str(Path(self.ctx["directory_path"]))
+        fname = Path(QFileDialog.getOpenFileName(self, 'Open file', home_dir, filter = "xlsx(*.xlsx)")[0])
+        parser = PCRParser(ctx=self.ctx, filepath=fname)
+        logger.debug(f"Attempting lookup for {parser.plate_num}")
+        sub = lookup_submission_by_rsl_num(ctx=self.ctx, rsl_num=parser.plate_num)
+        try:
+            logger.debug(f"Found submission: {sub.rsl_plate_num}")
+        except AttributeError:
+            logger.error(f"Submission of number {parser.plate_num} not found.")
+            return
+        # jout = json.dumps(parser.pcr)
+        count = 0
+        if hasattr(sub, 'pcr_info') and sub.pcr_info != None:
+            existing = json.loads(sub.pcr_info)
+        else:
+            # jout = None
+            existing = None
+        if existing != None:
+            try:
+                logger.debug(f"Updating {type(existing)}: {existing} with {type(parser.pcr)}: {parser.pcr}")
+                if json.dumps(parser.pcr) not in sub.pcr_info:
+                    existing.append(parser.pcr)
+                logger.debug(f"Setting: {existing}")
+                sub.pcr_info = json.dumps(existing)
+            except TypeError:
+                logger.error(f"Error updating!")
+                sub.pcr_info = json.dumps([parser.pcr])
+            logger.debug(f"Final pcr info for {sub.rsl_plate_num}: {sub.pcr_info}")
+        else:
+            sub.pcr_info = json.dumps([parser.pcr])
+        self.ctx['database_session'].add(sub)
+        logger.debug(f"Existing {type(sub.pcr_info)}: {sub.pcr_info}")
+        logger.debug(f"Inserting {type(json.dumps(parser.pcr))}: {json.dumps(parser.pcr)}")
+        self.ctx["database_session"].commit()
+        logger.debug(f"Got {len(parser.samples)} to update!")
+        for sample in parser.samples:
+            logger.debug(f"Running update on: {sample['sample']}")
+            update_ww_sample(ctx=self.ctx, sample_obj=sample)
+        dlg = AlertPop(message=f"We added PCR info to {sub.rsl_plate_num}.", status='information')
+        dlg.exec()
+
+
 class AddSubForm(QWidget):
     
     def __init__(self, parent):

src/submissions/frontend/custom_widgets/misc.py

@@ -2,6 +2,7 @@
 Contains miscellaneous widgets for frontend functions
 '''
 from datetime import date
+import typing
 from PyQt6.QtWidgets import (
     QLabel, QVBoxLayout,
     QLineEdit, QComboBox, QDialog, 
@@ -10,10 +11,11 @@ from PyQt6.QtWidgets import (
     QHBoxLayout,
 )
 from PyQt6.QtCore import Qt, QDate, QSize
+# from submissions.backend.db.functions import lookup_kittype_by_use
 # from submissions.backend.db import lookup_regent_by_type_name_and_kit_name
 from tools import check_not_nan
 from ..functions import extract_form_info
-from backend.db import get_all_reagenttype_names, lookup_all_sample_types, create_kit_from_yaml, lookup_regent_by_type_name#, lookup_regent_by_type_name_and_kit_name
+from backend.db import get_all_reagenttype_names, lookup_all_sample_types, create_kit_from_yaml, lookup_regent_by_type_name, lookup_kittype_by_use#, lookup_regent_by_type_name_and_kit_name
 from backend.excel.parser import SheetParser
 from jinja2 import Environment, FileSystemLoader
 import sys
@@ -329,9 +331,13 @@ class ImportReagent(QComboBox):
             else:
                 if len(relevant_reagents) > 1:
                     logger.debug(f"Found {prsr.sub[item]['lot']} in relevant reagents: {relevant_reagents}. Moving to front of list.")
-                    relevant_reagents.insert(0, relevant_reagents.pop(relevant_reagents.index(prsr.sub[item]['lot'])))
+                    idx = relevant_reagents.index(str(prsr.sub[item]['lot']))
+                    logger.debug(f"The index we got for {prsr.sub[item]['lot']} in {relevant_reagents} was {idx}")
+                    moved_reag = relevant_reagents.pop(idx)
+                    relevant_reagents.insert(0, moved_reag)
                 else:
                     logger.debug(f"Found {prsr.sub[item]['lot']} in relevant reagents: {relevant_reagents}. But no need to move due to short list.")
         logger.debug(f"New relevant reagents: {relevant_reagents}")
+        self.setObjectName(f"lot_{item}")
         self.addItems(relevant_reagents)
         

src/submissions/templates/submission_details.html

@@ -3,9 +3,10 @@
     <head>
         <title>Submission Details for {{ sub['Plate Number'] }}</title>
     </head>
+    {% set excluded = ['reagents', 'samples', 'controls', 'ext_info', 'pcr_info'] %}
     <body>
         <h2><u>Submission Details for {{ sub['Plate Number'] }}</u></h2>
-        <p>{% for key, value in sub.items() if key != 'reagents' and key != 'samples' and key != 'controls' and key != 'ext_info' %}
+        <p>{% for key, value in sub.items() if key not in excluded %}
             {% if loop.index == 1 %}
                 &nbsp;&nbsp;&nbsp;{% if key=='Cost' %}{{ key }}: {{ "${:,.2f}".format(value) }}{% else %}{{ key }}: {{ value }}{% endif %}<br>
             {% else %}
@@ -62,5 +63,25 @@
         {% endfor %}</p>
         {% endfor %}
         {% endif %}
+        {% if sub['pcr_info'] %}
+        {% for entry in sub['pcr_info'] %}
+        {% if 'comment' not in entry.keys() %}
+            <h3><u>qPCR Momentum Status:</u></h3>
+        {% else %}
+            <h3><u>qPCR Status:</u></h3>
+        {% endif %}
+        <p>{% for key, value in entry.items() if key != 'imported_by'%}
+            {% if loop.index == 1%}
+                &nbsp;&nbsp;&nbsp;{{ key|replace('_', ' ')|title() }}: {{ value }}<br>
+            {% else %}
+                {% if "column" in key %}
+                    &nbsp;&nbsp;&nbsp;&nbsp;{{ key|replace('_', ' ')|title() }}: {{ value }}uL<br>
+                {% else %}
+                    &nbsp;&nbsp;&nbsp;&nbsp;{{ key|replace('_', ' ')|title() }}: {{ value }}<br>
+                {% endif %}
+            {% endif %}
+        {% endfor %}</p>
+        {% endfor %}
+        {% endif %}
     </body>
 </html>

src/submissions/templates/submission_details.txt

@@ -1,6 +1,7 @@
 {# template for constructing submission details #}
-
+{% set excluded = ['reagents', 'samples', 'controls', 'ext_info', 'pcr_info'] %}
-{% for key, value in sub.items() if key != 'reagents' and key != 'samples' and key != 'controls' and key != 'ext_info' %}
+{# for key, value in sub.items() if key != 'reagents' and key != 'samples' and key != 'controls' and key != 'ext_info' #}
+{% for key, value in sub.items() if key not in excluded %}
 {% if key=='Cost' %} {{ key }}: {{ "${:,.2f}".format(value) }} {% else %} {{ key }}: {{ value }} {% endif %}
 {% endfor %}
 Reagents:
@@ -22,6 +23,10 @@ Attached Controls:
 {% if sub['ext_info'] %}{% for entry in sub['ext_info'] %}
 Extraction Status:
 {% for key, value in entry.items() %}
-    {{ key|replace('_', ' ')|title() }}: {{ value }}{% endfor %}
+    {{ key|replace('_', ' ')|title() }}: {{ value }}{% endfor %}{% endfor %}{% endif %}
-{% endfor %}
+{% if sub['pcr_info'] %}{% for entry in sub['pcr_info'] %}
+{% if 'comment' not in entry.keys() %}qPCR Momentum Status:{% else %}
+qPCR Status{% endif %}
+{% for key, value in entry.items() if key != 'imported_by' %}
+    {{ key|replace('_', ' ')|title() }}: {{ value }}{% endfor %}{% endfor %}
 {% endif %}

src/submissions/tools/__init__.py

@@ -1,11 +1,13 @@
 '''
 Contains miscellaenous functions used by both frontend and backend.
 '''
+import re
 import sys
 import numpy as np
 import logging
 import getpass
 from backend.db.models import BasicSubmission, KitType
+from typing import Tuple
 
 logger = logging.getLogger(f"submissions.{__name__}")
 
@@ -116,3 +118,14 @@ def check_if_app(ctx:dict=None) -> bool:
         return True
     else:
         return False
+    
+
+def retrieve_rsl_number(in_str:str) -> Tuple[str, str]:
+    in_str = in_str.split("\\")[-1]
+    logger.debug(f"Attempting match of {in_str}")
+    regex = re.compile(r"""
+        (?P<wastewater>RSL-WW-20\d{6})|(?P<bacterial_culture>RSL-\d{2}-\d{4})
+        """, re.VERBOSE)
+    m = regex.search(in_str)
+    return (m.group(), m.lastgroup)
+