Auditing Electronic Field Data - ACS Symposium Series (ACS

Chapter 8

Downloaded by NORTH CAROLINA STATE UNIV on December 29, 2017 | http://pubs.acs.org Publication Date: August 1, 2002 | doi: 10.1021/bk-2002-0824.ch008

Auditing Electronic Field Data Renée J . Daniel Perspective Consulting, 1034 Greystone Lane, Sarasota, FL 34232

Historically all Good Laboratory Practice (GLP) compliant field data have been collected manually and documented in an organized manner in a Field Notebook prepared by the Sponsor or Field Management Company. The paper Field Notebooks have generally been well organized and logistically easy to audit. In the past several years software programs have been developed that are capable of generating GLP field data electronically. Programs currently in use are American Agricultural Services (AASI) Advantage™ and Astrix FieldNotes. Electronic data have provided a challenge for Quality Assurance (QA) Auditors who have been trained to audit paper data. They have also provided an opportunity for learning how to deal with the idiosyncrasies of auditing electronic data. There are several critical areas that need to be addressed when working with electronically generated field data.

Training Management at a GLP facility must assure that personnel have adequate education, training and experience to operate the program and handle any unforeseen circumstances that arise due to the technology (i.e., software, hardware). Although training of GLP Study Personnel is extremely important, Management cannot overlook the fact that it is critical that QA Auditors receive training as well. Without thorough knowledge of software operation, QA

44

© 2002 American Chemical Society

Garner et al.; Capturing and Reporting Electronic Data ACS Symposium Series; American Chemical Society: Washington, DC, 2002.

45


cannot adequately audit the electronic field data. Attendance at an outside course (e.g., software developer) is highly recommended; however, training can also be obtained in-housefroma previously trained individual if resources are an issue. During an audit, QA should assure that training is adequate and documented.

Equipment Since GLP data must be entered directly and promptly, it is usually necessary to take a laptop to the field for data entry during events. Logistical issues must be considered such as durability, environmental conditions, adaptation for viewing the screen in sunlight, backup power and system security while in the field. Laptop in Field vs. Desktop in Office Although not as desirable, there may be situations where a desktop is used instead of a laptop such as in areas with intense heat or other environmental conditions, which preclude the use of a laptop in the field. In these cases, communication systems must be in place (e.g., radios, cell phones) to assure that data are recorded directly and promptly. It is more difficult to audit in these situations since QA has to assure that field procedures follow the protocol and Standard Operating Procedures (SOPs) and that the data are properly recorded. This may require QA to take notes in the field of the values recorded electronically which will need to be checked after the application/sampling. QA may also want to inspect morefrequentlysuch that the portion of the work conducted in the field is inspected separately from the data entry in the office. For the data entry inspection QA would listen to the communication of the values and assure that they are entered accurately and in the right area of the electronic notebook.

Power Considerations Some laptop batteries may last long enough to perform calibration and application. However, there may be unforeseen circumstances with the application, which may require longer access to the laptop such as a re-mixing situation or waiting for the wind to diminish. Also, the battery may have been


46

inadvertently not charged correctly which might shorten the available use time significantly. In order to assure that data are not lost, a source of backup power, such as an adapter that plugs into the vehicle cigarette lighter, should be available.


Backing Up and Transferring Data Systems for backing up data are also required. There are a variety of systems, which will meet the requirement, and the decision should be based on logistics, resources and systems already in place. Some appropriate back up systems arefloppydisks, zip drive, another hard drive, network server and CDROMs. Data should be backed up every day that original raw data are entered into the program. Ideally, backup data should be stored in afire-resistantarea or a location separatefromwhere the primary data are stored. Another equipment consideration relates to the transfer of data. Since study data will usually need to be transferred to the Study Director at regular intervals (typically after each application, sampling and shipment), the transfer rate is critical. Field sites, which are relatively isolated, may have poor telephone line quality and thus the upload can take hours. This will also be the case for downloading software updates. The upload/download time may be an important consideration if this will tie up the only telephone line or fax line. If available in the area, it is highly recommended that a faster form of transfer be obtained such as cable modem or DSL. Since telephone line quality is a common problem, it is recommended that the software developers send updates on CDROMs.

Hardware and Software Security For hardware security, the laptop should be kept in as secure a location as a paper notebook would be kept, whether in the field or in the office. In addition, the environmental conditions are particularly important and temperature extremes and high humidity must be avoided. Software security is maintained by using individual passwords. During a study inspection, QA should assure that the person entering the data is the person that has logged in to the system. If not, this is a GLP compliance issue since die person entering the data will not be die person identified as such in aie electronic data. QA personnel should have "read only" access to the data in order to avoid any inadvertent changes.


47


Validation and User Verification To conform to GLP requirements, all systems generating data electronically must be validated. For software currently in place, this is done in steps. The software developer thoroughly validates the program prior to its release. Sponsors also perform additional validation using their own protocols. The responsibility of the Field Site user is to verify that the program works on the system(s) to be utilized at their site. It is important to note that this verification must be done at the beginning of the season (prior to doing any field studies with die system) and following every software or significant hardware change. QA should check during an audit to assure that all systems have been adequately verified for the most current software build and that each verification has been properly documented. Current build numbers can be obtained for the software developer. In addition, QA should assure that the verification documentation is properly archived. FieldNotes user verification can be done by following the SOP written by Astrix which can be downloadedfromthe Astrix website. The SOP contains a script that is typed into the computer system. If the printout matches the SOP script then verification is complete and a record of this should be placed in the facility archives. This process takes an hour or less. For FieldNotes, calculations are not verified at the field site since it is felt that the software developer and Sponsors have already adequately performed validation of these steps. The user is simply assuring that the program operates on their system(s). It is important to note that the SOP script must be typed in exactly as written in order to confirm the printout accuracy. This should be checked by QA. For Advantage™, more extensive field site verification is conducted. A field site notebook is used which verifies every step in the data entry process. Following this procedure, a form is completed and returned to AASI where it is checked to assure that verification was properly conducted and documented. Due to the extensiveness of this verification, it takes approximately two hours to perform.

Maintenance and Repair Records Although computer equipment is used differently than other field equipment, it is generating raw data so a maintenance logbook is required. All maintenance and repairs to the computer system should be recorded. For example, software and hardware maintenance records would include, but not be limited to, system/program updates, disk scans, and defragmentations (e.g., preventative maintenance). Repair records would include, but not be limited to, calls to the software developer about problems/bugs and steps necessary to fix any software or hardware problems. The documentation must include whether American Chemical Society Library 1155 16th St., N.W. Garner et al.; Capturing and Reporting Electronic Data Washington, DCSociety: 20036Washington, DC, 2002. ACS Symposium Series; American Chemical

48 the procedure was routine and followed SOP and for non-routine must include nature of defect, how and when discovered and corrective action taken.


Standard Operating Procedures Since there are procedural differences between handling paper and electronic data, additional SOPs are required. Some of the areas that should be covered are: • • • • • • • • • •

Raw data definition; Assignment of users and access level; Security; Computer maintenance; Raw data backup method,frequencyand storage location; Procedures to be followed in the event of computer failure or unavailability; Trial receipt and transfer of incremental andfinaldata; Amendment/deviation documentation and notification procedures; Plot diagram generation; QA audit procedures.

Referring to a user's manual may cover some of the areas. Since some of the areas may overlap current facility SOPs, QA should assure that there are no discrepancies between electronic data SOPs and other facility SOPs. The electronic data SOPs must have management approval, should be sitespecific, where appropriate, and ideally should follow die format for facility SOPs. The electronic data SOPs do not necessarily need to be in the same binder or have the same numbering system as the rest of the feeility SOPs. However, it is critical for GLP compliance that they be accessible to those who need to follow the SOPs and to auditors. QA should assure that all required electronic data SOPs are in place, approved by management and are being followed. It is important to note that while in the past it was acceptable for contract field sites to use Bayer SOPs for FieldNotes studies, this practice will not work if FieldNotes is being used for another Sponsor. In addition, Bayer SOPs are lacking the SOP for site (user) verification, which needs to be in place at the Contract Field Site. If the Astrix SOPs for FieldNotes are downloaded, assure that all requirements of the SOPs will be followed. Ideally these SOPs should be used as a template with modification to meet facility needs. The downloaded SOPs should be carefully read prior to implementation to assure that there are no


49


discrepancies between die SOPs and die facility SOPs already in place or die practices in place for studies with electronic data. The AASI approach to electronic data SOPs is to provide guidance to the contract field sites, if requested. This includes direction on areas to be covered in SOPS and suggested content

Auditing Issues Preparation In order to effectively audit studies where data are generated electronically, QA should create a special checklist or add electronic issues to a current checklist. The checklist should include items such as assuring that build is current and validated and that maintenance is documented. Prior to conducting audits, QA should review the paper protocol plus any documented amendments and deviations. QA should assure that the paper protocol and amendments will be readily available during events in case they need to be referenced for information. The Field Investigator (FI) should enter amendments electronically if the system allows and this should be checked by QA. Any electronic entries previously made should be reviewed in advance if possible. It is a good idea for QA to print blank forms prior to going to the field since it may be hard to follow the screens during electronic data entry. For Advantage™, this can be done directlyfromthe program and for FieldNotes blank forms can be downloaded from the Astrix website. Printing of blank forms in advance is also a good idea for the FI, in case there is a problem with the electronic system. QA should also assure that backups are being done as required by protocol and/or SOP and that any required incremental updates are being sent to the Study Director within the allowed timeframe.

Paper vs. Electronic Data Audit Some QA Auditors prefer to audit printouts of the electronic data rafter than the electronic data itself. This may be partially due to lack of familiarity with electronic auditing and partially due to unavailability of software to QA. There are several arguments in favor of QA auditing the electronic, rather than the paper data. One reason is because there may be perceived compliance issues related to the printouts that do not exist in the electronic data (e.g., initials do not print on hard copy). Another is that QA may not be given printouts of all of the patinant data such as site logs. If QA does choose to audit the printouts, the site logs, audit trail and notes pages must be printed and audited as well. A third



50

reason is that die FI does not usually organize printouts as well as a paper notebook would be organized which may cause confusion. In some cases, die printouts may not indicate the actual plot identification (e.g., treatment 3), which could lead to confusion when auditing a study with several plots treated the same day with different rates. In order to audit the electronic data, QA must either have their own copy of the software or use the computer where the software is installed. In the case of FieldNotes, the latter may be more practical since the former would require the purchase of the program for QA. With respect to Advantage™, the program is available without charge to any Quality Assurance Auditor that needs it so resources should not be an issue. When auditing the electronic data, it may be somewhat difficult to follow the whole study since documentation cannot be spread out on a desk. For this reason, it is recommended that certain pages be printed out for easier reference such as the audit trail and notes pages.

Critical Areas to Consider The software programs perform calibration/application calculations and the calculation process is validated/verified. However, for added comfort level with the programs and the data, it is strongly suggested that QA conduct an independent check of calculations. There are scene electronic data that are directly entered by the FI and some that are entered automatically by the program (e.g., chronological log entries). QA should check not only the directly entered data, but also the computerentered data. In order to conduct a thorough data audit QA should make an effort to determine which data for each software program are direct entry and which are automatic. During a raw data audit, QA should assure that notes and various descriptions (e.g., sampling method) are clear and thorough. In addition, the audit frail should be checked for clarity and to assure that the original entry can be determined as required by GLP. Also, it is critical for QA to assure that all entries "make sense", especially in connection with event times and dates. With electronic notebook studies, there will be paper data to audit as well. This will include facility data (e.g., weather data, equipment maintenance records, temperature storage logs) and study specific data (e.g., axes, emails, paper notes). If any of the paper data has been transcribed into the electronic notebook, this should be checked by QA to assure accuracy. QA should assure that all paper data have adequate identification (e.g., study and trial numbers) and that all of the pertinent paper data or exact copies are sent to the Study Director for archiving.


51

Potential Concerns


Plot Maps Many plot maps are generated in a separate program (e.g., Microsoft Paint) and imported into die field software program. If there is any information on the electronic plot map that was not written elsewhere first (e.g., plot dimensions, slope, row direction), the electronic plot map becomes original raw data and entries must not be obscured if changes need to be made. Current software programs do not allow for easy viewing of a previous plot map once it has been changed, hi FieldNotes, for example, the audit trail will indicate that the change isfroman old image to a new image. The software developers may be able to retrieve the old image (plot map) in die program; however, it is not readily accessible to the FI. For Advantage™, plot diagrams must be created in a separate program and imported. If changes need to be made, the diagram must be removed and reimported which will create an audit trail indicating that the diagram has been replaced. The file for the previously imported map must be retained in order to view the original entry. If a slope change is made on the plot map in FieldNotes, the audit trail will have a numerical entry for the change (e.g.,from90 to 200 instead of Ε to W). This is due to the fact that the plot map is a picture and not text. The software developers will know how to translate the numbers but this information will need to be available during EPA audits as well. There are ways of which the FI and QA should be aware to make the plot map creation and modification procedures GLP compliant. Any original data (e.g., dimensions, slope, row direction, distance to markers, wind direction, buffer distances) should be entered first in text form into the field software where an audit trail will exist. If this is not possible (i.e., program design does not allow it) the original entry can be made onto a properly labeled sheet of paper or form that will be sent with the raw data. If this does not occur and actual raw data are enteredfirstinto a program, such as Microsoft Paint, this feet should be included as an exception on the compliance statement. For GLP compliance it must be assured that when the plot map needs to be changed, the original entry can still be determined during an EPA audit. This can be done by printing the original plot map prior to changing it and sending the printout (with explanation) along with the raw data. If this is not practical, a note should be attached to the changed electronic plot map indicating the exact original entry (e.g., row direction changedfromN-S to S-N). If a map is scanned and imported into the program, the original scanned map is the raw data and should be saved either as facility data or study-specific data, as applicable. When a plot map is drawn separately and imported, QA should assure that the imported map matches the original (e.g., all lines visible).


52 In situations, where some of the imported map is not clear, the original electronic (e.g., GIF) file should be printed and saved in case of a future EPA audit.


Data Entered Late If there are any situations where data need to be entered late and will appear unusual, these instances need to be thoroughly explained. For example, the battery dies on the laptop, there is no adapter available and only part of the calibration was done before the laptop shuts off. The FI will be transcribing some data later that day or night The program will time-stamp some data as the event occurred and some data at the time of transcription. The differences in the entry times will be atypical and should be explained to prevent perception of "creating data in the office". The explanation can be done simply by attaching a note to the transcribed data indicating the situation and referencing the source of the transcribed data.

Test Substance Tracking For FieldNotes, test substance tracking may be difficult when one container is used for several studies. The general log for test substance will automatically track usage at each application. For this reason, if QA is auditing the individual study data the numbers will not seem accurate. For example, 50 grams of chemical A is received. Study 15 application is made on 07/08/00 and 10 grams of test substance are used. The general log will indicate that 40 grams remains. Study 25 application is made on 07/10/00 using 20 grams so this amount is subtractedfromthe chemical inventory in the general log. When the FI goes to make the second application to Study 15 on 07/15/00 the amount of chemical remaining will be indicated in the log as 20 grams before the application is made. Since only 10 grams out of 50 were used for that study, QA may expect to see 40 grams remaining instead of 20 grams. If QA is trying to track use, this may be confusing. QA should be made aware of this during the audit. Consequently, it is highly recommended that a separate container of test substance be sent for each trial and/or study in order to facilitate tracking during the audit

Generic Menu Choices Many of the entries in both FieldNotes and Advantage™ are made using drop-down menus. The choices are not always the most accurate for the



53 situation at the field site (e.g., sprayer types, nozzles, crop stage). QA should check during the raw data audit to assure that the choices made are an accurate reflection of the practices utilized. It may be necessary to attach a note to the pertinent page to clearly explain certain situations such as unique nozzles or different sized nozzles for airblast applications. Explanations may also be required for airblast applications. Due to die variety of airblast calibration procedures utilized by field sites, the electronic forms may not correspond with the way calibrations are conducted at the site For example, if the form only allows for entry of total calibration output volume but the FI collects individual nozzle outputs, this should be indicated in an attached note including documentation of the individual nozzle outputs. This situation may also occur if an in-lineflowmeter is used for airblast calibrations where the initial volume is not typically noted. If the form requires entry of initial volume, this entry can be given as an estimate with a note attached explaining the situation.

Conclusion QA must be aware of the issues and requirements for electronic data in order to adequately assess data accuracy and GLP compliance during inspections and raw data audits. Several significant questions to keep in mind are "What are the original raw data?", "Are the electronic data complete, accurate, and GLP compliant?" and "Have all of the original raw data or exact copies been sent along with the study data for archiving?" In conclusion, as technology advances and the use of electronic notebooks for field studies becomes more prevalent, QA Auditors will need to comprehend not only the technology but also specialized electronic data auditing techniques: Techniques that may be critical to the GLP compliance of the study.


Auditing Electronic Field Data - ACS Symposium Series (ACS

Recommend Documents