CCP4 Interface: The Database Utility

	CCP4i: Graphical User Interface
	The Database Utility

Setting up

The Job List

The Database Menu: View Files from Job; Delete/Archive Files; Kill Job; ReRun Job; Edit Job Data; Preferences; System Administration

The CCP4_DATABASE

The Database Utility maintains a record of all tasks performed in a project. The Interface provides simple tools to archive and restore data files, which can function as a notebook or logbook. The user can add any other information. Thus, it is possible to keep track of a structure determination and refinement process without using a pen and paper.

Setting up

Setting up a Database for a project starts with choosing a Project directory using the Directories&ProjectDir interface. A database directory called CCP4_DATABASE is usually set up as a subdirectory of the Project directory.

The Job List

The Job List window (in the centre of the Main Window of the Interface, coloured salmon pink in the picture) is a reverse order list of all the jobs. The format of each line in the List is:

Job number: Each job is assigned a number at run time
Date or time: Lists the start time of a running job or the finish time. For jobs not run today the date of run is listed
Status: See Introduction
Task: Task name
Title: The title entered by the user in the task window

The Database Menu

The Database menu is the scrollable menu on the right-hand-side of the Main Window, below View Any File (coloured salmon pink in the picture). Most of the options in the menu require that one job is selected from the Job List by clicking the left mouse button on the desired line. If more than one job is currently selected, the action will be applied to the last selected. If no jobs are selected, the action will be applied to the job at the top of the list. More than one job may be selected for the option Delete/Archive Files.
All job selections can be cleared using the F2 function key on your keyboard.

The Database menu contains the following options:

View Files from Job: View LogFile in Web Browser; View LogSummary in Web Browser; View Log File; View Log Graphs; View Command Scripts; List of Input/Output Files
Delete/Archive Files
Kill Job
ReRun Job
Edit Job Data: Read/Edit Notebook; Edit Job Data; Enter Data for External Job
Preferences
System Administration: View Session Log; Configure Interface; Edit Modules File; Install New Task; Run Tests

View Files From Job

All of the files associated with the selected job (log file and all extant input and output files) are listed in a pop-up menu. The selected file is displayed with The FileViewer Utility (unless otherwise stated).

The pop-up menu contains the following:

View LogFile in Web Browser: If the log file is in HTML format, there is the option for them to be displayed in a web browser, including links to documentation and in-line graphs. Note that there may be some delay as Netscape first loads the Java applet to display graphs.
View LogSummary in Web Browser: If the log file is in HTML format, the sections tagged as SUMMARY are displayed in a web browser.
View Log File: Display log file with the FileViewer utility, regardless of the file format.
View Log Graphs: Display the graphs from a log file with The Loggraph Utility.
View Command Scripts: Display the command scripts for the CCP4 programs which were run for the selected job Beware: since these are temporary files written to the scratch area ($CCP4_SCR, which may or may not be the same as your TEMPORARY directory), they will probably not be kept indefinitely on your computer. Also, if you run a job on a remote machine with a different definition of $CCP4_SCR, these files may not be visible even if they do exist.

The remainder of the pop-up menu is filled with all other input and output files associated with the selected job.

Delete/Archive Files

This is a means to explicitly delete temporary files or output which you no longer wish to keep. It is also a means to 'archive' files which means saving a compressed version of the file to the database directory. Note that temporary files are usually written to the TEMPORARY directory and, depending on the setup in your lab, these may be deleted automatically after a period.

The default cleanup level can be set in the Preferences window. The program default is the 'safe' option to 'Delete temporary file' but most users will probably prefer the 'Delete output files and remove from database'. If you have selected more than one job, there is an option at the top of the window to reset the cleanup mode for all selected jobs.

For each of one or more jobs selected from the job list you have the option of several levels of cleanup:

Do nothing: Self-explanatory.
Delete temporary files: Delete the files with extension .tmp created by the run script on the TEMPORARY area. These will be mostly input command scripts for CCP4 programs but there may be other intermediate files. This is the default option. This default can be changed through the Preferences button.
Delete all output files: As above and delete the log file and all output files.
Delete output files & remove from database: As above and remove all record of the job from the database.

The log file and any other extant output files are listed with options to:
Delete, Keep, Archive, Archive&delete, Retain archive, Restore archive etc.
The default option is set to either Delete or Keep, dependent on the cleanup level chosen. This default will be overridden if you change the option for the individual file.

Archiving a file involves saving a compressed version of the file (by default it is compressed with the gzip utility) to the $Project/CCP4_DATABASE directory. The archived file has the original file name, with the extension .gz added.

If there is already an archived version of the file, options to Retain archive (i.e. keep it, no change) or Restore archive (uncompress and copy back to the original file name) are present.

The Restore options in the archive include either deleting the original file or, if not explicitly stated, the original is undeleted.

Kill Job

The Interface will terminate the currently selected job, or the last one if no job is explicitly selected. It will ask you to confirm that you want to kill the job. Any running job will have one process which is running a script and probably a child process which is the running program. The Interface will attempt to kill the child program process but if it can not find a program process after three retries, it will kill the script process. The Job List will show the job status as KILLED. The Interface will kill both local and remote jobs. Whilst killing local jobs usually takes immediate effect, you should expect some delay in killing a remote job.

This operation is very dependent on the operating system; if you have problems, the following information may help. To kill a job, the Interface reads the header of the log file which has the script process ID on the line commencing "#CCP4I PID". It uses the Unix commands 'ps -l' to find the currently running processes and the 'kill -9' command to kill a process. If the attempt to kill a process fails, the Interface should report where it has failed. If you have problems, please report them to CCP4 or look in the script $CCP4I_top/src/local.tcl at the procedures KillScript and KillRemoteScript.

ReRun Job

This provides for speedy rerunning of jobs that FAILED or ones you wish to repeat with minor changes of parameters.

Select a job from the Job List before clicking ReRun Job.

A Rerun Job window comes up. The task window(s) for the job may be reviewed and edited before rerunning the task.

Edit Job Data

The Edit Job Data menu contains the following:

Read/Edit Notebook

This is a means to save any comments about a job. Select a job from the Job List first.

Each job has its own notebook file in the database directory which is created when you select Read/Edit Notebook. The file is created with header information indicating to which job it belongs. You should edit in your comments after the header information.

Edit Job Data

There is an option to edit the data stored in the database. This might be useful if you rename a file and want to update the information in the database or if you want to change (or add) a title for a job.

Enter Data for External Job

There is also an option to Enter Data for External Job which will allow you to input information on jobs run outside the database. For example this might be useful for recording what you did using molecular graphics. You can record the names of input and output files and also then use the Edit Notebook facility to write up what you did in more detail.

Preferences

In order to change Interface defaults, e.g. where output files go, use the Preferences button.

System Administration

View Session Log

Each CCP4i run keeps a session log file recording only significant actions such as jobs run and files deleted. It may also contain error diagnostics which might be useful to the developer but probably not to the user. The log file is saved as $HOME/.CCP4/CCP4_session.log. Each time CCP4i starts up, any existing file of that name is overwritten. The session log file can be viewed from the Interface using the 'View Session Log File' option on the right hand side of the Main Window.

Configure Interface

The Interface can be configured to use different system resources such as web browsers, printers and remote machines. Most users should not need to change these as they should have been set up when CCP4i was installed. The one option they may like to change is size of fonts.

A detailed description of how the Interface can be configured to the user's preferences, can be found elsewhere.

The user can save the setup with the Save button and should usually choose the Save to user's home directory option. The Interface will be updated to reflect the new configuration but it may be necessary to close and reopen a window to see the new configuration.

Edit Modules File

The Edit Modules File button gives a window to edit the modules and task references which appear in the menus on the left-hand side of the main CCP4i window, and which are stored in the modules.def file.

Install New Task

The Install New Task button gives a window to copy new task definitions (from a Task source) into the current installation. There are also options to uninstall tasks which have previously been installed using this function.

Run Tests

See separate document on how to run test jobs.

The CCP4_DATABASE

All the database information is saved in the CCP4_DATABASE sub-directory of the Project directory. It is kept here so that if the Project directory is moved (between computers/labs etc.), the database is less likely to be left behind. In this directory are:

database.def: The information in the Job List and additional information is saved in this file when the Save Database to File option is used. The file has the standard .def file format.
Parameter def files: For each task run the parameters are saved in a .def file. The file name is constructed from the job number (jobid) and the name of the task (task), as follows:
jobid_task.def
This file will be used by the ReRun Job option.
Task.def files: If the Save&Restore option on a task interface is used to Save a Current Task Default, a file called taskname.def will be saved in the database directory. This file will be used to initialise the parameters whenever that task interface is used.
Archived files: All archived files have the same name as the original file with the extension .gz (assuming your system uses the gzip compress mechanism). These files are accesed via the Delete/Archive Files button.