The Winner is Jichun Wang

Dec 4, 2011 by Clinovo    1 Comment     Posted under: Code, Monthly Contest, New Technologies, Tips & Techniques

The winner of our last programming challenge is Jichun Wang. Jichun is a Sun Microsystems alumnus, now a senior software engineer at Synopsis. He uses perl to call a Google RESTful API and to parse the JSON outcome from google to get the result counting of google search. You can find his program here on my SAS_ACADEMY group.

Because the API he used is deprecated, you can find there is a significant difference in the counting using this API and that you get by googling directly in browser; however, he got the order correct!

Dive into CDISC Express (5) : Generate and validate SDTM domains and define.xml

Sep 5, 2011 by Clinovo    No Comments    Posted under: Best-Practices, CDISC, Clinical Data Management, Code, Tips & Techniques

Here is the fifth part of Dive into CDISC Express.

Part one | Part two | Part three | Part four

The following tasks, such as generating SDTM domains and define.xml, need just some clicking button work in CDISC Express using a well designed mapping file. Few words needed due to the software.

Step 3 of 6: Validate mapping file (Validate_Mapping_File.sas)

It would be back and forth to design, validate then modify and re-validate the mapping file. And sure finally, you will get all the work done, at least no syntax error (how to avoid semantic errors is upon your domain knowledge). A validated mapping file, named mapping.xls will be copied to …\ doc\Mapping file – validated version\ from the working file, tmpmaping.xls. You will see

The corresponding log file in folder …\ log\

A report in …\results\Mapping Validation\, named Mapping_validation.html

Also the temporary datasets in …\tempdata\ and …\temp\:

Step 4 of 6: Generate SDTM datasets (generate_SDTM.sas)

If mapping file is OK, generating SDTM domains is just clicking the button. After submitting the codes, you will see the log file, reports, SDTM datasets and temporary datasets in corresponding folders:

Step 5 of 6: Validate SDTM datasets (Validate_SDTM_Domains.sas)

The outputs files of validating SDTM datasets are all located in C:\Program Files\CDISC Express\SDTM Validation\:

Step 6 of 6: Generate Define.xml and xpt (generate_Definexml.sas)

Get the final define.xml file and SAS transport files (.xpt):

Recommended reading and action taken

For a quick start and deep understanding, you could read the official documentations in the following sequence:

C:\Program Files\CDISC Express\documentation\FAQ.htm

C:\Program Files\CDISC Express\documentation\Quick Start.htm

C:\Program Files\CDISC Express\documentation\User guide.htm

A video tutorial would be also helpful:

C:\Program Files\CDISC Express\documentation\videotutorial.htm

A must-read conference paper, An Excel Framework to Convert Clinical Data to CDISC SDTM Leveraging SAS Technology by Sophie McCallum and Stephen Chan of Clinovo, supplies a wonderful discussion the architectures of CDISC Express:

http://www.lexjansen.com/pharmasug/2011/ad/pharmasug-2011-ad08.pdf

Dive into CDISC Express (4) : Data manipulation techniques

Aug 29, 2011 by Clinovo    No Comments    Posted under: Best-Practices, CDISC, Clinical Data Management, Code, Tips & Techniques

Here is the fourth part of Dive into CDISC Express.

Part one | Part two | Part three

3. Data manipulation techniques in CDISC Express

CDISC Express supplies relative rich sets of data manipulation techniques assembling with SAS languages used for data mapping. Following is a not limited listing and I will keep it updated.

3.1 Reference one dataset

A raw dataset name appear in “Dataset” column indicate a “set” operation in SAS.

All dataset options can be used when referencing a dataset, such as

siteinv(drop=invcode)

siteinv(rename=(invcode=inv))

siteinv(where=(invcode ne “”))

You can also reference an external dataset. You should incorporate the external file in spreadsheet with name beginning with an underscore, “_”, and “_visits” in this case:

Then you can use it in any domains needed, e.g., TV domain:

There is a macro %cpd_importlist used to import the external dataset, “_visits”. Again, this macro roots in C:\Program Files\CDISC Express\macros\function_library\.

Using a macro call to re-sharp or modify an input dataset offers great flexibility referencing data. We will also discuss the benefits later on.

3.2 Assignment

You can assign a number, string and a dataset variable with any valid SAS functions to a SDTM domain variable in “Expression” column.

Sometimes a temporary variable needed for later calculation. You can produce such temporary variable in “Dataset” column with an assignment in the “Expression” column just similar with any other domain variables. Two differences: first, such temporary variables named begin with an asterisk, “*”; second, all temporary variables will not be included in the final domain. Once created, such temporary variables can be used for any other expressions.

There are three special symbols used in “Dataset” column of CDISC Express. Asterisk, “*” indicates a temporary variable, while other two are

Tilde, “~” : indicate a variable used for supplemental domain (SUPPQUAL).

Number sign, “#”: indicate a variable used for comments domain (CO).

Another symbol, at sign, “@”, used in “Expression” column, indicated referencing a variables produced before:

In this case, “AGEU” uses “AGE” as input, while “AGE” is calculated before. “@AGE” just indicates the dependency. In concept, it looks like the “calculated” option in SAS PROC SQL:

proc sql ;

select (AvgHigh – 32) * 5/9 as HighC ,

(AvgLow – 32) * 5/9 as LowC ,

(calculated HighC – calculated LowC)

as Range

from temps;

quit;

3.3 Match-merging

We already got a math-merging example before. If “all” appears as a dataset in the “Dataset” column, all the previous datasets should be merged first for later processing by the common key specified in “Merge Key” column. If no key assigned, patient ID is used by the system.

CDISC Express also supports two types of join, inner join and outer join (left, right, full) using data steps. The implementation has slightly difference with standard SQL, but the ideas are same.

We add a new column, “Join”, usually beside the “Merge Key” column.

There are two values for “Join”, “O” or “I” while “O” stands for “outer join” and “I”, “inner join”. A join indicator “I” equals a dataset option “in=” in action while “O” means no. Use the above as illustration, the corresponding SAS codes behind look like

data temp;

merge demog(in=a) siteinv(in=b);

by sitecode;

if b;

run;

This is so called “right outer join”. The combination of “I” and “O” in these two datasets can perform all the four types of join, one inner join and three outer join:

As we could see, if no “Join” column specified, CDISC Express will perform inner join by default.

So far CDISC Express cannot support multiply merge keys. For example, the following file is illegal currently:

The developer Romain indicated that such enhancements would be raised to the next round of product road map and he also proposed a work around. To use multiple keys for merging, we can create a temporary variable holding such multiple keys as a concatenation then this temporary variable can be used as a single merging key.

3.4 Concatenating

Above we discussed lots about “merge” operation in CDISC Express. This section dedicated for “set” operation. We already know how to “set” one dataset for referencing, but how to “set” multiple datasets, i.e, “Concatenating”?

Symmetrically, an “all” appears in “Dataset” column indicating merging operation, an “all (stack)” indicates concatenating operation:

The above file can be also translated to SAS codes for better understanding:

height;

set vtsigns(where=(height ne .));

VSTESTCD=”HEIGHT”;

VSTEST =”Height”;

VSORRES =put(height,best12.);

VSORRESU=”cm”;

VSSTRESC=put(height,best12.);

VSSTRESN=height;

VSSTRESU=”cm”;

run;

data weight;

set vtsigns(where=(weight ne .));

VSTESTCD=”WEIGHT”;

VSTEST =”Weight”;

VSORRES =put(weight,best12.);

VSORRESU=”kg”;

VSSTRESC=put(weight,best12.);

VSSTRESN=weight;

VSSTRESU=”cm”;

run;

data vs;

set height weight;

STUDYID =study;

DOMAIN =&domain;

USUBJID =%CONCATENATE(_variables=study sitecode patid);

VSSEQ =%SEQUENCE();

. . .

run;

3.5 Transpose

Clinical SAS programmers do lots of transpose operation to re-sharp the raw data to fit the CDISC standards. Currently there is no explicit guide in CDISC Express on how to transpose, but this is not the end of story.

There are two types of transpose:

Type I: from a wide dataset (more variables, less observations) to a long dataset (less variables, more observations), e.g. transposing a one-row-per-subject datasets to a multiple-row-per-subject dataset

Type II: from a long dataset (less variables, more observations) to a wide dataset (more variables, less observations), e.g. transposing a multiple-row-per-subject dataset to a one-row-per-subject datasets

As good practices, in SAS we always use data steps with “output” statement to perform type I transpose and use PROC TRANSPOSE for type II. Although CDISC Express doesn’t support transpose operation in an explicit way, at least you can perform type I transpose and surprisingly we already saw it before!

Just back to section of concatenating. The example is taken from C:\Program Files\CDISC Express\studies\example2\.

We can see the input data vtsigns is typical wide table (more variables, less observations):

And the final domain VS is a typical long table (less variables, more observations):

So obviously, such concatenating operation just did a wonderful type I transpose, from a wide table to a long table! More often, the compact SAS codes for type I transpose look like:

data vs;

set vtsigns;

if height ne . then do;

VSTESTCD=”HEIGHT”;

VSTEST =”Height”;

VSORRES =put(height,best12.);

VSORRESU=”cm”;

VSSTRESC=put(height,best12.);

VSSTRESN=height;

VSSTRESU=”cm”;

output;

end;

if weight ne . then do;

VSTESTCD=”WEIGHT”;

VSTEST =”Weight”;

VSORRES =put(weight,best12.);

VSORRESU=”kg”;

VSSTRESC=put(weight,best12.);

VSSTRESN=weight;

VSSTRESU=”cm”;

output;

end;

. . .

run;

3.6 All others: use macro!

Now we discussed almost all the common data derivation techniques in programmers’ daily life and the corresponding implementation in CDISC Express. At least we have one question unsolved: how to perform type II transpose, i.e. from a long table to a wide table?

It would be an open question for the developers of the application. But we can also solve this problem in current framework: use macro, customized macro. You can use macros in “Expression” and “Dataset” column. Macro used in “Dataset” column returns a dataset, while macro in “Expression” column returns series of string: that’s the basic structure you should consider when customize your own macros. For more, you can reference the macros in C:\Program Files\CDISC Express\macros\function_library\. For example, &concatenate used in “Expression” column; &cpd_importlist in “Dataset” column.

So it would be convenient to create temporary datasets using macros imbedded type II transpose operation in “Dataset” column. Every thing SAS can do, you can also implement it in CDISC Express. Just use macros, in “Expression” and “Dataset” column accordingly.

The raw data varies according to trial design and clinical data capture system and procedures. It is impossible and impractical to anticipate the CDISC SDTM converter such as CDISC Express to map all the data just clicking a button. The introducing of CDISC Express doesn’t keep programmers away. It just keeps most of the trivial work away from programmers’ daily life and let them more concentrated on creative work and be productive and efficient.

Following would be the close of such pages.