Support for running transform scripts during result updates.

[labkey-klum]

Rationale

This PR includes two main changes:

• Adds the ability to run transform scripts when assay results are updated, currently in the app this is only via grid or bulk (updates via file is not yet supported).
• Fixes the issue discovered during the implementation of adding plate data after the initial import (via re-import) where transform scripts wouldn't run in the proper sequence. The problem is that any merging of existing and new data occurs before a transform script. So if a transform script was needed to parse the new data, the result of the re-import would be only new rows would appear.

Related Pull Requests

• Support for running transform scripts during result updates. #6196
• Support for running transform scripts during result updates. commonAssays#833
• https://github.com/LabKey/limsModules/pull/1055

Changes

• Transform script invocation also occurs in the AssayResultUpdateService which is where updates happen. If the transform script has altered any rows, those will be merged with the original set of results.
• Merging of previous and new run data is now handled in AbstractAssayTsvDataHandler.insertRowData which would be after any transform scripts may have been run.
• Note: By default, existing transform scripts will only run during insert. Until the new UI to configure a script is implemented, a transform script can be run on update by commenting out lines : DataTransformService 94:95.
• Introduced a new replacement token : transformOperation so that a script author can conditionally add logic for different transform operations:

# demo usage of the operation replacement token by raising an error on update
if ("${transformOperation}" == "UPDATE") {
    errorsDf <- data.frame(type=c('error'), prop=c('test property'), msg=c('Invalid operation for update'));

Run info properties for Update

During update, a runProperties.tsv file is still written out for the transform script to parse. While no new properties have been introduced, the properties that are available during update are a subset of the properties available during insert. The list of run properties during update include (see the spec for more details):

• assayName
• assayType
• baseUrl
• containerPath
• errorsFile
• protocolDescription
• protocolId
• protocolLsid
• runDataFile
• userName
• workingDir

[cnathe]

minor: looks like this param is unused and can be removed

[labkey-klum]

good catch, done.

[cnathe]

I'm surprised that the run 2nd param isn't needed here, but maybe it is just for writing the transform run properties (which aren't supported for update).

[labkey-klum]

Yes, lucky for us it isn't. On the way out, it's to create some of the run properties info. Coming back in, we use it to create a protocol application so that in the experiment run graph we see the transform as an additional step (which is very cool). Unfortunately on update, there is no lineage graph that we can tie the operation to so it's not apparent that a transform has been run. I think this is okay for now but something we may want to consider in the future. I think having some derivation protocol would be the wrong way to go here but maybe an audit record?

[cnathe]

what is an example of when this exception would be hit? The ExpData data object here is the specific run data row right?

[cnathe]

would this be if for some reason the update transform script wrote out the same row primary key value to multiple rows?

[labkey-klum]

I can't think of a direct way a script could intentionally do this but if we made a mistake in the processing of script outputs, it would mean we had multiple files that were generated for the result data (and we wouldn't have a good way to pick the correct one), so this feels like it is an unrecoverable state.