Start With Study-Level Roles, Not Company-Level Promises
The first demo can be deceiving. A clinical trial data platform may look clean and fast, but the real test is whether access is scoped to the study, not just to the customer account. Ask the vendor to show how a person is added to one study as a viewer, collaborator, or owner. Then ask what happens when that person belongs to two studies with different teams. Can they see both studies by default, or only the ones they belong to? The answer tells you whether the platform treats study-level roles as a first-class control or as an afterthought.
Ask whether roles are assigned per study or inherited from an organization-wide group. If a person moves from one study to another, does their old access disappear automatically, or does it linger until someone remembers to remove it? Those are not edge cases in clinical research; they are everyday operational facts. Push for specifics on unblinding, too. A role that can see unblinded content should not be the same as a role that can see only blinded data. Ask whether unblinded files, records, and search results require a separate grant. If the vendor says “we hide it in the UI,” ask for the server-side rule. The right answer is that without the grant, those files and records stay hidden, whatever a person’s role. Also ask about sign-in and session behavior. A platform that sends a code by email for every sign-in and ends inactive sessions after 30 minutes is easier to defend than one that relies on long-lived passwords.
Ask Where the Data Lives and How the Audit Trail Is Built
“Cloud-hosted” is not an answer. Ask whether each customer runs in a separate AWS account with its own database and storage. Separation matters because it limits the blast radius of a mistake and makes access reviews simpler. Then ask about deletion. A platform built for clinical study data should not quietly remove records. Ask whether records are retired or superseded, and whether every file revision is kept. If the vendor says “we can delete on request,” ask who can request it, what evidence remains, and how the audit trail records the change.
The audit trail should be append-only, not a table that administrators can edit. Ask which events are written to the log: membership changes, role changes, unblinding actions, and record changes. Then ask how the log is reviewed. A complete audit trail is only useful if study teams can find the relevant entries without exporting everything to a spreadsheet. Ask whether the platform keeps changes to roles and unblinding separate from ordinary data edits, and whether the log shows the before and after state. A clinical trial data platform should make these answers easy to verify, not just easy to say.
Data-Cut Versioning and Blinding Grants Are Not Optional
Clinical study data arrives in cuts. Ask how each cut is stored. A strong answer is that every upload becomes a new revision with its own checksum, nothing is overwritten, and each person chooses which cut they are looking at. Then ask for compare mode. Can you open a dataset, filter to the subjects you care about, and see what changed between two cuts? Can you do that without creating a copy or asking a programmer to write a diff script? If the vendor cannot demonstrate this on a real study, the platform will soon become a folder of files with version names that no one trusts.
Blinding is the other side of versioning. Ask whether unblinded content needs its own grant, separate from a study role. Ask what happens to search results and document pages when a person lacks that grant. The right answer is that those files, records, and search results stay hidden, whatever a person’s role. Also ask how notes and issues are attached. A note on a single form, field, row, or document page is more useful than a comment in a separate tracker. And ask whether protocols, SAPs, and annotated CRFs are searchable page by page next to the data they describe. These are not luxury features; they are how study teams review data together without losing context.
Sandboxed Assistant Access and Pre-Use Benchmarking
If the platform includes an assistant that answers questions by writing and running code, ask where that code runs. The answer should be an isolated sandbox with read-only, short-lived access to one study. Ask whether the assistant can see other studies, whether it can write to the data, and how long its access lasts. Then ask whether the assistant shows the code it ran alongside the tables and charts. A statistician should be able to check every step, not just accept a number. If the assistant hides its work, the platform is asking you to trust a black box with your study data.
Benchmarking is the question most vendors skip. Ask how the assistant is tested before it reaches a study. A credible process uses real clinical analysis tasks with known right answers, and it includes planted traps such as partial dates, a superseded SAP, or an analysis dataset that is quietly wrong. Ask whether each assistant version or instruction change is tested as shipped, with the same tools and code sandbox it would use in the product. Ask whether every case has a reference result and is scored by exact checks and an independent judge. Finally, ask when the benchmark is rerun. A new assistant version or a change to its instructions should be benchmarked before it reaches your study. If the vendor cannot describe the test set, the traps, and the rerun trigger, you are not buying a validated workflow; you are buying a demo.
- Are roles assigned per study, with viewer, collaborator, and owner levels?
- Does unblinded content require its own grant beyond a study role?
- Is each customer isolated in a separate AWS account with its own database and storage?
- Are records retired or superseded rather than deleted, with every file revision kept?
- Is there an append-only log for membership, role, unblinding, and record changes?
- Does every data cut get its own checksum and a compare mode between cuts?
- Does the assistant run in an isolated sandbox with read-only, short-lived access to one study?
A Pampas-raised agronomist turned Copenhagen climate-tech analyst, Mat blogs on vertical farming, Nordic jazz drumming, and mindfulness hacks for remote teams. He restores vintage accordions, bikes everywhere—rain or shine—and rates espresso shots on a 100-point spreadsheet.