Investigating the Data Lifecycle

A useful data lifecycle starts with a clear purpose. Practitioners should be able to explain what the data is for, why it is needed and which decisions it supports. Organisations may use different names for the stages, but the key point is that the journey should be visible from collection to deletion. The lifecycle is also iterative, if a quality, privacy or security issue is found, practitioners may need to return to the source and correct the problem before the data is used again (Cabinet Office, 2020).

1. Data collection is the point at which data is created, captured or acquired. Sources may include forms, customer records, system logs, surveys, documents, public datasets or supplier data. Practitioners should ask: What is being collected? From whom? For what purpose? Who has approved it? Clear definitions, processes to validate and check and basic information reduce incomplete, duplicated or inconsistent records. For AI, the data should also be relevant and representative of the situation in which the system will be used. More data is not automatically better; personal data should be limited to what is necessary (ICO, nd a).

2. Data processing covers the actions used to organise and use data for a stated purpose. This may include checking, cleaning, de-duplicating, labelling, linking, analysing or producing reports. Practitioners should make sure the rules are repeatable and recorded. Small choices, such as removing unusual cases or filling missing values, can change the story the data tells. In AI work, training, testing and evaluation datasets should be separated appropriately, and known errors or limitations should be logged rather than hidden.

3. Data transformation changes the format, structure or meaning of data so it can be used. Examples include standardising dates, grouping free-text responses into categories, anonymising identifiers, combining records or creating features for an AI model. Transformation can make data easier to use, but it can also remove context or introduce bias. Practitioners should preserve lineage: the original source, each rule applied, who or what made the change and which version resulted. Government guidance on AI-ready datasets stresses the importance of structuring, documenting and governing data so it can be reused safely (GDS and DSIT, 2026).

4. Data storage concerns where data and its copies are held and how they are protected. Storage may include databases, cloud services, shared drives, local devices, backups, model-development spaces and supplier platforms. Practitioners should check who owns each location, who can access it, how it is protected and whether copies are controlled. AI projects often create extra files during testing and development, so teams need an inventory and audit trail. The ICO advises organisations to document movements and storage of personal data and remove intermediate files when they are no longer required (ICO, nd a).

5. Data sharing occurs when data is made available to another team, system, supplier, organisation or the public. Practitioners should check that sharing is necessary, lawful, proportionate and secure. They should also ask what the recipient is allowed to do with the data and whether it can be passed on again. Agreements should cover responsibilities, permitted use, retention, security, incident handling and deletion. The ICO describes responsible sharing as fair, safe and transparent (ICO, 2021).

6. Data retention is the period for which data remains available. Different records need different retention periods, depending on the purpose, legal duties, operational need, audit requirements and risk. Practitioners should check that raw data, transformed datasets, prompts, outputs, logs, model versions, test results and backups are covered by a retention schedule. Retention should be reviewed, not treated as permanent by default. Under the storage limitation principle, identifiable personal data must not be kept longer than necessary (ICO, nd b).

7. Data deletion is the controlled removal or destruction of data when it is no longer needed or retention has expired. Practitioners should make sure deletion covers copies, exports, temporary files, caches and supplier-held data. There should also be evidence that deletion has happened. Pressing delete may not securely erase data from storage media; sanitisation or destruction may be needed before equipment is reused or disposed of (NCSC, 2025). Where immediate deletion from backups is not possible, access should be restricted and deletion completed through the documented backup cycle.

Across all seven stages, practitioners should assign an owner, record lineage, monitor quality, apply security controls and review risks to people. NIST places testing, evaluation, verification and validation throughout the AI lifecycle, not only at the end (NIST, 2023). A lifecycle investigation therefore asks practical questions: Where is the data now? What happened to it? Why was that done? Who was responsible? What evidence proves the process is controlled?