Chapter 29 — Open-Source Components and Custom Development

1. Own the implementation, not an illusion of certainty

Open-source software changes what the enterprise can inspect and modify. It does not remove the need to decide what processing is permitted, where data may move or how an operator handles failure. Nor does buying commercial software remove those responsibilities. The useful comparison is allocation of work: which party maintains the component, integrates it, tests the deployed configuration and produces evidence after a change?

The original attraction remains valuable. An engineering team can inspect interfaces, restrict deployment paths and adapt a component rather than wait for a vendor roadmap. But the right to modify is not the capacity to maintain. A fork may add an immediate feature while making every later upgrade harder. The build/adopt decision should therefore include an owner, a bounded scope and a three-year maintenance hypothesis before source availability is treated as a procurement advantage.

Section 8(4) requires appropriate technical and organisational measures for effective observance, and Section 8(5) requires reasonable security safeguards.[1] Lines 343–347. Neither provision prescribes an open-source or commercial architecture. A public repository, a vendor certificate and a passing unit test are each narrower evidence than an assessment of the enterprise’s actual processing. This chapter uses software documentation as a starting point, not as legal certification.

2. Start with the supported baseline

Presidio’s retained documentation describes predefined and custom recognizers using named-entity recognition, regular expressions, rule logic and checksums, with Python/PySpark, Docker and Kubernetes usage options.[30] Lines 39–45. It also expressly warns that automated detection cannot guarantee discovery of all sensitive information and recommends additional protections.[30] Lines 47–49. That caveat is part of the source, not an editorial objection to the product.

For India, the retained supported-entities table includes IN_PAN, IN_AADHAAR, IN_VEHICLE_REGISTRATION, IN_VOTER, IN_PASSPORT and IN_GSTIN.[31] Lines 102–111. An Indian enterprise should not commission new Aadhaar, PAN and vehicle recognizers on the assumption that none exist. First establish what the selected package version actually loads, what language/model/configuration it uses, and how that baseline performs on the intended data. Published support is not measured quality, but it is the correct place to begin.

There is a further source-quality boundary. The entity table is a software support list, not an authoritative specification of government identifier validity. Its PAN description, for example, contains a “12 character” description.[31] Line 106. Do not build an identity-validation rule from that wording. Detection of a plausible identifier, validation of its format and verification that it belongs to a particular person are separate tasks. An actual implementation must source authoritative validation requirements separately where it needs them; this book does not fabricate such a validation service.

Fides is a different component family. The retained repository describes a privacy-engineering framework and licenses the tools under Apache Software License Version 2.0 while identifying the underlying fideslang specification as CC by 4.[38] Lines 124–139. That distinction matters: “the repository is open source” is not a complete licence review of code, taxonomy, models, container dependencies and any separately supplied commercial modules. Its documented optional MSSQL dependencies also illustrate how a connector can introduce deployment work beyond a basic package installation.[38] Lines 133–160.

The retained Fides documentation capture numbered 37 is a failed page, not evidence that Fides itself fails or lacks a feature. The usable repository supports an investigation, but this chapter does not infer complete runtime consent/rights support from an inaccessible page. Similarly, Privado’s local repository scan and optional dashboard synchronization establish an inspectable workflow description, not a complete runtime enforcement architecture.[18] Lines 25–41. These components can be combined only after their actual interfaces and responsibilities are verified.

3. Replace the universal build/buy boundary with a decision record

The author recommends evaluating each missing control separately. A generic policy engine may enforce client-authored rules; a commercial adapter may reach a derived store; custom code may be needed for a legacy monolith. It is not defensible to declare that all purpose enforcement must be built or that no product can reach a particular class of system without examining available integrations.

The record should name the task that remains unsolved after adoption. “We will use Presidio” leaves extraction, scanning schedules, findings access, classification review and inventory integration unspecified. “We will build a recognizer for our policy IDs because the measured baseline misses them” is a bounded engineering hypothesis. It is justified only by a test on the selected baseline, not by the nationality of the input data.

4. A bounded adoption case with a refused launch

The following is an author-designed CASE-001 extension, ADOPT-Q08-001, recorded in out/remediation/q08/adoption-model.json. The Company considers a local classification assistant for inventory review. Its output is a candidate classification requiring operator review, not automatic customer erasure, identity verification or a guarantee of anonymisation. It cannot authorise PUR-003 model training merely because a text has been masked. Purpose and identifiability assessments remain separate.

The proposed deployment extracts an approved document subset into an isolated scan job, runs a pinned recognizer/model configuration without optional external synchronization, emits restricted findings and deletes or restricts the temporary working data under the applicable retention decision. This is a proposed topology, not a tested guarantee that an unconfigured package makes no network calls. The PoV must inspect dependencies, model downloads, telemetry, logging and result paths. Any unreviewed outbound path blocks use of real customer content.

The selected runtime version remains deliberately unselected: Q08 has documentation snapshots, not an installed-package bill of materials. Before an actual trial, the engineering owner must freeze package, model, recognizer registry, configuration and container hashes, plus the relevant licences. The retained source hash provides a version for the documentary claim; it is not a substitute for the executable version. That unresolved adoption gate prevents an unsupported launch approval, while allowing the reader to evaluate costs and plan the experiment now.

The benchmark protocol should split documents by source/template and subject before dividing training, development and held-out sets. Otherwise duplicated forms can inflate apparent performance. Label exact spans and entity classes; have a second reviewer resolve ambiguous positives and hard negatives. Preserve multilingual, OCR-corrupted, short-context and rare-format strata. Tune on training/development only, and unlock the held-out set once for the declared candidate. Additional tuning requires a new hold-out plan rather than repeated peeking at the test set.

Measure extraction failures separately from recognizer misses: text never extracted cannot be rescued by an accurate text recognizer. Record which stores and formats were not scanned. A low false-positive rate on clean typed text does not establish recall on photographs or fragmented logs. The benchmark owner should report per-class results and the unresolved set, not only a single blended accuracy score that hides a high-consequence class.

5. A numerical illustration, not a Presidio benchmark

The following confusion counts are wholly hypothetical, supplied to demonstrate the calculations. They are not detector output, not measurements on the Company’s data, and not comparative claims about a product. The local verification script computes precision as TP/(TP+FP), recall as TP/(TP+FN), and F1 as 2TP/(2TP+FP+FN).

The Aadhaar row shows why high precision is not enough for discovery: twenty hypothetical positives are missed despite relatively few false alarms. The policy-identifier row suggests investigating customisation if real observations resembled these inputs, but it does not establish that Presidio has that weakness. An operator-review workflow may tolerate some false alarms while a release gate that assumes no personal data remains cannot tolerate the same uncertainty. Thresholds must follow intended use and review capacity, not a generic claim that a ninety-percent score is compliant.

This illustration also separates discovery misses from incomplete erasure. A detected record whose eligible deletion fails is an execution defect, not a false negative to average away. Conversely, restricted records retained under Rule 8(3) must not be counted as failed erasures simply because they still exist.[40] Lines 1153–1166. The statutory minimum and allowed retention purposes do not grant continued commercial use. The proposed scanner and its findings store need their own scoped retention decisions under the assumed future legal baseline.[39] Lines 49–59.[40] Lines 1005–1010.[41] Lines 25–38.

6. Three-year ownership economics

All figures below are invented planning inputs in INR, not supplier quotations, salary surveys or typical programme costs. The scope is the bounded classifier/connector component only; it is not Chapter 24’s whole privacy programme. The illustrative loaded rate is ₹2,000 per hour. Build/customisation effort is 420 hours in year one and 180 hours in each later year. Operations and regression effort is 240 hours each year. Infrastructure is ₹120,000 each year. Exit adds 80 hours in year three.

The computed three-year local total is ₹3,520,000. The commercial comparison assumes ₹400,000 setup, ₹450,000 annual licence, 120 internal hours per year at the same rate, and ₹200,000 exit: ₹2,670,000 over three years. Neither calculation includes taxes, discounting, inflation, outages or wider programme staff. The commercial option is assumed to meet the same scope solely for arithmetic; no supplier has demonstrated that equivalence.

Two sensitivities prevent a simplistic conclusion. If the hypothetical annual licence is ₹900,000 instead, the commercial total is ₹4,020,000. If local upgrades instead require an extra 220 hours in each of years two and three, the local total rises to ₹4,400,000. Both changes are scenario parameters, not forecasts of release cadence. There is no retained basis for claiming that open-source projects typically issue major releases every twelve to eighteen months or that commercial platforms are usually cheaper after maintenance.

Capacity is a separate gate. The example reserves only 600 engineering/operations hours in year one, while the local plan demands 660, a shortfall of 60 hours. A lower cost estimate cannot manufacture those hours. The teaching decision is therefore “defer local launch pending capacity, runtime/licence and quality evidence,” not “approve because open source is free.” The commercial option also remains unapproved until the quoted scope, data movement, quality and exit evidence are tested. At the higher licence assumption the local path may become economically attractive, but only if the missing capacity and technical gates can actually be closed.

7. Upgrade, fork and exit responsibilities

Assign a primary and backup maintainer. Keep a list of upstream dependencies, selected versions, local patches and the tests each patch exists to satisfy. Prefer an extension point over a fork when it meets the requirement, because it can reduce the amount of code the client must reconcile; that is an engineering recommendation, not a claim about a particular project’s support policy. Every upgrade proposal should show changed detection behavior and a rollback or restriction plan, not only a successful installation.

Re-run the held-out regression suite for changed recognizers, models, extraction libraries and connector semantics. Preserve previous expected behavior and explain intentional changes. A new model can alter both precision and recall; a connector update can alter which data is scanned without changing the recognizer at all. Security review should cover the scanner’s privileged access and findings store as well as the package dependency list. A vulnerability is a risk requiring assessment and response, not automatic proof of a legally adjudicated safeguard breach.

Exit must recover useful inventory, mappings, configuration and unresolved findings without requiring the old runtime. Import a representative export into a neutral schema and reconcile relationships, not merely file counts. Keep the legal disposition of retained evidence separate from deleting working copies. If a vendor or maintainer disappears, the enterprise needs enough documentation and rights to continue safely or restrict the affected workflow while replacing it.

The reader’s final adoption record should contain the chosen scope, candidate baseline, licence/dependency gates, deployment/data movement, corpus protocol, actual measured results when available, cost assumptions, capacity and exit owner. Q08 supplies the numerical illustration and reproducible calculations, not a fictitious benchmark. Chapter 30 applies the same distinction between assumptions, evidence and permission to award a contract.

Source notes

Retained source hashes and URLs are listed in out/remediation/q08/source-manifest.json. The arithmetic inputs and actual local calculation results are out/remediation/q08/adoption-model.json and out/remediation/q08/verification-results.json. Run python3 -B out/remediation/q08_verify_calculations.py from the book root after installing the documented local formula-engine dependencies.

Retained file map

The line ranges cited above refer to these exact local captures:

• [1] research/legal/evidence/01_dpdp_act_2023_gazette.txt
• [18] research/solutions/evidence/18-docs.privado.ai-quick-start-privado.md
• [30] research/solutions/evidence/30-microsoft.github.io-index-presidio.md
• [31] research/solutions/evidence/31-microsoft.github.io-pii-entities-supported-by-presidio-presidio.md
• [38] research/solutions/evidence/38-github.com-github-ethyca-fides-the-privacy-engineering-compliance-framework-github.md
• [39] research/legal/evidence/02_gsr_843e_commencement.txt
• [40] research/legal/evidence/05_gsr_846e_dpdp_rules_2025.txt
• [41] research/legal/evidence/06_gsr_892e_corrigendum.txt

Sources

[1] https://www.meity.gov.in/static/uploads/2024/06/2bf1f0e9f04e6fb4f8fef35e82c42aa5.pdf — https://www.meity.gov.in/static/uploads/2024/06/2bf1f0e9f04e6fb4f8fef35e82c42aa5.pdf [18] https://docs.privado.ai/getting-started-with-privado/getting-started-with-privado [30] https://microsoft.github.io/presidio [31] https://microsoft.github.io/presidio/supported_entities [38] https://github.com/ethyca/fides [39] https://www.meity.gov.in/static/uploads/2025/11/c56ceae6c383460ca69577428d36828b.pdf — G.S.R. 843(E), DPDP Act commencement notification [40] https://www.meity.gov.in/static/uploads/2025/11/53450e6e5dc0bfa85ebd78686cadad39.pdf — Digital Personal Data Protection Rules, 2025, G.S.R. 846(E) [41] https://www.meity.gov.in/static/uploads/2025/12/3c7ebbae0e5456f493f486e6845df86b.pdf — Corrigenda to G.S.R. 846(E), G.S.R. 892(E)

Contents · Reader guide and citation conventions · Artifact index