Berlin's Digital Archives Are Riddled With Duplicate Images — And Officials Say the Clean-Up Bill Is Growing

Berlin's public sector is sitting on a storage problem that has been building for years and is now, according to administrators and digital infrastructure specialists consulted this week, reaching a tipping point. Across municipal databases — from the Senatsverwaltung für Stadtentwicklung on Württembergische Straße to the Zentral- und Landesbibliothek Berlin in Mitte — duplicate images have accumulated in volumes that are straining both server capacity and the staff hours required to manage them.

The issue matters now because the SPD-led Senate coalition has committed to a broad digitisation push under its Berliner Digitalstrategie framework, with expanded funding earmarked through 2027. As agencies rush to digitise physical archives and migrate legacy systems, technical teams say the duplication problem is being baked into new infrastructure rather than resolved before migration. The result, specialists warn, is that clean-up costs compound with every passing quarter.

What the Experts Are Saying

Digital asset management consultants working with Berlin-based public clients describe the core problem in consistent terms: image files are ingested multiple times across departments without deduplication protocols, and metadata tagging is inconsistent enough that automated matching tools flag false negatives. One widely cited benchmark in the sector holds that public institutions with archives exceeding 500,000 digital assets typically carry a duplication rate of between 15 and 30 percent — a range that, applied to Berlin's larger repositories, translates to tens of thousands of redundant files.

The Stadtbibliothek network, which spans 74 branch locations across the city's 12 Bezirke, has been piloting a deduplication review as part of its ongoing Digitales Magazin project. Technical staff there have described the workflow challenge publicly at industry forums: without a unified content identifier system across branches, the same scanned periodical page can exist in three or four versions under different file names, each version consuming server space and appearing separately in search returns. For archivists working under already tight staffing conditions — the library system has flagged recruitment gaps repeatedly since 2023 — manually resolving these conflicts is not a realistic option.

Officials at the Senatsverwaltung für Kultur und gesellschaftlichen Zusammenhalt, which oversees significant portions of Berlin's publicly funded digitisation work, have acknowledged the problem in general terms at budget committee hearings this spring, without committing to a specific remediation timeline or cost figure. Independent data governance advisers who work with Berlin's Bezirksämter say the absence of a citywide deduplication standard — something comparable to the UK's Government Digital Service content guidelines, which have been in force since 2012 — is the structural gap that allows the problem to persist.

Costs, Timelines, and What Comes Next

Storage is not free. Commercial cloud pricing for the kind of enterprise-grade services used by Berlin's public institutions runs roughly between €0.02 and €0.05 per gigabyte per month, depending on redundancy and retrieval tier. For an archive carrying 20 percent more data than it should because of duplication, the overhead is not trivial — and it scales directly with every new digitisation tranche added to the system.

The practical path forward, according to specialists in the field, involves three steps that Berlin's digital teams are already discussing but have not uniformly adopted. First, agreeing on a shared hashing standard — a technical fingerprint for each image file — so that duplicates can be identified automatically at the point of ingest rather than retrospectively. Second, establishing a central image registry, potentially housed within the existing IT infrastructure of the Berlin state government's ITDZ Berlin, the public IT service provider based in Straße des 17. Juni. Third, running a one-time retrospective audit before the next major migration cycle, currently anticipated for late 2026.

Vendors and administrators working on Berlin's tech infrastructure say decisions made in the next six months will either lock in the duplication problem for another decade or give the city a genuinely clean foundation for its digital public record. The Digitalstrategie framework provides the political cover and, nominally, the budget. What is still missing, insiders say, is someone with the authority to make the call.