Berlin Takes a Systematic Approach to Purging Duplicate Images From Public Records — but Rivals Are Moving Faster

Berlin's Senate Department for Urban Development and Housing confirmed this spring that its digital property and planning archive — one of the largest municipal image repositories in Germany — contains an estimated 340,000 duplicate or near-duplicate image files accumulated since digitisation began in earnest around 2009. The backlog is costing the city real money: storage contracts with the city's IT service provider ITDZ Berlin run to millions of euros annually, and redundant files are a documented drag on retrieval speeds for planners working on projects from Mitte to Marzahn.

The timing matters. Berlin is in the middle of a contested overhaul of its housing and planning workflows, with the SPD-led coalition pushing to speed up building permit approvals to address the city's chronic housing shortage. Slow, cluttered digital systems are a direct bureaucratic bottleneck. Every wasted second a planner spends sifting duplicate aerial photographs of Tempelhof or repeat construction-site snapshots from Friedrichshain is time not spent approving new units.

What Berlin Is Actually Doing

The city's primary response has been a phased deduplication programme run through the Landesarchiv Berlin on Eichborndamm in Reinickendorf and coordinated with the Senatsverwaltung für Digitalisierung. The programme uses perceptual hashing — an algorithmic method that identifies visually identical or near-identical images even when file names or metadata differ — rather than simple checksum matching, which would miss re-saved or slightly compressed copies. A pilot covering roughly 80,000 files from the urban planning catalogue ran from October 2024 through March 2025. According to documents published by the Senatsverwaltung, the pilot flagged 23 percent of tested files as duplicates eligible for deletion or archival consolidation.

The Bezirksamt Pankow, which manages one of the city's densest local planning queues, began its own parallel clean-up in January 2026, targeting neighbourhood documentation files for the Prenzlauer Berg and Weissensee districts. Staff there are working with open-source tooling rather than proprietary software, a deliberate cost-control decision given Pankow's constrained district budget.

How Berlin Compares With Amsterdam, Vienna and Warsaw

Berlin is not the only European capital wrestling with this. Amsterdam's Stadsarchief completed a comparable deduplication exercise across its historical photograph collection in 2023, reducing its active image database by roughly 18 percent and freeing an estimated 14 terabytes of primary storage. Vienna's MA 01 — its municipal administrative department — has embedded automated duplicate detection directly into its document management system since 2022, meaning new duplicates are flagged at the point of upload rather than discovered years later in retrospective audits. Warsaw, which undertook rapid digitisation after flood damage to municipal buildings in 2010, built deduplication into its initial ingest pipeline from the start.

By those benchmarks, Berlin is catching up rather than leading. The city's federated structure — twelve largely autonomous Bezirke, each with its own records systems and IT procurement histories — makes a single unified solution harder to mandate. What Amsterdam or Vienna can enforce centrally, Berlin must negotiate borough by borough. The Senatsverwaltung für Digitalisierung has been pushing a unified metadata standard since 2023, but adoption across all twelve districts is not expected to be complete before the end of 2027 at the earliest.

The financial stakes are not trivial. ITDZ Berlin's published rate card for tier-one storage stands at roughly €0.08 per gigabyte per month. Across a repository inflated by years of duplicate accumulation, even modest reductions in stored volume translate into six-figure annual savings — money that the housing-focused coalition would rather direct toward the BerlinPass housing subsidy programme or BVG infrastructure upgrades.

For residents and businesses dealing with the city's planning offices — whether filing renovation permits in Kreuzberg or submitting commercial development documents near Ostbahnhof — the practical upshot is straightforward: upload documents once, in the highest resolution available, with clear file names and consistent metadata. The city's deduplication tools work best when source material is clean. A guidance document from the Landesarchiv Berlin, updated in April 2026, recommends TIFF or high-resolution JPEG formats and advises against re-submitting previously lodged photographs under new file names, which has historically been one of the main drivers of the duplicate problem in the first place. The full citywide rollout of automated duplicate detection at the upload stage is now scheduled for a 2027 pilot across three districts, with Pankow, Mitte and Tempelhof-Schöneberg named as the likely test cases.